diff --git a/.gitignore b/.gitignore
index 5f09bd5..83a7e98 100644
--- a/.gitignore
+++ b/.gitignore
@@ -1,6 +1,7 @@
# Python
__pycache__/
*.py[cod]
+/src/skills/library/
*$py.class
*.so
.Python
@@ -85,14 +86,48 @@ node_modules/
desktop/dist/
desktop/out/
packaging/windows/build/
+packaging/windows/bin/electron/
+packaging/windows/bin/*.pdb
npm-debug.log*
+/JonathanAi.exe
+/JonathanAi-Setup.exe
+/.jonathan-ai-processes.json
+/.jonathan-ai-runtime-*.ready
+/codex-last-message.txt
+/codex-task-prompt.txt
# Local desktop captures (user-initiated screenshots)
.clawd/captures/
+# Local evaluation checkouts plus Jonathan's downloaded image models/outputs.
+# None of these directories are runtime source dependencies or installer payloads.
+/integrations/Fooocus/
+/integrations/.image-models/
+/integrations/.fooocus-venv/
+/integrations/python-runtimes/
+/integrations/.uv-cache/
+/integrations/.pip-cache/
+/integrations/.fooocus-cache/
+/integrations/Fooocus-outputs/
+/integrations/upstream/
+/Integrations/ECC/
+/Integrations/Kronos/
+/Integrations/Securo/
+/Integrations/ClaudeDB/
+/Integrations/BenjaminPlus/
+/Integrations/FullstackAgent/
+/Integrations/Procoder/
+/Integrations/DrawAI/
+/Integrations/Kindergrimm/
+/integrations/.kronos-venv/
+/integrations/.kronos-cache/
+
# Durable agent memory and chat sessions stay on the machine, never in git
+/Skills/
.clawd/memory/
.clawd/sessions/
+.clawd/checkpoints/
+.clawd/agent_profiles.json
# Claude Code personal settings
CLAUDE.local.md
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 4e213b0..0f2b3a2 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,19 +1,211 @@
# Changelog
+## [0.4.8] - 2026-08-27
+
+- Fixed the desktop startup failure that left the conversation window blank or displayed `escapeHtml is not defined`. The shared escaping primitive is restored, persisted user/assistant messages render again, and Markdown, engine-status, image, attachment and artifact rendering retain HTML escaping.
+- Hardened response delivery: if the browser's Server-Sent Events connection is aborted, the conversation automatically resumes through the bounded job-events endpoint, keeps the working indicator active, and reloads the saved transcript when the job completes. A completed answer can no longer disappear merely because the live socket closed.
+- Replaced the nonfunctional `RemoteTrigger` placeholder with permission-gated SSH command execution, SCP upload/download, Windows Remote Management execution and bounded TCP reachability checks for explicitly named computers and servers. It uses existing SSH keys/agent or stored Windows credentials, never accepts passwords in model/tool input, validates hosts, defaults to strict host-key checking, uses argument arrays with `shell=False`, and remains subject to the remote machine's authorization, ACL and elevation rules.
+- Added regression tests for visible response infrastructure, remote-action permission gating, host validation, strict SSH invocation and password exclusion; visually verified a new local-model reply in the real conversation window.
+- Updated product, desktop, installer, documentation and test version surfaces for 0.4.8.
+
+## [0.4.7] - 2026-08-25
+
+- Evaluated Apache-2.0/MIT LoopX and adopted its strongest non-duplicative idea as Jonathan's independent `LongHorizonControl`: durable objectives, workspace scope, bounded turns, dependency-aware todos, agent claims with expiring leases, explicit human/evidence gates, UTF-8 evidence receipts, handoffs, and fail-closed continuation/closure decisions. No upstream source, dashboard, adapter, PWA/Tauri shell, branding, service, or runtime is bundled.
+- Evaluated `cporter202/scraping-apis-for-devs`. It is an unlicensed catalog of thousands of changing third-party listings, predominantly paid Apify actors, rather than a scraping engine or API implementation. Jonathan's audited `PublicApiCatalog`, connectors, `ContentReach`, WebFetch and browser tools already cover safe discovery and retrieval, so no catalog mirror, vendor listing, credential flow, actor, branding or automatic scraper is bundled.
+- Evaluated MIT OGAM. Jonathan already covers its local/LAN model discovery, GGUF-capable runtimes, vision, Whisper, diffusion, attachments, retrieval, tool loops and user approval; the useful non-duplicative resource-awareness principle is now native in the local-model connector, with stored model bytes, live runtime RAM, available system RAM and conservative fit guidance. No React Native app, Android/iOS native module, mobile model bundle, paid Pro feature, branding or upstream runtime is included.
+- Fixed desktop dependency setup to keep npm/Electron caches inside Jonathan's install folder and to report success only when the Electron runtime binary exists, preventing silent headless fallback and repeated first-launch repair.
+- Evaluated CC BY-SA 3.0 Awesome Selfhosted and adopted the useful operator workflow without copying its 2,000+ changing links: native `SelfHostedService` registers Docker/Podman Compose, Windows-service and systemd workloads; reports status and bounded logs; checks local/LAN health; creates downloadable bounded backups; and performs approved start/stop/restart/pull-update actions with fixed argument lists and no shell. A visible independent `self-hosted-service-operator` skill governs discovery, licensing, secrets, backup, rollback and verification. No upstream catalog, description, non-free list, branding or service deployment is bundled.
+- Updated product, desktop, installer, documentation and test version surfaces for 0.4.7.
+
+## [0.4.6] - 2026-08-25
+
+- Evaluated Apache-2.0 Awesome LLM Apps as an agent/RAG/MCP cookbook. Its templates duplicate Jonathan's provider, agent-team, retrieval, skill and connector foundations, so no second framework, demo application, prompt pack or upstream runtime is bundled.
+- Evaluated MIT Public APIs and adopted its useful discovery metadata shape as Jonathan's native `PublicApiCatalog`: an offline audited starter index can be searched by category/auth/HTTPS/CORS and approved entries can become persistent connector cards, without cloning or depending on the upstream list.
+- Evaluated Humanizer Stack's mixed MIT/CC BY-SA two-pass writing workflow and added an independent bundled `writing-polish` skill for opt-in surface and structure editing, voice-sample calibration, meaning preservation, honest review notes, and explicit authorship/detector boundaries; no upstream skill text, scanner, research digest, or installer is included.
+- Evaluated AI Workflow Hub 2000. It is an n8n workflow-JSON collection with no repository license, per-workflow trust manifest, or execution isolation; bulk importing credentialed third-party automations would weaken Jonathan's audited connector/tool boundary, so no workflow, branding, service dependency, or upstream checkout is bundled.
+- Evaluated MIT JustInCache n8n Workflows. Its SQLite search/filter UI is useful but already covered by Jonathan's native searchable catalogs, while its 2,000+ workflow graphs require a separate n8n runtime and broad external credentials; no corpus, FastAPI service, container stack, branding, or automatic workflow execution is bundled.
+- Evaluated MIT Open Generative AI. Its broad image/video/lip-sync UI is backed by a required MuAPI gateway key rather than a bundled local model fleet and duplicates Jonathan's desktop/media surfaces; no second Electron/Next.js shell, paid gateway, submodule, branding, unfiltered-policy bypass, or upstream runtime is included. Jonathan retains local image/audio/3D engines and explicit ComfyUI/external connectors for heavier video models.
+- Evaluated MIT Taste Skill and adopted its compatible adjustable-art-direction principle as an independent bundled `interface-art-direction` skill: layout variance, motion intensity and information density are chosen from the product context, then checked against accessibility, responsive behavior and rendered evidence. No experimental upstream prompt, code skeleton, image asset, branding or installer is included.
+- Evaluated Vercel Labs Agent Skills. Because its repository currently lacks a top-level license file despite a README MIT statement, no upstream skill text is redistributed; Jonathan adds an independent `react-performance-review` skill for evidence-led waterfall, bundle, rendering-boundary, rerender and loading analysis with before/after verification.
+- Evaluated MIT Awesome Design MD and adopted its compatible persistent-design-context principle: a project-owned UTF-8 `DESIGN.md` is now loaded into every conversation agent under a separate bounded design-system section. No third-party brand analysis, preview page, token catalog, trademark, or upstream checkout is bundled.
+- Evaluated CC0-style gztchan Awesome Design. It is a curated external-link list rather than an executable tool, and linked services have independent terms and changing availability; Jonathan's live search, connector, design-skill and `DESIGN.md` paths already cover the useful workflow, so no stale link mirror or upstream checkout is bundled.
+- Evaluated MIT Screenshot to Code and adopted its useful image-to-working-interface and render-and-compare workflow as an independent bundled `screenshot-to-interface` skill. Jonathan uses its existing uploads, multimodal agents, project preview and coding tools without adding a second FastAPI/Vite app, paid provider requirement, Chromium download, branding or upstream runtime.
+- Evaluated MIT OpenCode. Its terminal/desktop clients, build/plan agents, provider catalog, sessions, LSP, MCP and permission model overlap Jonathan's existing native shell, per-conversation agents, Plan Mode, connectors and approval gateway; no second Bun/Tauri runtime, branding, cloud service, installer or upstream checkout is bundled.
+- Evaluated MIT OpenCode Voice and OpenAI Whisper. Jonathan now includes user-initiated conversation microphone recording, local Whisper-family multilingual transcription/translation with first-use model download, raw-transcript fallback behavior, and local OS speech synthesis to downloadable WAV through native `SpeechStudio`; no background listening, mandatory normalization LLM, Piper/SoX/OpenCode runtime, branding or upstream checkout is bundled.
+- Evaluated MIT ScrapeGraphAI. Its LLM graph pipelines substantially overlap Jonathan's WebSearch/WebFetch/browser, local-model, worker and structured-output stack while adding a second provider abstraction, Playwright footprint and opt-out telemetry; no crawler framework, hosted-credit API, branding, telemetry or upstream checkout is bundled.
+- Evaluated MIT Agent Reach and adopted its useful channel-health concept as native `ContentReach`: public RSS/Atom and YouTube metadata/captions use lightweight maintained libraries, while GitHub/GitLab and authenticated social services remain in Jonathan's explicit connector/approval system. Jonathan never reads browser cookies automatically and does not install Agent Reach, OpenCLI, proxy services, sponsored APIs or upstream skills.
+- Evaluated BSD-3-Clause Scrapling. Jonathan adopted safer visible-page text extraction that drops scripts, styles, templates, head content and hidden elements before model context; stealth fingerprinting, CAPTCHA/Turnstile bypass, cookie sessions, proxy rotation, anti-bot evasion, a second browser bundle and the upstream runtime are intentionally excluded.
+- Evaluated MIT Higgsfield AI Skills. Its image/video/audio/brand/product/website workflows are coupled to the Higgsfield hosted CLI, authentication, vendor models and branded templates, duplicating Jonathan's native media, design, 3D, website, artifact and skill systems; no vendor skill text, CLI, assets, model catalog, branding or paid-service dependency is bundled.
+- Evaluated Apache-2.0 Higgsfield GPU orchestration. Its 0.0.3-era Linux cluster stack requires remote Ubuntu nodes, Docker, SSH deploy keys, passwordless sudo and distributed-training infrastructure, so it is not a safe standard-desktop dependency; Jonathan retains explicit NVIDIA/cluster connectors and native system administration rather than installing it.
+- Evaluated Augmented Unreality, a legacy Unreal Engine 4.19 camera/ArUco tracking plugin. Its Unreal, OpenCV-contrib, GStreamer, calibration-asset and platform-native build requirements do not improve Jonathan's desktop conversation vision, image editing or 3D artifact workflow, so no obsolete engine plugin or runtime dependency is bundled.
+- Evaluated the MIT Unreal Augmented Reality Toolkit. Its four-commit C++ implementation only copies webcam frames into an Unreal dynamic texture, while marker detection remains a roadmap item; Jonathan's attachment/media preview and vision tools already cover the useful capability without an Unreal/OpenCV SDK build, so no upstream source or dependency is bundled.
+- Evaluated MIT Unreal Engine MCP and adopted the useful live-editor workflow as Jonathan's independent `UnrealStudio`: it discovers installed editors and projects, inspects descriptors, generates reviewable declarative scene scripts, and runs an explicitly approved project-local Python script through `UnrealEditor-Cmd`. The acquired hosted service, API keys, branding, TCP/C++ plugin, arbitrary remote execution and upstream runtime are not bundled.
+- Evaluated MIT 9router. Its localhost proxy, provider-format rewriting, subscription OAuth storage, multi-account rotation, tool-result compression and silent paid/free fallback duplicate Jonathan's native providers, connector vault, token meters and bounded selected-provider retry while adding privacy, billing and semantic-routing risk; no proxy, credential harvester, prompt modifier, branding, cloud sync or upstream runtime is bundled.
+- Replaced the managed Fooocus/Gradio checkout with Jonathan's native in-process Diffusers text-to-image and image-to-image engine, preserving downloadable artifacts while removing the upstream service/runtime dependency and avoiding heavy model imports during status checks.
+- Evaluated MIT OmniRoute and adopted a small native selected-provider resilience layer: transient 408/409/425/429/5xx and transport failures receive one bounded retry, repeated failures open a cooldown circuit, visible streams are never replayed, and prompts are never silently moved to another provider.
+- Evaluated Anthropic's official Claude plugin catalog. Jonathan already supports the portable `SKILL.md` and MCP concepts through its audited skill/connector systems; Claude-specific hooks, commands, marketplace runtimes, and mixed third-party licenses are therefore not bundled or auto-enabled.
+- Evaluated CC BY 4.0 Task Observer by Eoghan Henn/rebelytics and adopted its compatible friction-evidence principle: sanitized user corrections, tool failures and repeated tool sequences now form confidence-scored review candidates, while drafts remain visible and never activate without user approval.
+- Evaluated Apache-2.0 Headroom and adopted its compatible stable-prefix principle directly in Jonathan's Anthropic adapter via prompt-cache markers. No proxy, Rust extension, ML compressor, prompt rewriting, dropped messages, account, or upstream runtime is used.
+- Evaluated Anthropic's Apache-2.0 community plugin marketplace mirror. Its third-party entries, mixed licenses, external runtimes, stale pins and catalog-scale failures are not bulk-imported; Jonathan retains explicit connector setup and per-skill security review/enablement.
+- Evaluated Apache-2.0 YuE and MIT ACE-Step UI. Their full-song models remain optional external/local-server choices because the documented model, CUDA, VRAM and disk requirements are unsuitable for every standard install; Jonathan now includes a native `AudioStudio` for deterministic instrumental WAV generation, trim, normalize, mix and conversation downloads.
+- Added a revision-pinned, independently updatable ECC integration that indexes the complete skills and agent-template catalog while lazily loading only explicitly enabled managed skills.
+- Added UTF-8 catalog import with provenance, atomic replacement, recoverable archives, duplicate avoidance, full-repository reporting, and critical checks scoped to content that can actually be activated.
+- Added `ECCIntegration` agent tool and desktop controls for sync, security scan, full import, per-skill enable/disable, agent-template instantiation, and inspectable Memory Vault export/import.
+- Added native-Windows evidence capture and Git-history learning with secret/instruction sanitization, project isolation, confidence-scored instincts, validated `SKILL.md` drafts, and mandatory review before activation.
+- Added a Jonathan-native probabilistic OHLCV research engine, informed by the Kronos evaluation, with recency-weighted drift/volatility, Monte Carlo intervals, downloadable CSV artifacts, and no upstream checkout, model download, or automatic trading.
+- Added a Jonathan-native SQLite personal-finance vault, informed by the Securo evaluation, for accounts, transactions, categories, recurring flags, budgets, goals, assets and currency-separated reports with no containers, cloned service, or payment requirement.
+- Evaluated Pydantic AI's typed tools/outputs, provider adapters, durable execution, evaluation, and observability. It remains a future opt-in adapter rather than a second core agent loop because duplicating Jonathan's sessions, permissions, tools, and provider routing would increase startup cost and correctness risk without adding an immediate user capability.
+- Unified desktop visual states with a consistent system font, focus-visible keyboard treatment, compact connector hierarchy, reduced-motion support, and preserved per-conversation/per-agent token tracking without billing gates.
+- Added a redacted, UTF-8, tamper-evident append-only execution ledger inspired by Apache Maka's runtime-fact model; job lifecycle, messages, tool calls/results, permissions, failures, completion, usage, and artifacts survive separately from provider context projections.
+- Added local earned-autonomy evidence profiles after evaluating Universal Earned Autonomy Layer: distinct approval/rejection tracking per conversation and tool, Wilson lower-bound confidence, hard risk ceilings, replay resistance, and transparent recommendations that never silently widen permissions.
+- Evaluated CopilotKit OpenBot without importing its alpha Docker/Bun/PostgreSQL/licensed-service stack; adopted its compatible preflight-gateway principle so every approved tool action is secret-redacted and durably recorded before execution, then recorded again with its result, failing closed if preflight audit persistence fails.
+- Evaluated the functional Scandinavian Design agent skill and its ten-site visual harness without copying its unlicensed skill/source; independently refined Jonathan's dark desktop with neutral solid surfaces, reduced decorative gradients/glow/blur, consistent compact radii, sentence-case section labels, and retained focus/contrast/reduced-motion accessibility.
+- Evaluated the MIT Autoprompt Skill and adopted its compatible separation-of-judgment pattern as opt-in Fast/Balanced/Verified internal-worker modes. Verified adds an independent final reviewer, while all worker/reviewer usage rolls into the owning conversation's informational token count; no mode runs implicitly.
+- Added Jonathan-native `CodeMemory` for UTF-8 SQLite notes, incremental project scans, text/usages/explain/path queries, and symbol/call/import graphs; Claude DB is a design reference only and no upstream runtime, hooks, account, or subscription are required.
+- Evaluated the MIT Benjamin Plus efficiency skill and incorporated its compatible measured workflow disciplines into both full and lightweight agent prompts: one-pass reconnaissance, bounded inspection, batched dependency probes, exact verification, and low-chatter waits, always subordinate to correctness.
+- Evaluated the AGPL Fullstack Agent setup conductor. Its Claude/Obsidian-centered memory, UI, updater, permissions and installer duplicate Jonathan, while voice/face/webcam features belong to separate component repositories; the meta-installer is therefore not embedded in Jonathan.
+- Evaluated Apache-2.0 Procoder and rewrote its compatible fail-closed principles as Jonathan's native engineering gate: unchecked work fails, canonical tests are detected, conflict/secret/junk hygiene is enforced, and no upstream binary or hook is installed.
+- Evaluated Apache-2.0 DrawAI and added a native raster-to-editable engine using Jonathan's installed Pillow/OpenCV/OCR/PowerPoint stack; it exports editable SVG, native-shape PPTX, review PNG and JSON without an upstream checkout or >8 GB model pack.
+- Evaluated the Unlicensed/Public Domain Kindergrimm procedural art engine and added an independent deterministic animated SVG character/rig generator with JSON recipes.
+- Evaluated MIT Scroll Craft and added a native bundled `scroll-storytelling` skill covering distinctive page grammars, feeling curves, accessibility, efficient scroll mechanics, reduced motion, and visual verification without an upstream plugin/runtime.
+- Evaluated MIT Hermes3D and added a native spatial agent floor to Jonathan's existing control center, showing live named-agent desks, working/idle state, unread mail and token totals with direct focus controls; no 3D frontend, gateway, adapter or upstream runtime is required.
+- Updated the standard installer/bootstrap to synchronize and import ECC automatically when online while keeping offline installation usable and retryable.
+- Upgraded the Electron shell to 44, moved build caches inside the ignored packaging workspace, hardened build failure checks, and produced a fresh standard Windows installer with a clean high-severity dependency audit.
+- Updated all version, installer, desktop, feature, license, test, and documentation surfaces for 0.4.6.
+
+## [0.4.5] - 2026-08-24
+
+- Added an explicit, persistent, and revocable Full Device & Network Access approval that expands every conversation agent from workspace-only access to the signed-in operating-system account.
+- Added installed application, browser, and PATH tool discovery plus audited application/file/URL launching and mouse, keyboard, hotkey, and desktop screenshot automation.
+- Kept the mode disabled by default, requires the exact `ENABLE FULL ACCESS` confirmation, preserves Windows ACL/UAC boundaries, and records enable/revoke actions in the redacted audit log.
+- Added desktop controls showing access scope, elevation status, and a live installed-tool inventory.
+
+## [0.4.4] - 2026-08-24
+
+- Added one visible shared `Skills` library with a desktop editor; every conversation agent can discover, validate, create, update, and execute its UTF-8 `SKILL.md` files.
+- Added a concrete self-learning workflow and prompt guidance so agents use `SkillManager` instead of claiming they cannot create operational files.
+- Added fast local image inspection for dominant colors, shapes, people/faces, QR codes, sharpness, and optional RapidOCR text recognition while still sending original pixels to vision models.
+- Bounded provider context without truncating persisted history and stopped resending multi-megabyte image pixels on every tool turn.
+- Switched the Windows release to the standard Inno Setup install/upgrade wizard, preserving the user skill library during upgrades.
+
+## [0.4.3] - 2026-08-24
+
+- Fixed OpenAI-compatible/local agent turns dropping multimodal block messages, so LM Studio, Ollama, vLLM, and OpenAI-compatible vision models receive uploaded image pixels.
+- Preserved binary file and image bytes for file-picker and drag-and-drop uploads instead of decoding pathless images as garbled clipboard text.
+- Added bounded PDF, DOCX, XLSX/XLSM, PPTX, and ZIP extraction plus persistent download links for uploaded files.
+- Added local Fooocus image-to-image variations for prompt-guided edits, with cloud 401/403 fallback.
+- Simplified the conversation surface to responses, uploaded/generated media, downloads, permissions, errors, and a bottom-left progress robot; raw tool cards stay hidden.
+- Kept staged paths and extracted attachment context available to the agent while hiding backend attachment instructions from the visible transcript.
+
+## [0.4.2] - 2026-08-24
+
+- Make the installed local Fooocus engine the default for image generation, even when a stale cloud key exists.
+- Fall back to Fooocus when an explicitly selected cloud image endpoint returns HTTP 401 or 403.
+- Expose an explicit local/cloud generation-engine selector in Image Studio.
+- Close Pillow image handles deterministically so Windows downloads and overwrites do not fail with file-in-use errors.
+
+## [0.4.1] - 2026-08-24
+
+- Render uploaded images directly inside their persistent conversation messages.
+- Persist generated image artifacts per conversation across reloads and restarts.
+- Add authenticated inline image-view and forced-download routes for uploads and generated files.
+- Preserve original attachment filenames after per-session staging.
+
+## [0.4.0] - 2026-08-24
+
+- Added fast full-text search across persistent conversations, summaries, and shared memory.
+- Added persistent named agents with canonical conversations, parallel execution, local mailboxes, cross-agent work, and per-agent token totals.
+- Added recoverable pre-run checkpoints, Undo, Retry, and checkpoint restore APIs.
+- Added a one-click system diagnostics and safe local repair dashboard.
+- Bundled 200 permissively licensed skill packages with their supporting files and notices.
+- Removed Jonathan's former 100-message conversation truncation; stored chat history is now unbounded by an app quota.
+
All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+## [0.3.3] - 2026-08-24
+
+### Fixed
+- Normal desktop launches skip dependency repair after a version-specific healthy-runtime marker is written, removing the repeated native-launcher and Electron bootstrap delay.
+- Electron selects a free localhost port, verifies the authenticated status endpoint, handles backend spawn errors, and writes structured startup diagnostics to `~/.clawd/run/desktop.log` instead of accepting an unrelated stale backend.
+- Ordinary local-model conversation omits the 50-tool schema payload, uses bounded recent history, and limits output so short prompts respond promptly.
+- Actionable local-model requests receive a focused tool subset; Tool Search can expand that subset on later turns without sending every schema up front.
+- Restored session metadata now reflects the provider/model that actually handles the conversation.
+
+## [0.3.2] - 2026-08-23
+
+### Added
+- Managed official Fooocus checkout under `integrations/Fooocus`, isolated Python 3.10 runtime, CPU/CUDA-aware PyTorch installation, pinned Gradio-compatible web dependencies, and automatic first-run SDXL model download.
+- Fooocus connector card with install/repair, start/stop, embedded localhost preview, direct prompt generation, output discovery, and conversation downloads.
+- Agent-facing `Fooocus` tool plus Image Studio local-generation fallback when no cloud image API key is configured.
+- Device-wide local AI discovery for Ollama, LM Studio, Jan, GPT4All, llama.cpp, KoboldCpp, Hugging Face caches, and additional OpenAI-compatible loopback ports.
+- Automatic startup/selection of an installed Ollama or LM Studio model when no working provider is configured, with endpoint-scoped local credentials.
+
+### Fixed
+- Redirected Fooocus, pip, uv, Torch, Hugging Face, Transformers, Matplotlib, and XDG caches away from Windows home folders with restrictive ACLs.
+- Preserved completed Fooocus images when the legacy Gradio client cannot deserialize its final gallery payload.
+- Safely tracks and stops only the managed Fooocus Python process tree.
+
+## [0.3.1] - 2026-08-23
+
+### Fixed
+- Uploaded images are staged per conversation and sent to vision-capable providers as real multimodal image blocks instead of path-only text.
+- OpenAI-compatible providers receive native `image_url` blocks while Anthropic receives base64 image source blocks.
+- Image Studio can use uploaded paths, generate images, perform prompt-based AI edits, and expose direct downloadable results.
+
+### Changed
+- Replaced the long connector table with compact clickable cards and closable provider-specific detail panels.
+- Added external-service connections for OAuth 2.0 authorization code, username/password, API key, bearer token, and unauthenticated endpoints.
+
+## [0.3.0] - 2026-08-23
+
+### Added
+- `ImageStudio` for local image creation, gradients, text overlays, OpenCV text removal/inpainting, compositing, resizing, cropping, rotation, flipping, conversion, OpenAI-compatible image generation, and conversation downloads
+- `ThreeDStudio` for colored GLB/GLTF/OBJ/STL/PLY primitives, `.mlb` alias correction, Blender 5.2 discovery/verified portable installation, textured materials, HD Eevee/Cycles rendering, BLEND/FBX/USD conversion, and downloadable model/render artifacts
+- Universal `Artifact` publishing for any workspace file and safe ZIP packaging for directories
+- `SystemAdmin` for inventory, CPU/memory/disk/network monitoring, threshold evaluation, process/service control, approved package installs, and downloadable JSON/Markdown reports
+- Secret-redacted UTF-8 JSONL audit logging for every desktop tool invocation, exposed through the admin tool and `/api/audit`
+- `Repository` for GitHub/GitLab-compatible clone, status, log, diff, fetch, pull, and review-branch push without merge behavior
+- `SkillManager` for creating, updating, validating, and packaging project or durable user `SKILL.md` capabilities
+- Searchable/exportable `SharedMemory`, with durable fact and conversation metadata retention expanded beyond the old 40-conversation cap
+- Desktop engine status and direct generated-image/download cards in the conversation
+- `BusinessManager` with durable customers, projects, tasks, invoices, income/expense ledger, dashboards, and downloadable JSON/CSV/HTML reports
+- `FinanceMarkets` for delayed quotes/history, technical indicators, symbol screens, portfolio valuation, and watchlist monitoring reports
+- Optional Alpaca paper/live `Trading` connector configured outside the model context; every submit/cancel requires a fresh approval and audit entry
+- The install wizard now treats the complete Python, Electron, finance, image, system, and Blender stacks as installed dependencies and fails visibly if a required runtime cannot be provisioned
+
+### Fixed
+- Blender 5.2 Eevee engine compatibility and reliable detection of Python tracebacks even when Blender exits with status 0
+- Colored GLB export no longer requires the optional SciPy dependency
+
+## [0.2.9] - 2026-08-23
+
+### Added
+- Compact desktop layout, animated working robot, and a collapsible live workspace preview with a bounded file tree
+- User-selected project/download directory plus automatic installable-output detection and safe source ZIP packaging, exposed as conversation download links
+- Conversation removal from the sidebar without deleting the saved transcript or shared-memory record
+- Prominent Connect tools entry for GitHub, GitLab, MCP servers, and OpenAI-compatible external agents
+- Simultaneous conversation-scoped agent instances with isolated transcripts, workspace/tool state, approvals, tasks, and jobs; the sidebar can cross-view running agents without redirecting their output
+- Read-only `AgentInstances` tool for deliberate cross-conversation context and a real `Terminal` tool that discovers PowerShell, cmd, Bash/Git Bash, WSL, zsh, fish, and nushell while inheriting installed CLIs from PATH
+
+### Fixed
+- Preserved object-shaped `tool_result.content` normalization for Dustin's live Anthropic 400 while restricting the no-tools fallback to the exact legacy tool-schema error; authentication, rate-limit, and unrelated request failures are surfaced normally
+- Combined atomic session writes with explicit UTF-8 reads/writes so non-ASCII chats survive restart on Windows
+- Electron now uses a writable Jonathan-owned profile and an in-process software GPU fallback, preventing Windows profile-lock and missing GPU-runtime crashes
+- PowerShell now executes after user approval instead of returning a placeholder error, and Bash discovers Git Bash on Windows instead of assuming `bash.exe` is on PATH
+
## [0.2.8] - 2026-08-23
### Fixed
-- Windows desktop start no longer dies in `Session.load` with `UnicodeDecodeError` on session JSON. Session files are always opened and written as UTF-8 (`encoding='utf-8'`), so bytes such as `0x8F` that are undefined in cp1252 load correctly
+- Session JSON is explicitly read and written as UTF-8 on Windows
## [0.2.7] - 2026-08-23
### Fixed
-- Anthropic `tool_result.content` is now always a string or a list of content blocks. Dict/object/list tool outputs (Read, Write, Glob, resumed session JSON) are `json.dumps`'d before `chat`, `chat_stream`, and `chat_stream_response`, so Anthropic no longer returns `Found an object, but tool_result content must either be a string or a list of content blocks`
+- Anthropic-compatible message history now JSON-encodes object-shaped `tool_result.content`, including results loaded from older persisted chats, while preserving valid content-block lists
+- Unrelated Anthropic errors such as authentication and rate limits are still surfaced; only the existing exact tool-schema 400 fallback retries without tools
## [0.2.6] - 2026-08-23
@@ -220,6 +412,7 @@ The focus was on building a solid foundation with clean architecture, comprehens
---
+[0.2.9]: https://github.com/GoDeskio/Clawd-Code/releases/tag/v0.2.9
[0.2.8]: https://github.com/GoDeskio/Clawd-Code/releases/tag/v0.2.8
[0.2.7]: https://github.com/GoDeskio/Clawd-Code/releases/tag/v0.2.7
[0.2.6]: https://github.com/GoDeskio/Clawd-Code/releases/tag/v0.2.6
diff --git a/FEATURE_LIST.md b/FEATURE_LIST.md
index 981dd08..390f826 100644
--- a/FEATURE_LIST.md
+++ b/FEATURE_LIST.md
@@ -36,23 +36,47 @@
| 交互式 REPL | ✅ | 支持交互式输出、历史记录、Tab 补全、多行输入 |
| Slash Commands | ✅ | 已支持 `/help`、`/clear`、`/save`、`/load`、`/multiline`、`/exit` |
| 多 Provider 抽象 | ✅ | 已支持 Anthropic / OpenAI / GLM / Minimax / Hugging Face / Local LLM |
+| 本机 AI 自动发现 | ✅ | v0.4.7:扫描 Ollama / LM Studio / Jan / GPT4All / llama.cpp / KoboldCpp / Hugging Face 缓存与常用回环端口;显示模型存储、运行时 RAM、可用系统 RAM 与保守 fit 提示;无可用云配置时自动启动并连接已安装模型 |
+| Jonathan 本地图像引擎 | ✅ | v0.4.6:内置 Diffusers 文生图/图生图;自动安装依赖、首次下载模型、CUDA/CPU、输出下载与 Image Studio 本地回退;不依赖 Fooocus/Gradio |
+| Audio Studio | ✅ | v0.4.6:内置原创程序化 WAV 生成、裁剪、标准化、混音和会话下载;重型歌曲模型可通过显式外部/MCP 连接使用 |
+| Voice / Speech Studio | ✅ | v0.4.6:会话内用户主动麦克风录音、本地 Whisper 系列多语言转写/翻译、首次使用下载所选模型、本机语音合成与 WAV 下载;不后台监听、不要求付费 API |
| Provider 配置管理 | ✅ | 支持默认 Provider、Base URL、默认模型配置 |
| 会话持久化 | ✅ | 支持保存/加载本地会话;重启后侧栏仍在;New Chat 新建空线程;桌面左侧可重命名 |
-| 独立桌面 Agent | ✅ | v0.2.8:会话 JSON 以 UTF-8 读写(避免 Windows cp1252);Anthropic tool_result 对象序列化为字符串;会话持久化;New Chat / `/new` 空线程;跨会话记忆;内部 Workers;schema 400 去掉 tools 重试;首次启动自动补依赖;Setup 原地升级且不合并 main |
+| 独立桌面 Agent | ✅ | v0.3.0:每个对话拥有隔离且可并行运行的 Agent;完整终端与 PATH CLI;图像生成/编辑/文字移除;Blender 高精度材质 3D 与 GLB/GLTF/OBJ/STL/FBX/BLEND/USD;任意文件下载;系统监控、资源评估、进程/服务/包管理;UTF-8 脱敏审计日志;GitHub/GitLab 仓库自动化;自建 SKILL.md;可搜索共享记忆;原子 UTF-8 会话;New Chat 空线程;实时预览;Setup 原地升级且不合并 main |
+| 可验证内部协作 | ✅ | v0.4.6:Fast/Balanced/Verified 三种显式模式;Verified 由未参与创作的独立 Agent 检查目标、冲突和缺失证据;所有 worker/reviewer token 计入所属会话且不构成配额 |
+| 长程目标控制 | ✅ | v0.4.7:跨会话持久目标、工作区 scope、依赖 todo、Agent claim/lease、人工/证据 gate、UTF-8 evidence receipt、handoff、turn budget 与 fail-closed continuation;无需第二 Agent runtime |
+| ECC 技能与 Agent 目录 | ✅ | v0.4.6:完整目录同步、MIT 来源/版本记录、导入前安全扫描、全部离线索引、按需启用避免上下文膨胀、可实例化 Agent 模板与 Memory Vault 互操作 |
+| 自动技能学习 | ✅ | v0.4.6:本地 Git 历史测量、项目隔离、敏感信息/提示注入清洗、置信度 instinct、UTF-8 SKILL.md 草稿、用户审核后才激活 |
+| 写作润色技能 | ✅ | v0.4.6:可选的表层与结构双重编辑、样本文风校准、含义/事实/引用保留,不承诺规避检测器或虚假作者身份 |
+| 界面艺术指导技能 | ✅ | v0.4.6:按产品语境设置布局变化、动效强度和信息密度,保留可访问性、响应式和实际渲染验证 |
+| React 性能审查技能 | ✅ | v0.4.6:以测量为先检查请求瀑布、客户端包、渲染边界、重复渲染和加载体验,并验证前后结果 |
+| 截图转可用界面技能 | ✅ | v0.4.6:读取会话中的截图/设计图,遵循项目现有技术栈生成可维护界面,使用项目预览进行桌面/窄屏渲染对比与验证;无需独立截图转代码服务 |
+| 本地市场预测 | ✅ | v0.4.6:Jonathan 内置 EWMA/Monte Carlo OHLCV 概率预测、Mini/Small/Base 路径规模、P10/P50/P90 CSV、无外部源码/模型、无自动下单 |
+| 私有个人财务库 | ✅ | v0.4.6:Jonathan 内置 SQLite 账户/交易/分类/周期项目/预算/目标/资产/多币种汇总、无容器、无外部应用、无付费要求 |
+| 本地代码图与代码记忆 | ✅ | v0.4.6:Jonathan 内置 Python/SQLite、增量扫描、文本/引用/解释/依赖路径、无 Claude hook、Node、外部仓库、云端或订阅 |
+| 可编辑图形与角色 | ✅ | v0.4.6:内置 Raster→SVG/PPTX/OCR/JSON、确定性动画 SVG 角色和 rig 配方,无外部模型或仓库 |
+| 追加式运行事实 | ✅ | v0.4.6:每会话 UTF-8 JSONL、秘密脱敏、SHA-256 哈希链、任务/工具/权限/完成/失败/usage 可恢复事实;上下文裁剪不删除历史证据 |
+| 赢得式自主权 | ✅ | v0.4.6:按会话/工具记录互不重复的人工批准与拒绝,使用 Wilson 置信下界和风险上限提出透明建议;建议本身绝不扩大权限,系统/终端/写入仍由用户批准控制 |
| 工具 schema 清洗 | ✅ | 发给 Anthropic/OpenAI 的每个工具都带 `input_schema.type` |
| 会话消息管理 | ✅ | 支持会话历史维护与序列化 |
| 错误恢复 / 重新登录 | 🟡 | 已有基础认证错误处理与重新配置流程 |
| Token / Cost 跟踪 | ✅ | 桌面每个聊天窗口显示 input/output/running total,随会话持久化;仅信息展示,不是配额墙 |
| GitHub / GitLab | ✅ | 本机 token 或设备登录;clone/pull/push、建仓、PR/MR;默认不推 default branch |
| MCP / 其他 Agent | ✅ | 设置中添加/列出/启用 MCP 与 OpenAI 兼容 agent URL;Cursor/Codex/local hook |
+| 公共 API 发现 | ✅ | v0.4.6:内置离线审核起始目录,按类别/认证/HTTPS/CORS 搜索,经批准转换为持久连接器;不依赖远程目录仓库 |
+| 公共内容 Reach | ✅ | v0.4.6:内置频道健康状态、RSS/Atom 阅读、YouTube 公开元数据与字幕;认证社交平台仍走显式连接器,不自动读取浏览器 Cookie |
| 上下文构建 | ✅ | workspace / git / `CLAUDE.md`;跨会话记忆 `~/.clawd/memory`(facts + 其他对话标题/摘要,不含全文) |
+| DESIGN.md 设计上下文 | ✅ | v0.4.6:项目根目录 UTF-8 `DESIGN.md` 以独立、有界设计系统段自动提供给每个会话 Agent |
| Claude Code Agent Loop | ✅ | 已实现 agent_loop.py,支持工具调用循环 |
| `/resume` 会话恢复体验 | 🚫 | 暂无独立恢复流程与 UI |
| `/compact` 对话压缩 | 🚫 | 暂无自动/手动压缩能力 |
| `/doctor` 诊断系统 | 🚫 | 暂无环境、配置、权限、依赖诊断命令 |
| Hook 系统 | 🟡 | 已有 Cursor/Codex/local agent hook 与 inbound POST;非通用 pre/post tool hook |
| 权限系统 | ✅ | 路径沙箱 + 文档写入询问 + 桌面端对 Bash/Write/Edit/Web 的交互批准 |
-| 桌面应用 | ✅ | Electron/浏览器壳 + 本地 Python host;Windows 上为 JonathanAi.exe 与玻璃拟态仪表盘 |
+| 工具执行网关 | ✅ | v0.4.6:允许的工具动作在执行副作用前写入脱敏 preflight 审计记录,完成/拒绝/失败后再写结果;preflight 无法持久化时关闭执行 |
+| 自托管服务操作 | ✅ | v0.4.7:持久注册 Docker/Podman Compose、Windows service 与 systemd;固定参数、无 shell 拼接;状态/log、LAN health、可下载 backup、审批后 start/stop/restart/update;配套可见 operator skill |
+| 远程电脑与服务器 | ✅ | v0.4.8:对明确主机执行审批门控的 SSH/WinRM 命令、SCP 上传/下载和 TCP 可达性检查;仅使用现有密钥/系统凭据,严格验证主机,无密码模型输入,不能绕过远端 ACL/UAC/sudo/防火墙 |
+| 桌面应用 | ✅ | v0.4.8:修复持久对话渲染;SSE 中断时自动切换到 job-event 轮询并在完成后重载保存的回答;Electron/浏览器壳 + 本地 Python host |
| 安装向导 | ✅ | Windows Setup exe 可见向导(Next/Install/Finish)+ Desktop/开始菜单快捷方式;默认源码目录 `~/Jonathan/Jonathan-Ai` |
| 自更新 | ✅ | 仅从 GoDeskio/Clawd-Code 检查并快进更新 |
diff --git a/JonathanAi-0.4.8-source.zip b/JonathanAi-0.4.8-source.zip
new file mode 100644
index 0000000..677bf57
Binary files /dev/null and b/JonathanAi-0.4.8-source.zip differ
diff --git a/JonathanAi-Setup.exe b/JonathanAi-Setup.exe
new file mode 100644
index 0000000..beda498
Binary files /dev/null and b/JonathanAi-Setup.exe differ
diff --git a/JonathanAi.exe b/JonathanAi.exe
new file mode 100644
index 0000000..eb177dc
Binary files /dev/null and b/JonathanAi.exe differ
diff --git a/MANIFEST.in b/MANIFEST.in
index c5d58e2..b7286ff 100644
--- a/MANIFEST.in
+++ b/MANIFEST.in
@@ -1,12 +1,14 @@
include README.md
include VERSION
include LICENSE
+include THIRD_PARTY_NOTICES.md
include CLAUDE.md
include MVP_PLAN.md
recursive-include src *.py
recursive-include src *.json
recursive-include src/desktop/web *
recursive-include src/install/web *
+recursive-include src/skills/bundled *
include desktop/package.json
include desktop/main.js
include desktop/preload.js
diff --git a/README.md b/README.md
index e7da0e4..84066a8 100644
--- a/README.md
+++ b/README.md
@@ -6,14 +6,64 @@
**A local desktop and CLI agent, built as a Python reimplementation of Claude Code**
-**Version 0.2.8** — Jonathan Ai is itself the AI agent. Chats persist on disk and survive restart. New Chat starts a blank thread. Shared memory (`~/.clawd/memory`) lets facts from one chat inform another. Internal workers can plan a goal in parallel without Cursor, Codex, or MCP. Session JSON is always UTF-8 (Windows no longer decodes as cp1252). Anthropic tool results that are objects are stringified so resumed chats do not 400. A first Anthropic chat — including desktop streaming — retries without tools if a schema 400 still appears. First launch repairs the venv and pip packages automatically. The Windows launcher finds `%USERPROFILE%\Jonathan\Jonathan-Ai`.
+**Version 0.4.8** — Jonathan Ai's conversation window now reliably renders persisted user and assistant messages. If Windows or the embedded browser aborts the live response stream, the UI automatically resumes from the bounded job-events endpoint and reloads the saved transcript at completion instead of silently losing the answer.
+
+`RemoteTrigger` is now a real approval-gated remote-computer tool rather than a placeholder. It can probe an explicitly named host, execute commands through SSH or Windows Remote Management, and transfer files through SCP using existing OS credentials or SSH keys. It does not accept passwords in model input, defaults to strict SSH host-key verification, and cannot bypass the remote machine's authentication, ACL, firewall, UAC or sudo rules.
+
+Jonathan also retains native durable control for long-running work. `LongHorizonControl` keeps objectives, workspace scope, bounded turns, dependency-aware todos, agent claims/leases, human gates, evidence receipts, and handoffs in atomic UTF-8 state shared across conversations. Its continuation and closure decisions fail closed when judgment, evidence, prerequisites, or budget are missing.
+
+Local-model discovery now also reports on-disk model size, live runtime RAM, available system RAM, and conservative fit guidance before a large model is selected. The measurements are local and informational; Jonathan does not silently stop another runtime or send hardware details to a service.
+
+Every conversation also writes a redacted, tamper-evident append-only runtime ledger under `~/.clawd/events`. Provider context can stay bounded for speed without deleting job, tool, permission, completion, usage, or artifact evidence. Token counts remain persisted per conversation and agent and are informational only—there is no quota, payment, or purchase path.
+
+When a project contains `DESIGN.md`, Jonathan loads its UTF-8 visual system into a separate bounded context section for every conversation agent. This keeps project colors, typography, components, layout, responsive rules, and design constraints persistent without copying a third-party brand catalog into the application.
+
+The Device & System card also shows an earned-autonomy profile for each conversation and gated tool. It counts only distinct human decisions, uses a conservative Wilson confidence bound, and caps terminal, write, device, repository, and trading actions at approval-required. Recommendations are visible evidence—not authority—and can never enable a tool or full-device access without the user's explicit grant.
+
+Tool execution is audit-gated: after policy and user approval, Jonathan writes a secret-redacted `started` record before invoking the tool and a second record with success, refusal, or failure. If the preflight audit cannot be persisted, the side effect does not run.
+
+The 0.4.8 shell uses a quiet, neutral desktop system: compact native typography, solid surfaces, low-decoration hierarchy, consistent radii, clear focus rings, semantic state colors, and reduced-motion support. It keeps the dense operational layout instead of applying oversized website spacing.
+
+Pydantic AI was evaluated for typed outputs, broad model adapters, durable execution, evaluations, and telemetry. Jonathan keeps its existing single runtime for this release; adding a second agent loop would duplicate session, permission, provider, and tool state. Its strongest capabilities remain candidates for narrow future adapters rather than a wholesale dependency.
+
+The Workers control now offers **Fast**, **Balanced**, and **Verified** modes. Verified uses an additional independent model pass to challenge the worker outputs against the original goal and synthesize the final response; Fast avoids that latency for small jobs. These modes are explicit because deeper orchestration costs more time and provider tokens, and every worker/reviewer token is added to the owning conversation's informational total.
+
+### Media, 3D, and administrator tools
+
+- `ImageStudio`: create and convert PNG/JPEG/WebP/BMP/GIF/TIFF/PDF, add/remove text, inpaint, composite, crop, resize, rotate, flip, or call a configured OpenAI-compatible image endpoint.
+- `ThreeDStudio`: make colored GLB/GLTF/OBJ/STL/PLY assets locally; use Blender for textures, BLEND/FBX/USD, advanced conversion, and high-definition Eevee/Cycles renders. The common `MLB` typo is accepted and corrected to `.glb`.
+- `Artifact`: publish any file to the conversation download shelf, or safely ZIP a directory.
+- `SystemAdmin`: inventory, monitor, evaluate, report, control approved processes/services, and install approved packages. Every tool action is written to a secret-redacted audit ledger at `~/.clawd/audit/actions.jsonl`.
+- `DeviceControl`, `Terminal`, `PowerShell`, and `RemoteTrigger`: after explicit approval, launch installed applications, automate the signed-in desktop, run installed shells/CLIs, and administer explicitly named SSH/WinRM computers. Full-device mode expands local path scope but never bypasses Windows ACL/UAC or another computer's credentials and firewall.
+- `Repository`: inspect, clone, fetch, pull, and push GitHub/GitLab repositories without merging; protected branches require explicit authorization.
+- `AudioStudio`: generate deterministic original instrumental WAV beds and trim, normalize, or mix local PCM WAV files into conversation downloads without an account or cloud service.
+- `SkillManager` and `SharedMemory`: create/validate/package reusable `SKILL.md` files and search/export durable cross-conversation memory.
+- `LongHorizonControl`: coordinate durable goals across separate conversation agents with bounded turns, dependency-aware work, leases, judgment gates, evidence receipts, handoffs, and explicit continuation checks.
+- `SelfHostedService`: register and operate approved Docker/Podman Compose, Windows-service, or systemd applications; inspect status/logs, create downloadable backups, check local/LAN health, and run lifecycle/update actions without shell interpolation. The `self-hosted-service-operator` skill is visible in Jonathan's skill list.
+- `ECCIntegration`: synchronize, audit, import, update, enable/disable, and search ECC skills/agent templates; exchange unreviewed Memory Vault documents; and create reviewable skills from measured Git history.
+- `BusinessManager`: durable local customers, projects, tasks, invoices, income/expense ledger, dashboard, and downloadable JSON/CSV/HTML business reports.
+- `KronosForecast`: create built-in Mini/Small/Base probabilistic EWMA/Monte Carlo OHLCV research forecasts as downloadable CSV files; never places an order.
+- `PersonalFinanceVault`: manage built-in SQLite accounts, transactions, categories, recurring items, budgets, goals, assets, reports, and currencies; no container, subscription, or payment is required.
+- `CodeMemory`: build an incremental local code graph, search durable notes, find symbol usages, explain dependencies, and trace paths through the project using a Jonathan-owned SQLite database with no cloud account or Claude hooks.
+- `Procoder`: run Jonathan's built-in fail-closed changed-file gate, detected project tests, secret/conflict hygiene, release readiness, and related engineering reports; it never installs hooks or edits the project.
+- `EditableGraphics`: vectorize an uploaded raster diagram into editable SVG, native PowerPoint shapes, OCR text, a review PNG, and a JSON package with no separate model pack.
+- `CharacterStudio`: generate deterministic animated SVG character rigs and editable JSON recipes from a seed without a model or API.
+- `FinanceMarkets`: current/delayed quotes, price history, RSI/moving averages, multi-symbol screens, portfolio valuation, and timed watchlist reports.
+- `Trading`: optional Alpaca paper/live brokerage connection. Credentials are entered directly in Connect Tools; paper is the default, and every order submission or cancellation requires a fresh approval.
+- `PublicApiCatalog`: search Jonathan's audited offline starter index by category, authentication, HTTPS, or CORS, inspect the provider's documentation, and—with approval—create a persistent connector card. Provider terms, credentials, quotas, and data policies still apply.
+- `writing-polish` skill: opt-in two-pass editing for surface clarity and document structure, calibrated to supplied voice samples while preserving facts, citations, uncertainty, and honest authorship.
+- `interface-art-direction` skill: set deliberate layout-variance, motion-intensity, and information-density controls for new interfaces or evidence-based redesigns, with accessibility and responsive verification built in.
+- `react-performance-review` skill: diagnose request waterfalls, client bundle cost, rendering boundaries, rerenders, expensive hot paths, and loading behavior using reproducible before/after evidence.
+- `screenshot-to-interface` skill: turn an uploaded screenshot or mockup into maintainable code in the project's existing stack, then render and compare desktop/narrow previews before packaging the result.
+- `SpeechStudio`: record only when the user clicks Voice, transcribe/translate locally with a Whisper-family model, and synthesize responses to downloadable WAV with the device voice engine.
+- `ContentReach`: check public channel readiness, read RSS/Atom feeds, and retrieve public YouTube metadata/captions without silently reusing browser cookies.
*From TypeScript Source → Rebuilt in Python with ❤️*
***
-[](https://github.com/GPT-AGI/Clawd-Code/stargazers)
-[](https://github.com/GPT-AGI/Clawd-Code/network/members)
+[](https://github.com/GoDeskio/Clawd-Code/stargazers)
+[](https://github.com/GoDeskio/Clawd-Code/network/members)
[](https://opensource.org/licenses/MIT)
[](https://www.python.org/downloads/)
@@ -193,7 +243,7 @@ clawd desktop # Start the desktop host
The first-run wizard installs everything needed to run the desktop agent: it detects the OS, saves the full source tree under a **Jonathan** folder, creates a Python venv, installs backend and desktop-shell dependencies, writes provider config placeholders (no API keys), and verifies the agent can start a session.
-**Chat on first run with only a local model or one API key.** After install, open Jonathan Ai, pick Anthropic / OpenAI / GLM / Hugging Face / Local LLM, and send a message. MCP servers, Cursor, Codex, and other agents are optional — they are not required and are omitted from the provider request until you connect one. Tokens stay on this machine. The product version is **0.2.8** (see the `VERSION` file, UI header, and Windows installer).
+**Chat on first run with only a local model or one API key.** After install, Jonathan first discovers installed local runtimes and connects a reachable model automatically when no working provider exists. The installer prepares the built-in code memory, forecasting, engineering-gate, finance, editable-graphics, and long-horizon control engines; only the audited ECC catalog needs an optional online source sync. You can pick Anthropic / OpenAI / GLM / Hugging Face / Local LLM and send a message. MCP servers, Cursor, Codex, and other agents are optional — they are omitted from provider requests until connected. Tokens stay on this machine. The product version is **0.4.8**.
**Windows (real desktop app):**
@@ -337,7 +387,11 @@ Treat any OpenAI-compatible server as the **Local LLM** provider:
| Hugging Face TGI (self-hosted) | `http://127.0.0.1:3000/v1` |
| Custom | loopback or LAN URL + optional API key |
-On first run and in settings, **Scan local ports** probes those addresses, lists available models, and lets you pick a default. Custom URLs that resolve to the public internet are rejected. Local endpoints are never bound or advertised on the WAN by Jonathan Ai.
+On first run and in settings, **Scan local ports** also inventories installed runtimes and common model stores, starts a known Ollama/LM Studio service when needed, lists available models, and connects the first reachable model. Custom URLs that resolve to the public internet are rejected. Local endpoints are never bound or advertised on the WAN by Jonathan Ai. Credentials are scoped to their endpoint, so an Ollama credential is not sent to LM Studio.
+
+#### Jonathan local image generation
+
+Open **Connect tools → Local diffusion** to verify Jonathan's built-in Diffusers engine, generate or edit an image from an uploaded source, and publish outputs directly into the conversation. The installer supplies its Python packages; the first generation downloads the configured model (`stabilityai/sdxl-turbo` by default) into Jonathan's private cache. It uses CUDA when available and otherwise runs on CPU. Fooocus informed the product requirements, but Jonathan does not clone, launch, or depend on Fooocus, Gradio, or an upstream service.
#### GitHub and GitLab
@@ -387,7 +441,7 @@ python -m src.desktop --port 8765
This binds `http://127.0.0.1:8765/` only, serves the chat UI, and opens a browser. Use `--no-browser` in CI.
-The dashboard is a glassmorphism prompt UI: conversations on the left (title + last activity; **double-click or right-click to rename**), frosted chat cards, and a monochrome + eye-glow palette matching the robot sketch. The header shows **v0.2.8** and the token meter (informational only). Conversations persist in the sidebar; New Chat is always empty. Shared memory is local only (`~/.clawd/memory`). Jonathan Ai is the agent — no second AI connection is required.
+The dashboard is a compact prompt UI: conversations on the left (title + last activity; **double-click or right-click to rename**), clear chat cards, and a monochrome + eye-glow palette matching the robot sketch. The header shows **v0.4.8** and the token meter (informational only). Conversations persist in the sidebar; New Chat is always empty. Shared memory is local only (`~/.clawd/memory`), while durable goal governance is stored under `~/.clawd/goals`. User-authored skills are visible under `Jonathan-Ai/Skills`; managed ECC content is stored below `Skills/.managed/ecc` and only enabled entries are loaded. Jonathan Ai is the agent — no second AI connection is required.
**Option B — Electron desktop shell (tray, notifications, folder picker)**
@@ -414,8 +468,7 @@ Desktop extras on top of the CLI:
- **New Chat** always opens a brand-new empty conversation (new id, empty transcript, reset token meter). Previous chats stay in the sidebar and survive restart.
- Shared agent memory in `~/.clawd/memory` (facts + compact titles/summaries of other chats). Threads stay isolated; the model only sees a short brief, not full dumps.
- Standalone agent: send message → stream reply → show tools → new chat → rename chat → informational token meter. **Workers** plans a goal into isolated internal workers (shared memory only). No required Cursor/Codex/MCP connection.
-- Windows session JSON (0.2.8): `Session.load` / `save` always use `encoding='utf-8'` so a UTF-8 byte such as `0x8F` no longer crashes desktop start under cp1252.
-- Anthropic 400 fix (0.2.7): `tool_result.content` that is a dict/object/list is `json.dumps`'d before `chat`, `chat_stream`, and `chat_stream_response`, including resumed sessions that already stored object-shaped tool results. Schema 400s (`tools.N.custom.input_schema.type`) still retry the same turn without tools.
+- Anthropic 400 fix (0.2.7): `tool_result.content` dicts and ordinary lists are JSON-encoded before `chat`, `chat_stream`, and `chat_stream_response`, including old persisted sessions. Valid content-block lists are preserved. Unrelated 400/401/429 errors are not hidden by the tool-schema fallback.
- Anthropic 400 fix (0.2.6): desktop streaming (`chat_stream_response`) and `chat()` send only `{name, description, input_schema:{type:object, properties}}`, including MCP/dynamic and custom-wrapped tools. If Anthropic still returns `tools.N.custom.input_schema.type: Field required`, the same turn retries with tools omitted so the first message still answers. First launch auto-installs missing venv/pip/Electron when possible. `/new` always opens a new empty session. Setup never merges `main` into this branch.
- Workspace/folder picker (Electron dialog, or a path prompt in the browser)
- Approve / deny / always-allow-this-session permission prompts
@@ -489,6 +542,36 @@ Example:
- Arguments: use `$ARGUMENTS`, `$0`, `$1`, or named args like `$path` (from `arguments`).
- Placeholder syntax: use `$path`, not `${path}`.
+#### ECC catalog and automatic skill learning
+
+Open **Skills library → ECC catalog** to synchronize the official `affaan-m/ECC` repository, run a full security scan, import the complete skills/agent-template catalog, and enable or disable individual skills. Jonathan records the exact upstream revision and MIT provenance. The complete catalog remains on disk, but only enabled entries are parsed into agent context; this avoids duplicate hooks and large prompt/startup costs.
+
+#### Native local market forecasting
+
+Open **Connect tools → Market forecasts** or use `KronosForecast` with a timestamped CSV containing a close column (OHLCV columns are accepted). Jonathan's built-in engine estimates recency-weighted log-return drift and volatility, generates bounded Monte Carlo paths, and exports median/P10/P90 forecast CSVs. It requires no cloned repository or model download. Results are research artifacts, not investment advice or automatic trading signals; validate them with walk-forward tests, transaction costs, slippage, portfolio constraints, and independent risk controls.
+
+#### Private personal finance vault
+
+Open **Connect tools → Personal finance vault** or use `PersonalFinanceVault`. Jonathan stores accounts, transactions, categories, recurring flags, budgets, savings goals, assets, and currency-separated totals in `~/.clawd/finance/personal-finance.sqlite`. It is an embedded UTF-8/WAL SQLite engine: no Docker, remote service, bank credential, subscription, or cloned finance application is required.
+
+#### Local code graph and durable code memory
+
+Open **Connect tools → Code memory** or use `CodeMemory`. Jonathan's built-in Python/SQLite engine incrementally scans supported source files, indexes symbols/calls/imports, performs live text search, and stores only explicitly requested notes in `~/.clawd/code_memory/jonathan-code-memory.sqlite`. No Node helper, Claude installer, hook, cloned repository, remote database, or account is used. Existing UTF-8 chat/session and shared-memory systems remain authoritative.
+
+Benjamin Plus was evaluated and adopted as a compact native workflow discipline in Jonathan's system prompt. Fullstack Agent was evaluated but not embedded because it is a setup script for a separate Claude/Obsidian stack. Procoder, DrawAI, Kindergrimm, and Scroll Craft were used only as licensed design references: their useful capabilities were rewritten as Jonathan's built-in engineering gate, editable-graphics engine, procedural character studio, and scroll-storytelling skill.
+
+OmniRoute informed Jonathan's in-process transient retry/circuit-breaker layer, but no gateway or cross-provider auto-fallback is installed: changing providers remains explicit so prompts, credentials, privacy boundaries, and possible charges cannot move unexpectedly. Task Observer informed the correction/failure/repeated-workflow review queue in automated skill learning; observations are sanitized, confidence scored, and never silently activated. Anthropic's Claude plugin catalog was evaluated, but its Claude-specific marketplace and hooks duplicate Jonathan's audited `SKILL.md`, MCP, connector, and agent systems and are not bundled.
+
+Headroom's provider-cache analysis informed a native Anthropic prompt-cache marker on Jonathan's system prefix. Jonathan does not install Headroom, route traffic through a proxy, compress user intent, discard turns, or add a retrieval service; the persistent UTF-8 transcript remains authoritative.
+
+Anthropic's community plugin marketplace was also evaluated. It is a directory of third-party sources rather than an application engine, and its entries have independent licenses, credentials, hooks, and runtime requirements. Jonathan does not bulk-install it; users connect MCP/external tools explicitly and enable reviewed skills individually.
+
+YuE and ACE-Step UI were evaluated for full-song generation. Both are valuable optional GPU systems, but their multi-gigabyte model stacks and hardware/runtime requirements would make a standard desktop install slower and less reliable. Jonathan therefore ships an always-available native `AudioStudio` baseline for original procedural WAV generation and editing. Users with a separately approved local music server can connect it through Jonathan's existing external/MCP connector; no cloned music UI is required.
+
+**Learn from project** analyzes up to 200 local Git commits. It measures commit prefixes, frequently changed areas, file types, co-change pairs, and test placement. Repository text and commit messages are sanitized as untrusted evidence, possible secrets are redacted, and the result is saved as a confidence-scored draft. Review the draft in the editor and click **Validate & save** before it becomes active. Jonathan never silently overwrites a user skill or automatically turns a draft into executable policy.
+
+Memory Vault export writes inspectable `ecc.memory.v1` Markdown under `~/.clawd/memory/vault`. Imported vault entries remain `unreviewed` context, reject credential/instruction-override patterns, and do not grant tools or permissions.
+
***
@@ -622,14 +705,14 @@ If you find this useful, please **star** ⭐ the repo!
**本地桌面与 CLI Agent,基于真实 Claude Code 源码的 Python 重实现**
-**版本 0.2.8** — Jonathan Ai 本身就是 AI Agent。会话会持久化;New Chat 总是空线程。跨会话记忆只存在本机 `~/.clawd/memory`。MCP / Cursor / Codex 均为可选。会话 JSON 始终按 UTF-8 读写,避免 Windows cp1252 解码失败。对象形状的 tool_result 会序列化为字符串,避免 Anthropic 400。若仍因工具 schema 返回 400,同一轮会去掉 tools 重试。Windows 启动器会在 `%USERPROFILE%\Jonathan\Jonathan-Ai` 找到 venv;若提示找不到 Electron/venv,请再运行 Setup 原地升级。
+**版本 0.3.0** — Jonathan Ai 本身就是 AI Agent。会话会持久化;New Chat 总是空线程。跨会话记忆只存在本机 `~/.clawd/memory`。MCP / Cursor / Codex 均为可选。新增图像、Blender 3D、系统管理、审计、仓库和技能工具。Windows 会话使用原子 UTF-8 持久化。启动器会在 `%USERPROFILE%\Jonathan\Jonathan-Ai` 找到 venv。
*从 TypeScript 源码 → 用 Python 重建 ❤️*
***
-[](https://github.com/GPT-AGI/Clawd-Code/stargazers)
-[](https://github.com/GPT-AGI/Clawd-Code/network/members)
+[](https://github.com/GoDeskio/Clawd-Code/stargazers)
+[](https://github.com/GoDeskio/Clawd-Code/network/members)
[](https://opensource.org/licenses/MIT)
[](https://www.python.org/downloads/)
@@ -811,7 +894,7 @@ clawd desktop # 启动桌面 host
### 安装
```bash
-git clone https://github.com/GPT-AGI/Clawd-Code.git
+git clone https://github.com/GoDeskio/Clawd-Code.git
cd Clawd-Code
# 创建虚拟环境(推荐使用 uv)
@@ -881,7 +964,7 @@ Windows:双击 `packaging/windows/bin/JonathanAi-Setup.exe`,向导点 Next /
./install.sh --yes
# 默认源码目录:~/Jonathan/Jonathan-Ai
# 只从 https://github.com/GoDeskio/Clawd-Code 安装与更新
-# 版本 0.2.8:会话 JSON 用 UTF-8;对象 tool_result 序列化为字符串;会话持久化;/new 与 New Chat 均为空线程;跨会话记忆在 ~/.clawd/memory;首次启动自动补依赖;Setup 原地升级且不合并 main
+# 版本 0.3.0:图像与 Blender 3D、系统管理与审计、仓库自动化、自建技能、持久共享记忆;Setup 原地升级且不合并 main
# 安装后可在向导或桌面设置中连接 Hugging Face / 本地 LLM / GitHub / GitLab / MCP,无需重装
# Token 只保存在 ~/.clawd/config.json,不会写入安装包或 git
# 聊天窗口的 token 计数只做展示,不是付费墙
@@ -903,7 +986,7 @@ python -m src.cli desktop # Python host + 浏览器 UI
cd desktop && npm install && npm start # Electron 壳
```
-密钥不会写入 Git。桌面端复用现有 agent loop、工具、skills 与会话。Jonathan Ai 本身就是 Agent:发送消息、流式回复、显示工具、新建/重命名会话、信息性 token 计数、provider 设置。不需要第二个 AI 连接。头部显示版本 **0.2.8**。内部 Workers 可并行拆任务,不需要其它 Agent。桌面流式请求若遇 Anthropic schema 400 会去掉 tools 重试。对象 tool_result 会先转成字符串。会话会留在侧栏,New Chat 总是新的空对话。若桌面快捷方式仍指向上级 Jonathan 文件夹,请再运行 Setup。
+密钥不会写入 Git。桌面端复用现有 agent loop、工具、skills 与会话。Jonathan Ai 本身就是 Agent:发送消息、流式回复、显示工具、新建/重命名会话、信息性 token 计数、provider 设置。不需要第二个 AI 连接。头部显示版本 **0.3.0**。内部 Workers 可并行拆任务,不需要其它 Agent。会话会留在侧栏,New Chat 总是新的空对话。
***
diff --git a/THIRD_PARTY_NOTICES.md b/THIRD_PARTY_NOTICES.md
new file mode 100644
index 0000000..88beb36
--- /dev/null
+++ b/THIRD_PARTY_NOTICES.md
@@ -0,0 +1,584 @@
+# Third-party notices
+
+Jonathan Ai includes or adapts selected permissively licensed reference material.
+Product and company names are not used as Jonathan branding; attribution below is
+retained because the licenses require it.
+
+## LoopX design reference
+
+Jonathan's independent `LongHorizonControl` was informed by the durable-state,
+bounded-turn, judgment-gate, claim/lease and evidence-receipt architecture in
+`huangruiteng/loopx`, offered under Apache-2.0 and MIT licenses. Jonathan does
+not include or require upstream source, prompts, adapters, dashboard, PWA,
+Tauri shell, branding, package, service or runtime. See
+https://github.com/huangruiteng/loopx.
+
+## Scraping API catalog evaluation
+
+`cporter202/scraping-apis-for-devs` was evaluated as a discovery source. It is
+a large directory of changing third-party offerings—predominantly hosted
+Apify actors—and does not show a top-level software license. Jonathan does not
+copy the catalog, descriptions, links, branding, actors, credential flows, or
+vendor dependencies. Safe API discovery remains in Jonathan's audited native
+catalog and explicit connectors. See
+https://github.com/cporter202/scraping-apis-for-devs.
+
+## OGAM design reference
+
+Jonathan's local-model resource display was informed by the visible model
+residency and memory-fit concepts in `off-grid-ai/OGAM`, licensed under MIT.
+Jonathan independently measures its already supported desktop runtimes and
+does not include upstream React Native code, Android/iOS native modules,
+models, mobile application, paid Pro features, branding, service or runtime.
+See https://github.com/off-grid-ai/OGAM.
+
+## Awesome Selfhosted design reference
+
+The service-discovery and operator workflow at
+`awesome-selfhosted/awesome-selfhosted`, licensed CC BY-SA 3.0, was evaluated.
+Jonathan independently implements a generic lifecycle tool and concise
+operator skill; it does not copy or bundle the upstream catalog, descriptions,
+non-free list, logos, branding, generated site, or any listed application.
+Each discovered service remains subject to its own current license and official
+deployment documentation. See
+https://github.com/awesome-selfhosted/awesome-selfhosted.
+
+## Local diffusion design reference
+
+Jonathan's local image requirements were informed by `lllyasviel/Fooocus`,
+licensed under GPL-3.0. Jonathan contains an independent Diffusers engine and
+does not include or require Fooocus source, Gradio, checkpoints, branding, or
+runtime. See https://github.com/lllyasviel/Fooocus.
+
+## Provider resilience design reference
+
+Jonathan's independent selected-provider retry and circuit-breaker layer was
+added after reviewing the local gateway/resilience model in
+`diegosouzapw/OmniRoute`, licensed under MIT. No OmniRoute source, gateway,
+provider catalog, Node runtime, dashboard, account, or service is included or
+required. See https://github.com/diegosouzapw/OmniRoute.
+
+## Task-observation design reference
+
+Jonathan's independent friction-evidence review queue was informed by Task
+Observer / `rebelytics/one-skill-to-rule-them-all`, created by Eoghan Henn
+(rebelytics.com) and licensed CC BY 4.0. No upstream skill text, reference
+bundle, Claude hooks, or runtime is included or required. See
+https://github.com/rebelytics/one-skill-to-rule-them-all.
+
+## Claude plugin catalog evaluation
+
+The `anthropics/claude-plugins-official` catalog was evaluated for portable
+skill and MCP conventions. No catalog, plugin, hook, command, agent, MCP
+server, marketplace runtime, or third-party credential is included or
+automatically enabled; individual catalog entries carry their own licenses.
+See https://github.com/anthropics/claude-plugins-official.
+
+## Context-cache design reference
+
+Jonathan's independent Anthropic stable-prefix cache marker was added after
+reviewing the context/cache architecture documented by
+`headroomlabs-ai/headroom`, licensed under Apache-2.0. No Headroom source,
+proxy, Rust extension, ML model, MCP server, compressor, account, or runtime is
+included or required. See https://github.com/headroomlabs-ai/headroom.
+
+## Community plugin catalog evaluation
+
+The Apache-2.0 index at `anthropics/claude-plugins-community` was evaluated.
+No marketplace index, community plugin, hook, command, agent, MCP server,
+credential, or third-party runtime is included or automatically enabled;
+individual entries have separate licenses and trust requirements. See
+https://github.com/anthropics/claude-plugins-community.
+
+## Generative-music evaluations
+
+The Apache-2.0 YuE model/repository and MIT `fspecii/ace-step-ui` were evaluated
+for local full-song workflows. Jonathan's procedural WAV engine is an
+independent implementation and includes no YuE or ACE-Step code, weights,
+frontend, Gradio service, FFmpeg/Demucs bundle, branding, or runtime. See
+https://github.com/multimodal-art-projection/YuE and
+https://github.com/fspecii/ace-step-ui.
+
+## Agent learning and skill library
+
+The bundled skill library and design references are derived from the
+`NousResearch/hermes-agent` project at revision
+`057dcdf236f8a6a26721c10fcc6ccb72726e272a`.
+
+MIT License
+
+Copyright (c) 2025 Nous Research
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
+
+Individual skill packages may carry additional author and license metadata in
+their own `SKILL.md` or `LICENSE` files. Those notices are preserved alongside
+the package.
+
+## Everything Claude Code integration
+
+Jonathan Ai can synchronize and adapt the `affaan-m/ECC` project. Managed
+content records the exact upstream revision and retains this notice. The ECC
+catalog is not Jonathan branding and is not enabled wholesale in prompts.
+
+MIT License
+
+Copyright (c) 2026 Affaan Mustafa
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
+
+## Kronos financial-market design reference
+
+Jonathan's independent EWMA/Monte Carlo research-forecast engine was added
+after evaluating the market-specific modeling workflow documented by
+`shiyu-coder/Kronos`, licensed under MIT. No upstream source, model code,
+weights, checkout, branding, or runtime is included or required. See
+https://github.com/shiyu-coder/Kronos.
+
+Copyright (c) 2025 Yu Shi and Kronos contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
+
+## Securo personal-finance design reference
+
+Jonathan's independent SQLite personal-finance vault was informed by the
+feature taxonomy publicly documented by `securo-finance/securo`, licensed
+under GNU AGPL-3.0. No Securo source, container, frontend, service, branding,
+database schema, or runtime is copied, linked, or required. See
+https://github.com/securo-finance/securo.
+
+## Apache Maka design reference
+
+Jonathan's append-only runtime-fact ledger and parts of the compact desktop
+state vocabulary were informed by the documented architecture and design
+principles of Apache Maka (Incubating), licensed under Apache License 2.0.
+Jonathan's implementation is independent Python/CSS code; no Maka runtime,
+branding, logo, or package is included. See https://github.com/apache/maka.
+
+Apache Maka
+Copyright 2025-2026 The Apache Software Foundation
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not
+use this file except in compliance with the License. You may obtain a copy at
+https://www.apache.org/licenses/LICENSE-2.0 . Unless required by applicable
+law or agreed to in writing, software distributed under the License is
+distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the specific language
+governing permissions and limitations under the License.
+
+## Universal Earned Autonomy Layer design reference
+
+Jonathan's local approval-evidence profiles were informed by the trust,
+risk-ceiling, replay-resistance, and explicit-enactment design documented in
+`MMVFIRM/Universal-Earned-Autonomy-Layer` at revision
+`174dd650c23862c1703f842d6bc8fc01c0973eaf`, licensed under Apache License
+2.0. Jonathan's implementation is independent and remains part of its existing
+permission enforcement; no upstream runtime, branding, keys, service, or
+production-deployment claim is included. See
+https://github.com/MMVFIRM/Universal-Earned-Autonomy-Layer.
+
+## OpenBot design evaluation
+
+The preflight-audit gateway pattern was informed by the documented design of
+`CopilotKit/OpenBot`, licensed under MIT. Jonathan's implementation is
+independent Python code in its existing dispatcher. No OpenBot runtime,
+branding, CopilotKit service, license token, container image, or source code is
+included. See https://github.com/CopilotKit/OpenBot.
+
+## Scandinavian Design evaluation
+
+Jonathan's interface was evaluated against the public design guidance and
+demonstration harness in `ericzakariasson/scandinavian-design`. That repository
+does not publish a license file, so Jonathan includes none of its skill text,
+scripts, media, CSS, or other source. Jonathan's neutral compact desktop styles
+are an independent implementation. See
+https://github.com/ericzakariasson/scandinavian-design.
+
+## Autoprompt Skill design reference
+
+Jonathan's opt-in verified-worker mode was informed by the independent-judgment
+and explicit-concurrency principles documented in `Spielewoy/autoprompt-skill`
+at revision `91dc6edf336abea96ceb10eaf7efab77ca4c7007`, licensed under MIT.
+Jonathan's implementation is independent Python/HTML code for its existing
+internal workers; no upstream installer, provider adapter, branding, or skill
+payload is included. See https://github.com/Spielewoy/autoprompt-skill.
+
+## Claude DB code-memory design reference
+
+Jonathan's independent Python/SQLite code graph was informed by the documented
+incremental graph and explicit-memory workflow in `Avijit07x/claude-db` at
+revision `6f2fc33cfe6fd70ce2b05393ecfbdd3224c41a03`, licensed under Apache
+License 2.0. No upstream source, Node runtime, database adapter, Claude hook,
+checkout, or binary is included or required. See
+https://github.com/Avijit07x/claude-db.
+
+## Benjamin Plus design reference
+
+Jonathan's compact agent-efficiency guidance was informed by the measured
+workflow rules in `JetBrains/benjamin-plus-skill` at revision
+`0f7b2df6887d18afb73e60b3a696e765e86eea5b`, licensed under MIT. Jonathan
+uses an independent shortened prompt compatible with its own tools and does
+not install upstream hooks. See
+https://github.com/JetBrains/benjamin-plus-skill.
+
+## Fullstack Agent evaluation
+
+`jaredrhod/fullstack-agent` at revision
+`d3c7374c89680628f38e1e63aa9d64a0a7170f7b` was evaluated under AGPL-3.0.
+It is a setup conductor for separate Claude Code, Obsidian, voice, browser,
+and webcam repositories. No source, branding, runtime, or installer from it
+is included in Jonathan. See https://github.com/jaredrhod/fullstack-agent.
+
+## Procoder design reference
+
+Jonathan's independent engineering gate was informed by the fail-closed,
+unchecked-is-not-green contract documented in `azrtydxb/procoder` at revision
+`16122832b5d1c858b4792ad14798d02b0931039d`, licensed under Apache License
+2.0. No upstream binary, source, plugin, hook, configuration, or checkout is
+included or required. See https://github.com/azrtydxb/procoder.
+
+## DrawAI design reference
+
+Jonathan's independent raster-to-editable SVG/PPTX engine was informed by the
+public artifact goals documented in `Renaissance-Mind/DrawAI` at revision
+`b7a43c651cf9fe10a771728b0bdcd7557d762602`, licensed under Apache License
+2.0. No upstream source, runtime, model, workbench, checkout, or model terms
+are included or required. See https://github.com/Renaissance-Mind/DrawAI.
+
+## Kindergrimm design reference
+
+Jonathan's independent deterministic SVG character recipe/rig engine was
+informed by `albertobeiz/kindergrimm` at revision
+`5857b1e1cae2713d6714ad7dd7f89626bb242f0f`, released under the Unlicense.
+No upstream game, source, artwork, branding, or checkout is included. See
+https://github.com/albertobeiz/kindergrimm.
+
+## Scroll Craft design reference
+
+Jonathan's independent `scroll-storytelling` skill was informed by the
+design-floor and verification concepts documented in
+`nateherkai/scroll-craft`, licensed under MIT. No upstream engine, scripts,
+templates, media, plugin, generated assets, or checkout is included. See
+https://github.com/nateherkai/scroll-craft.
+
+## Hermes3D design reference
+
+Jonathan's independent spatial agent-floor UI was informed by the fleet
+visibility goal documented in `iamlukethedev/Hermes3D`, licensed under MIT.
+No upstream 3D office, Three.js/Next.js frontend, gateway, adapter, token,
+asset, source, or checkout is included. See
+https://github.com/iamlukethedev/Hermes3D.
+
+## Named-agent roster reference
+
+The named-agent roster design was informed by `NousResearch/Hermes-Bot-Mode`
+at revision `80fee22582b871b9765a65e2992b8b5a8211c9f8`, licensed under MIT.
+Jonathan's roster implementation is independent and uses Jonathan's existing
+session runtime.
+
+## Coding harness reference
+
+Checkpoint, interruption, and recovery design was informed by
+`xai-org/grok-build` at revision
+`07b2f7144fd5c5c9d3dd1966937a87852d2dbdb8`, licensed under the Apache
+License 2.0. Jonathan's implementation is independent; no vendor branding or
+service authentication code is included.
+
+The unofficial reconstructed desktop repository was reviewed only to
+understand general interaction ideas. Its notice grants no source license, so
+no source code or binary from it is included in Jonathan Ai.
+
+## Awesome LLM Apps evaluation
+
+`Shubhamsaboo/awesome-llm-apps`, licensed under Apache License 2.0, was
+evaluated as a cookbook of self-contained agent, RAG, MCP, voice and skill
+examples. Jonathan already implements those foundations in its own runtime, so
+no template, prompt pack, application, dependency, branding, tutorial content,
+or upstream checkout is included. See
+https://github.com/Shubhamsaboo/awesome-llm-apps.
+
+## Public APIs design reference
+
+Jonathan's independent offline `PublicApiCatalog` uses the general
+category/authentication/HTTPS/CORS discovery vocabulary documented by
+`public-apis/public-apis`, licensed under MIT. Jonathan ships a deliberately
+small, independently maintained starter index and does not copy, download,
+mirror, execute, or depend on the upstream catalog or validation scripts. Each
+provider's own documentation and terms remain authoritative. See
+https://github.com/public-apis/public-apis.
+
+## Humanizer Stack design evaluation
+
+The two-pass surface/structure separation documented by
+`NulightJens/humanizer-stack` was evaluated. The repository's own work is MIT,
+while its attribution file identifies material under additional terms including
+CC BY-SA 4.0. Jonathan therefore includes no upstream skill text, reference
+material, scanner, research digest, installer, or symlink. The bundled
+`writing-polish` skill is an independent implementation focused on meaning,
+voice, clarity, structural variety and honest authorship. See
+https://github.com/NulightJens/humanizer-stack.
+
+## AI Workflow Hub 2000 evaluation
+
+`emretasss/AI-Workflow-Hub-2000-` was evaluated as a collection of n8n workflow
+JSON. The repository does not publish a license file, and the workflows can
+reference external credentials and side effects. Jonathan therefore includes
+none of its workflows, documentation, branding, hosted service, or source and
+does not clone or execute the repository. Jonathan's existing cron, connector,
+worker, permission and preflight-audit systems remain the supported native
+automation path. See
+https://github.com/emretasss/AI-Workflow-Hub-2000-.
+
+## JustInCache n8n Workflows evaluation
+
+`JustInCache/n8n-workflows`, licensed under MIT, was evaluated for its
+searchable workflow catalog and validation claims. The collection requires a
+separate n8n runtime (which has its own licensing terms) and workflows can hold
+credentials or invoke open-world side effects. Jonathan therefore includes no
+workflow corpus, FastAPI/SQLite catalog service, container/Kubernetes stack,
+branding, or automatic execution path. Jonathan's native catalogs, connector
+cards, cron tasks, permission gateway and execution ledger remain the supported
+automation boundary. See https://github.com/JustInCache/n8n-workflows.
+
+## Open Generative AI evaluation
+
+`Anil-matcha/Open-Generative-AI`, licensed under MIT, was evaluated for its
+image, video, lip-sync and multi-reference studio surfaces. Its documented
+model fleet requires a MuAPI access key/cloud gateway and its Electron/Next.js
+application duplicates Jonathan's shell and media workspace. Jonathan includes
+no upstream source, UI, submodule, model registry, gateway client, branding,
+installer, content-policy bypass, or paid-service dependency. Jonathan's native
+local engines and explicitly configured ComfyUI/external connectors remain the
+supported media paths. See
+https://github.com/Anil-matcha/Open-Generative-AI.
+
+## Taste Skill design reference
+
+Jonathan's independent `interface-art-direction` skill was informed by the
+adjustable layout-variance, motion-intensity and information-density concept
+documented by `Leonxlnx/taste-skill`, licensed under MIT. Jonathan includes no
+upstream skill wording, experimental prompt, GSAP skeleton, reference image,
+research text, example, branding, shell script, or checkout. Its implementation
+uses Jonathan's own accessibility and verification requirements. See
+https://github.com/Leonxlnx/taste-skill.
+
+## Vercel Labs Agent Skills evaluation
+
+`vercel-labs/agent-skills` was evaluated for its React and web-design guidance.
+At the time of evaluation the repository README stated MIT, but the repository
+had no top-level license file and maintained open issues requesting one.
+Jonathan therefore includes no upstream skill wording, examples, scripts,
+assets, package, or checkout. The bundled `react-performance-review` skill is
+an independent implementation based on generally applicable measurement,
+waterfall, bundle, rendering-boundary and verification practices. See
+https://github.com/vercel-labs/agent-skills.
+
+## Awesome Design MD reference
+
+Jonathan's bounded project-owned `DESIGN.md` context loading was informed by
+the persistent plain-text design-system workflow documented by
+`VoltAgent/awesome-design-md`, licensed under MIT. Jonathan includes no
+third-party brand analysis, extracted CSS tokens, preview HTML, trademarked
+visual identity, catalog, hosted service, or upstream checkout. Only the user's
+own project-root `DESIGN.md` is loaded. See
+https://github.com/VoltAgent/awesome-design-md.
+
+## gztchan Awesome Design evaluation
+
+`gztchan/awesome-design` was evaluated as a CC0-style curated list of external
+design resources. It contains links rather than an application engine, and
+each linked stock, icon, font, prototyping, testing, book, event or community
+resource has its own availability and terms. Jonathan includes no copied list,
+description, third-party asset, branding, or checkout; live web discovery and
+the native design/connector systems remain the supported path. See
+https://github.com/gztchan/awesome-design.
+
+## Screenshot to Code design reference
+
+Jonathan's independent `screenshot-to-interface` skill was informed by the
+image-to-code and rendered-preview verification workflow documented by
+`abi/screenshot-to-code`, licensed under MIT. Jonathan includes no upstream
+source, prompts, frontend, backend, assets, branding, model routing, Chromium
+runtime, API-key requirement, hosted service, or checkout. It uses Jonathan's
+existing attachment, vision, coding, preview and artifact systems. See
+https://github.com/abi/screenshot-to-code.
+
+## OpenCode evaluation
+
+`anomalyco/opencode`, licensed under MIT, was evaluated for its terminal and
+desktop clients, build/plan/general agents, provider support, session model,
+LSP, MCP and permission controls. Those foundations overlap Jonathan's native
+desktop/CLI, independent conversation agents, Plan Mode, provider adapters,
+connectors, language tools and audited approval gateway. Jonathan includes no
+OpenCode source, Bun/Tauri runtime, provider catalog, UI, branding, hosted
+service, installer, binary, prompts, configuration, or upstream checkout. See
+https://github.com/anomalyco/opencode.
+
+## OpenCode Voice and Whisper design references
+
+Jonathan's user-initiated voice interaction and raw-transcript fallback were
+informed by `renjfk/opencode-voice`, licensed under MIT. Jonathan's local
+multilingual transcription is compatible with the model architecture and MIT
+model weights published by `openai/whisper`; the standard install uses the
+independent `faster-whisper` runtime for practical Windows/Python support.
+Jonathan includes no upstream OpenCode plugin, prompts, SoX, Piper, global
+keybinds, hosted endpoint, mandatory LLM normalization, background microphone
+listener, Whisper source checkout, or branding. Model weights download only
+when transcription is first requested. See https://github.com/renjfk/opencode-voice
+and https://github.com/openai/whisper.
+
+## ScrapeGraphAI evaluation
+
+`ScrapeGraphAI/Scrapegraph-ai`, licensed under MIT, was evaluated for its
+prompt-driven single/multi-page extraction, local-model support and graph
+pipelines. Jonathan already composes WebSearch, WebFetch, browser automation,
+local/cloud providers, internal workers and structured outputs inside its own
+audited permission boundary. Jonathan includes no upstream scraper source,
+Playwright browser download, second provider abstraction, hosted-credit API,
+telemetry, proxy/anti-bot integration, generated scraper, branding or checkout.
+Sites' access rules, terms, privacy and user approval remain authoritative. See
+https://github.com/ScrapeGraphAI/Scrapegraph-ai.
+
+## Agent Reach design reference
+
+Jonathan's independent `ContentReach` tool was informed by the per-channel
+health/routing goal documented by `Panniantong/Agent-Reach`, licensed under
+MIT. Jonathan includes no upstream CLI, channel modules, installer, skills,
+configuration, browser session integration, cookies, proxy recommendations,
+sponsored services, Jina/Exa/OpenCLI dependency, Bilibili/X/Reddit/Xiaohongshu
+adapter, branding or checkout. Public RSS and YouTube support use declared
+libraries; authenticated services require explicit Jonathan connectors and
+user approval. See https://github.com/Panniantong/Agent-Reach.
+
+## Scrapling design evaluation
+
+`D4Vinci/Scrapling`, licensed under BSD-3-Clause, was evaluated for its DOM
+selection, adaptive crawl, export, throttling, robots and browser features.
+Jonathan independently strengthened WebFetch's visible-text boundary by
+dropping executable, template, head and hidden regions before model context.
+Jonathan includes no upstream parser/runtime, agent skill, MCP server, browser,
+stealth fingerprinting, CAPTCHA/Turnstile bypass, cookies, proxy rotation,
+anti-bot behavior, sponsor integration, branding or checkout. See
+https://github.com/D4Vinci/Scrapling.
+
+## Higgsfield AI Skills evaluation
+
+`higgsfield-ai/skills`, licensed under MIT, was evaluated for its hosted
+image/video/audio generation, identity training, brand/product assets,
+thumbnails, explainer videos, website deployment and game workflows. The
+skills install and authenticate a Higgsfield vendor CLI and route work to its
+hosted model catalog. Jonathan includes no upstream skill wording, prompts,
+scripts, CLI, plugin manifest, assets, brand kit, model identifiers, account,
+hosted job dependency, branding or checkout. Jonathan's native media, design,
+3D, website, artifact and editable skill systems remain provider-neutral. See
+https://github.com/higgsfield-ai/skills.
+
+## Higgsfield training framework evaluation
+
+`higgsfield-ai/higgsfield`, licensed under Apache License 2.0, was evaluated as
+an early distributed GPU workload/training framework. It requires remote
+Ubuntu GPU nodes, Docker, SSH/deploy keys, passwordless sudo, GitHub Actions
+and tightly coupled distributed-training dependencies. Those prerequisites are
+inappropriate for a standard Windows desktop install and would broaden system
+authority substantially. Jonathan includes no upstream orchestrator, training
+code, container setup, workflow, credentials, branding or checkout; advanced
+clusters remain explicit user-configured infrastructure. See
+https://github.com/higgsfield-ai/higgsfield.
+
+## Augmented Unreality evaluation
+
+`adynathos/AugmentedUnreality` was evaluated as a possible augmented-reality
+feature reference. It targets Unreal Engine 4.19 and implements camera/video
+capture, calibration and ArUco marker tracking through Unreal-native code,
+OpenCV-contrib and optional GStreamer. Those engine- and platform-specific
+requirements do not add a generally usable capability to Jonathan's existing
+desktop image understanding, editing and 3D artifact tools. No source, assets,
+branding, binary, plugin or runtime dependency from the project is bundled.
+
+Project: https://github.com/adynathos/AugmentedUnreality.
+
+## Unreal Augmented Reality Toolkit evaluation
+
+`lmalave/unreal-augmented-reality-toolkit`, licensed under MIT, was evaluated
+as a possible media/AR feature reference. The repository's implemented scope is
+limited to copying webcam RGBA frames into an Unreal Engine dynamic texture;
+marker detection and further computer-vision behavior are roadmap items. This
+does not improve Jonathan's native attachment previews, image understanding or
+editing tools enough to justify an Unreal/OpenCV SDK build. No source, binary,
+asset, branding or runtime dependency from the project is bundled.
+
+Project: https://github.com/lmalave/unreal-augmented-reality-toolkit.
+
+## Unreal Engine MCP design reference
+
+`flopperam/unreal-engine-mcp`, licensed under MIT, was evaluated for live Unreal
+Editor automation. Jonathan independently implements the compatible high-level
+workflow in native `UnrealStudio`: local editor/project discovery, descriptor
+inspection, generation of reviewable project-scoped scene scripts, and an
+approval-gated `UnrealEditor-Cmd` execution path. No hosted service, API key,
+brand, prompt/skill text, TCP server, C++ plugin, source or binary from the
+project is bundled.
+
+Reference: https://github.com/flopperam/unreal-engine-mcp.
+
+## 9router evaluation
+
+`decolua/9router`, licensed under MIT, was evaluated for provider routing,
+quota tracking and request translation. Its separate localhost proxy, broad
+OAuth/API credential store, multi-account rotation, content compression,
+request logging and automatic cross-provider fallback overlap Jonathan's
+native provider adapters, explicit connector vault, informational token meters
+and selected-provider resilience. Automatically moving prompts among providers
+can change privacy, billing, model behavior and tool semantics, so Jonathan does
+not import or silently enable that architecture. No source, prompt modifier,
+branding, cloud service or runtime dependency from the project is bundled.
+
+Project: https://github.com/decolua/9router.
diff --git a/VERSION b/VERSION
index a45be46..cb498ab 100644
--- a/VERSION
+++ b/VERSION
@@ -1 +1 @@
-0.2.8
+0.4.8
diff --git a/desktop/electron-builder.yml b/desktop/electron-builder.yml
index 163eeee..58a38c0 100644
--- a/desktop/electron-builder.yml
+++ b/desktop/electron-builder.yml
@@ -2,7 +2,7 @@ appId: io.godesk.jonathanai
productName: Jonathan Ai
copyright: MIT
directories:
- output: ../packaging/windows/bin/electron
+ output: ../packaging/windows/build/electron
files:
- main.js
- preload.js
@@ -16,4 +16,6 @@ win:
- target: dir
arch:
- x64
-electronVersion: 36.0.0
+electronVersion: 44.0.0
+electronDownload:
+ cache: ../packaging/windows/build/cache/electron
diff --git a/desktop/main.js b/desktop/main.js
index 8d6e1c0..c316836 100644
--- a/desktop/main.js
+++ b/desktop/main.js
@@ -1,15 +1,26 @@
-const { app, BrowserWindow, Tray, Menu, Notification, dialog, clipboard, desktopCapturer, nativeImage, ipcMain } = require("electron");
-const { spawn } = require("child_process");
+const { app, BrowserWindow, Tray, Menu, Notification, dialog, clipboard, desktopCapturer, nativeImage, ipcMain, shell } = require("electron");
+const { spawn, spawnSync } = require("child_process");
const fs = require("fs");
const http = require("http");
+const net = require("net");
const os = require("os");
const path = require("path");
const HOST = "127.0.0.1";
-const PORT = Number(process.env.CLAWD_DESKTOP_PORT || 8765);
+let PORT = Number(process.env.CLAWD_DESKTOP_PORT || 8765);
const TOKEN = process.env.CLAWD_DESKTOP_TOKEN || require("crypto").randomBytes(18).toString("hex");
app.setName("Jonathan Ai");
+// The packaged shell must also run on Windows systems where Chromium's GPU
+// helper DLL/runtime is unavailable. This UI is lightweight and needs no GPU.
+app.disableHardwareAcceleration();
+app.commandLine.appendSwitch("in-process-gpu");
+// Keep Electron's singleton lock/cache beside Jonathan's local configuration.
+// Some Windows profiles deny lock-file creation in the legacy roaming AppData
+// directory, which made the launcher silently fall back to a browser window.
+const USER_DATA = path.join(os.homedir(), ".clawd", "electron");
+fs.mkdirSync(USER_DATA, { recursive: true });
+app.setPath("userData", USER_DATA);
function hasCliAndRuntime(dir) {
const cli = path.join(dir, "src", "cli.py");
@@ -41,6 +52,25 @@ function resolveRoot() {
}
const ROOT = resolveRoot();
+const PROCESS_RECORD = path.join(ROOT, ".jonathan-ai-processes.json");
+const RUN_DIR = path.join(os.homedir(), ".clawd", "run");
+const LOG_PATH = path.join(RUN_DIR, "desktop.log");
+
+fs.mkdirSync(RUN_DIR, { recursive: true });
+
+function logRuntime(event, details = {}) {
+ const payload = {
+ at: new Date().toISOString(),
+ event,
+ pid: process.pid,
+ ...details,
+ };
+ try {
+ fs.appendFileSync(LOG_PATH, `${JSON.stringify(payload)}\n`, "utf8");
+ } catch (_err) {
+ // Logging must never prevent the desktop from starting.
+ }
+}
let mainWindow = null;
let tray = null;
@@ -55,17 +85,37 @@ function pythonCommand() {
return process.platform === "win32" ? "python" : "python3";
}
+function runtimeReady() {
+ try {
+ const version = fs.readFileSync(path.join(ROOT, "VERSION"), "utf8").trim();
+ const marker = path.join(ROOT, `.jonathan-ai-runtime-${version}.ready`);
+ const electron = path.join(ROOT, "desktop", "node_modules", "electron", "dist", process.platform === "win32" ? "electron.exe" : "electron");
+ return fs.existsSync(pythonCommand()) && fs.existsSync(electron) && fs.existsSync(marker);
+ } catch (_err) {
+ return false;
+ }
+}
+
function bootstrapDeps() {
- const { spawnSync } = require("child_process");
+ if (runtimeReady()) {
+ logRuntime("bootstrap_skipped", { reason: "runtime marker present" });
+ return true;
+ }
+ logRuntime("bootstrap_started");
try {
- spawnSync(pythonCommand(), ["-m", "src.install.bootstrap", "--source-dir", ROOT], {
+ const result = spawnSync(pythonCommand(), ["-m", "src.install.bootstrap", "--source-dir", ROOT], {
cwd: ROOT,
env: { ...process.env, CLAWD_SOURCE_DIR: ROOT },
- timeout: 180000,
+ timeout: 7200000,
stdio: "ignore",
});
- } catch (_err) {
+ const ok = result.status === 0;
+ logRuntime("bootstrap_finished", { ok, status: result.status, signal: result.signal || "" });
+ return ok;
+ } catch (error) {
+ logRuntime("bootstrap_failed", { error: String(error?.message || error) });
// Setup can still repair the venv. Chat host starts anyway.
+ return false;
}
}
@@ -73,46 +123,120 @@ function startPython() {
const args = ["-m", "src.cli", "desktop", "--host", HOST, "--port", String(PORT), "--no-browser", "--token", TOKEN];
python = spawn(pythonCommand(), args, {
cwd: ROOT,
- env: { ...process.env, CLAWD_DESKTOP_TOKEN: TOKEN },
+ env: { ...process.env, CLAWD_DESKTOP_TOKEN: TOKEN, CLAWD_DESKTOP_MANAGED_BY_ELECTRON: "1" },
stdio: ["ignore", "pipe", "pipe"],
});
- python.stdout.on("data", (chunk) => process.stdout.write(chunk));
- python.stderr.on("data", (chunk) => process.stderr.write(chunk));
+ logRuntime("backend_spawned", { backend_pid: python.pid || null, port: PORT });
+ python.stdout.on("data", (chunk) => {
+ process.stdout.write(chunk);
+ logRuntime("backend_stdout", { text: String(chunk).trim().slice(0, 4000) });
+ });
+ python.stderr.on("data", (chunk) => {
+ process.stderr.write(chunk);
+ logRuntime("backend_stderr", { text: String(chunk).trim().slice(0, 4000) });
+ });
+ python.on("error", (error) => {
+ logRuntime("backend_spawn_error", { error: String(error?.message || error) });
+ });
python.on("exit", (code) => {
+ logRuntime("backend_exited", { code });
if (!app.isQuiting && code) {
dialog.showErrorBox("Jonathan Ai backend stopped", `Python host exited with code ${code}`);
}
});
+ writeProcessRecord();
+}
+
+function writeProcessRecord() {
+ const payload = {
+ source_dir: ROOT,
+ owner_pid: process.pid,
+ backend_pid: python?.pid || null,
+ kind: "electron",
+ port: PORT,
+ owner_executable: process.execPath,
+ updated_at: new Date().toISOString(),
+ };
+ const temp = `${PROCESS_RECORD}.${process.pid}.tmp`;
+ fs.writeFileSync(temp, JSON.stringify(payload, null, 2), "utf8");
+ fs.renameSync(temp, PROCESS_RECORD);
+}
+
+function removeProcessRecord() {
+ try {
+ const payload = JSON.parse(fs.readFileSync(PROCESS_RECORD, "utf8"));
+ if (Number(payload.owner_pid) === process.pid) fs.unlinkSync(PROCESS_RECORD);
+ } catch (_err) {
+ // Missing/stale process records are repaired on the next launch or upgrade.
+ }
}
function waitForHealth(timeoutMs = 60000) {
const started = Date.now();
return new Promise((resolve, reject) => {
+ let settled = false;
+ let retryTimer = null;
const attempt = () => {
- const req = http.get({ host: HOST, port: PORT, path: "/api/health", timeout: 1000 }, (res) => {
+ if (settled) return;
+ const req = http.get({
+ host: HOST,
+ port: PORT,
+ path: "/api/ready",
+ timeout: 1000,
+ headers: { "X-Clawd-Token": TOKEN },
+ }, (res) => {
if (res.statusCode === 200) {
+ settled = true;
+ if (retryTimer) clearTimeout(retryTimer);
+ logRuntime("backend_ready", { port: PORT, elapsed_ms: Date.now() - started });
resolve();
return;
}
retry();
});
- req.on("error", retry);
+ req.once("error", retry);
req.on("timeout", () => {
req.destroy();
- retry();
});
};
const retry = () => {
+ if (settled) return;
if (Date.now() - started > timeoutMs) {
+ settled = true;
reject(new Error("desktop host did not become ready"));
return;
}
- setTimeout(attempt, 250);
+ if (retryTimer) clearTimeout(retryTimer);
+ retryTimer = setTimeout(attempt, 250);
};
attempt();
});
}
+function canBind(port) {
+ return new Promise((resolve) => {
+ const probe = net.createServer();
+ probe.unref();
+ probe.once("error", () => resolve(false));
+ probe.listen({ host: HOST, port, exclusive: true }, () => {
+ probe.close(() => resolve(true));
+ });
+ });
+}
+
+async function chooseAvailablePort(preferred, attempts = 20) {
+ for (let offset = 0; offset < attempts; offset += 1) {
+ const candidate = preferred + offset;
+ if (await canBind(candidate)) {
+ if (candidate !== preferred) {
+ logRuntime("port_conflict_avoided", { preferred, selected: candidate });
+ }
+ return candidate;
+ }
+ }
+ throw new Error(`No free localhost port between ${preferred} and ${preferred + attempts - 1}`);
+}
+
function iconPath() {
const candidates = [
path.join(ROOT, "src", "desktop", "web", "robot.png"),
@@ -122,15 +246,16 @@ function iconPath() {
return candidates.find((item) => fs.existsSync(item)) || "";
}
-function createWindow() {
+function createWindow({ loading = false } = {}) {
const icon = iconPath();
mainWindow = new BrowserWindow({
width: 1280,
height: 840,
minWidth: 880,
minHeight: 600,
- title: "Jonathan Ai 0.2.8",
+ title: "Jonathan Ai 0.4.8",
backgroundColor: "#0b0c0f",
+ show: true,
autoHideMenuBar: true,
icon: icon || undefined,
webPreferences: {
@@ -140,7 +265,30 @@ function createWindow() {
sandbox: false,
},
});
- mainWindow.loadURL(`http://${HOST}:${PORT}/`);
+ logRuntime("window_created", { loading });
+ mainWindow.webContents.on("did-fail-load", (_event, code, description, url) => {
+ logRuntime("window_load_failed", { code, description, url });
+ });
+ mainWindow.webContents.on("render-process-gone", (_event, details) => {
+ logRuntime("window_renderer_gone", { reason: details.reason, exit_code: details.exitCode });
+ });
+ mainWindow.webContents.setWindowOpenHandler(({ url }) => {
+ if (/^https?:\/\//i.test(url)) shell.openExternal(url);
+ return { action: "deny" };
+ });
+ if (loading) {
+ const startupHtml = `
${enabled ? "● Full access enabled" : "○ Full access disabled"}OS scope: ${scope}${approval}
`
+ + `
${access.uac_required_for_elevation ? "○" : "●"} Administrator elevation${access.uac_required_for_elevation ? "Windows will display UAC when administrator rights are required" : "Administrator rights are active"}
Jonathan Ai
- v0.2.8 · standalone
+ v0.4.8 · standalone
@@ -32,14 +32,24 @@
+
+
Project downloads
+
—
+
+
Conversations
-
Double-click or right-click to rename. Titles stay with the session.
+
Double-click to rename. Right-click to rename or remove; removed history stays in memory.
+
-
+
+
+
+
+
@@ -54,6 +64,10 @@
New chat
Checking updates…
+
+
+
+
@@ -82,13 +96,38 @@
New chat
+
-
+
+
+
+
+ Working…
+
+
Agent instances
+
Each conversation runs independently. Select an instance to cross-view it.
+
+
+
@@ -172,101 +211,267 @@
Provider & model
-
-
Git & other agents
-
Optional. Jonathan Ai chats on its own. Connect MCP or another agent only if you already run one. Tokens stay in ~/.clawd/config.json. Jonathan Ai never pushes main/master unless you name that branch. Default GitHub owner is GoDeskio.
-
-
GitHub
-
Not connected
-
-
-
-
-
+
+
+
Connect tools
Choose a small card, then complete only that connection’s requirements.
+
-
-
-
-
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
-
-
-
GitLab
-
Not connected
-
-
-
-
-
-
-
-
-
-
-
-
-
-
+
+
-
MCP servers
-
Add a real MCP server the user already has (stdio command or HTTP URL). Test lists its tools; Jonathan Ai can call them in a turn.
-
-
-
-
-
-
-
-
-
-
-
+
+
Image Studio
+
Generation uses Jonathan's local diffusion engine by default. Cloud generation is optional; prompt-based cloud editing requires an OpenAI-compatible image endpoint.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
-
Other agents
-
OpenAI-compatible agent URL (Cursor / Codex / local). Or drop ~/.clawd/hooks/cursor.json. Other tools can POST /api/hooks/inbound on localhost with the desktop token.
-
-
-
-
-
-
+
+
Jonathan local diffusion engine
+
Jonathan runs Diffusers directly inside its installed Python environment. No Fooocus checkout, Gradio service, account, or API key is required. First generation downloads the configured image-model weights.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
AI providers
Configure chat models in Settings. OpenAI image credentials can be saved separately in Image Studio without changing the current Anthropic chat model.
+
+
+
+
+
GitHub
Not connected
+
+
+
+
+
+
+
GitLab
Not connected
+
+
+
+
+
+
+
MCP servers
Connect a stdio command or HTTP MCP server and test its advertised tools.
+
+
+
+
+
+
Other agents
+
+
+
+
+
External OAuth, password & API connections
+
+
+
+
+
+
+
+
Business & finance
Market monitoring and local business records work without a broker. Alpaca defaults to paper trading; every order still needs separate approval.
+
+
+
+
+
Native market forecasts
+
Run Jonathan's built-in probabilistic EWMA/Monte Carlo engine on a timestamped OHLCV CSV. Forecasts are research artifacts—not investment advice, broker instructions, or automatic trading signals.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
Personal finance vault
+
Jonathan's built-in local SQLite vault for accounts, transactions, categories, recurring items, budgets, goals, assets, reports, and currencies. It needs no container, cloned application, subscription, or payment.
+
+
+
+
+
+
+
Local code memory
+
Incrementally maps symbols and dependencies, searches the project, and stores only notes you explicitly request. Data stays in Jonathan’s local SQLite database; no Claude hooks, cloud account, or subscription is installed.
+
+
+
+
+
+
+
+
+
+
+
+
+
Engineering quality gate
+
Runs Jonathan's built-in fail-closed engineering controller against the current project. Unchecked work is reported as failing. It does not edit files, install hooks, commit, tag, file issues, or upgrade itself.
+
+
+
+
+
+
+
+
Raster to editable graphics
+
Decompose a technical figure, diagram, or slide screenshot with Jonathan's local color-region vectorizer and OCR, then download editable SVG and native-shape PPTX outputs. No separate model pack, account, or cloned application is required.
+
+
+
+
+
+
+
+
Procedural character studio
Create a deterministic editable SVG character and JSON rig recipe from a seed. The same seed always redraws the same character; no model, API, or external application is used.
+
+
+
+
+
3D & Blender
Create GLB/GLTF/OBJ/STL locally. Blender adds BLEND/FBX/USD, textures and high-definition rendering.
+
+
Full Device & Network Access
+
Off by default. When enabled, every conversation agent can read and write across mounted drives, run installed shells and CLIs, launch applications and browsers, and use network tools without repeated Jonathan prompts. Windows ACL and UAC prompts still apply.
+
+
+
Enabling requires typing ENABLE FULL ACCESS. You can revoke it here at any time.
+
+
Earned autonomy
+
Approval history is measured separately for each conversation and tool. Recommendations use conservative confidence bounds and never grant access by themselves.
+
+
Administrator, audit & Local terminals
+
+
-
-
+
+
-
-
+
+
+
Agent control center
Search every chat, run persistent named agents in parallel, and verify the local runtime.
+
+
+
Conversation & memory search
+
+
+
+
+
+
Persistent named agents
+
Each named agent owns one permanent conversation, its own workspace, token totals, and a local inbox. Agents can run at the same time.
+
+
+
+
+
+
+
+
+
System check
Checks the provider, dependencies, terminal backends, persistent state, search, Git, Node, ripgrep and FFmpeg.
+
+
+
+
+
+
+
+
+
+
Shared skills library
Every conversation agent can discover and execute these editable SKILL.md files.
+
+
+
+
Folder
—
+
+
+
+ ECC catalogChecking managed integration…
+
The full MIT-licensed catalog stays indexed on disk. Only enabled skills enter agent context.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
Changes are UTF-8 and become available to every agent immediately.
+
+
diff --git a/src/desktop/web/styles.css b/src/desktop/web/styles.css
index ffd8b7d..a69bcc4 100644
--- a/src/desktop/web/styles.css
+++ b/src/desktop/web/styles.css
@@ -1,17 +1,22 @@
:root {
- --bg: #0b0c0f;
- --bg-2: rgba(18, 19, 24, 0.72);
- --panel: rgba(28, 30, 36, 0.55);
- --glass: rgba(255, 255, 255, 0.06);
- --line: rgba(255, 255, 255, 0.1);
- --text: #f4f7ff;
- --muted: #9aa0a6;
- --accent: #f4f7ff;
- --accent-glow: rgba(244, 247, 255, 0.55);
+ --bg: #0a0a0a;
+ --bg-2: #101010;
+ --panel: #171717;
+ --glass: rgba(255, 255, 255, 0.055);
+ --line: rgba(255, 255, 255, 0.12);
+ --text: #f5f5f5;
+ --muted: #ababab;
+ --accent: #f5f5f5;
+ --accent-glow: transparent;
--danger: #e07a6a;
- --shadow: 0 24px 60px rgba(0, 0, 0, 0.45);
- --radius: 18px;
- font-family: "Segoe UI", "Sora", "Iowan Old Style", Palatino, serif;
+ --success: #8cf0ae;
+ --focus: #b9d7ff;
+ --surface-hover: rgba(255, 255, 255, 0.085);
+ --shadow: 0 12px 32px rgba(0, 0, 0, 0.34);
+ --radius: 7px;
+ color-scheme: dark;
+ font-family: "Segoe UI Variable", "Segoe UI", Inter, system-ui, sans-serif;
+ font-size: 12px;
}
* { box-sizing: border-box; }
@@ -19,90 +24,86 @@ html, body {
height: 100%;
margin: 0;
color: var(--text);
- background:
- radial-gradient(900px 520px at 12% -10%, rgba(244, 247, 255, 0.16), transparent 55%),
- radial-gradient(700px 420px at 110% 10%, rgba(160, 180, 210, 0.12), transparent 50%),
- linear-gradient(180deg, #101114 0%, #0b0c0f 100%);
+ background: var(--bg);
}
body::before {
- content: "";
- position: fixed;
- inset: 0;
- pointer-events: none;
- background-image: linear-gradient(rgba(255,255,255,0.015) 1px, transparent 1px),
- linear-gradient(90deg, rgba(255,255,255,0.015) 1px, transparent 1px);
- background-size: 28px 28px;
- mask-image: radial-gradient(circle at 30% 20%, #000 20%, transparent 70%);
+ content: none;
}
button, input, select, textarea {
font: inherit;
color: inherit;
}
button { cursor: pointer; }
+button, input, select, textarea { transition: border-color 140ms ease, background-color 140ms ease, box-shadow 140ms ease, transform 140ms ease; }
+button:focus-visible, input:focus-visible, select:focus-visible, textarea:focus-visible, a:focus-visible, summary:focus-visible {
+ outline: 2px solid var(--focus);
+ outline-offset: 2px;
+}
+button:disabled { cursor: not-allowed; opacity: .5; transform: none !important; }
+::selection { color: #0b0c0f; background: #dceaff; }
+* { scrollbar-color: rgba(244,247,255,.28) transparent; scrollbar-width: thin; }
#app {
display: grid;
- grid-template-columns: 320px 1fr;
+ grid-template-columns: 190px minmax(0, 1fr);
height: 100%;
position: relative;
}
.sidebar {
- margin: 16px 0 16px 16px;
- padding: 18px 16px;
+ margin: 8px 0 8px 8px;
+ padding: 9px 8px;
display: flex;
flex-direction: column;
- gap: 14px;
+ gap: 7px;
background: var(--bg-2);
border: 1px solid var(--line);
- border-radius: 24px;
- backdrop-filter: blur(22px) saturate(140%);
- -webkit-backdrop-filter: blur(22px) saturate(140%);
+ border-radius: 8px;
box-shadow: var(--shadow);
}
-.brand { display: flex; gap: 12px; align-items: center; }
+.brand { display: flex; gap: 6px; align-items: center; }
.brand img {
- width: 48px;
- height: 48px;
- border-radius: 14px;
- box-shadow: 0 0 24px var(--accent-glow);
+ width: 26px;
+ height: 26px;
+ border-radius: 7px;
+ box-shadow: none;
object-fit: cover;
background: #121214;
}
.brand strong { display: block; letter-spacing: 0.03em; }
-.muted { color: var(--muted); font-size: 13px; }
+.muted { color: var(--muted); font-size: 9px; }
.primary, .ghost {
- border-radius: 12px;
+ border-radius: 7px;
border: 1px solid var(--line);
- padding: 9px 13px;
+ padding: 5px 7px;
background: var(--glass);
- backdrop-filter: blur(12px);
}
.primary {
- background: linear-gradient(180deg, #f4f7ff 0%, #c9d0dc 100%);
+ background: #f5f5f5;
color: #121214;
border-color: transparent;
font-weight: 650;
- box-shadow: 0 0 22px rgba(244, 247, 255, 0.22);
+ box-shadow: none;
}
+.primary:hover { transform: translateY(-1px); background: #ffffff; box-shadow: 0 4px 14px rgba(0, 0, 0, .28); }
+.primary:active, .ghost:active, .danger:active { transform: translateY(0); }
.ghost:hover { background: rgba(255, 255, 255, 0.1); }
.workspace-card, .session-list {
background: var(--panel);
border: 1px solid var(--line);
border-radius: var(--radius);
- padding: 12px;
- backdrop-filter: blur(16px);
+ padding: 6px;
}
-.path { font-size: 12px; word-break: break-all; margin: 6px 0 10px; }
-.section-title { font-size: 11px; text-transform: uppercase; letter-spacing: 0.12em; color: var(--muted); }
-.sidebar-hint { margin: 0 0 4px; font-size: 12px; }
-.session-list { flex: 1; overflow: auto; display: flex; flex-direction: column; gap: 8px; }
+.path { font-size: 9px; word-break: break-all; margin: 3px 0 5px; }
+.section-title { font-size: 8px; font-weight: 600; letter-spacing: 0.02em; color: var(--muted); }
+.sidebar-hint { margin: 0 0 2px; font-size: 8px; }
+.session-list { flex: 1; overflow: auto; display: flex; flex-direction: column; gap: 4px; }
.session-item {
text-align: left;
background: transparent;
border: 1px solid transparent;
- padding: 10px;
- border-radius: 12px;
+ padding: 5px;
+ border-radius: 7px;
width: 100%;
}
.session-item strong, .session-item .session-title { display: block; }
@@ -135,70 +136,80 @@ button { cursor: pointer; }
padding: 8px 10px;
}
.context-menu button:hover { background: rgba(255, 255, 255, 0.08); }
+.context-menu .danger-action { color: #ff9d91; }
.session-item.active, .session-item:hover {
background: rgba(244, 247, 255, 0.08);
border-color: var(--line);
}
+.session-item.active { box-shadow: inset 2px 0 0 var(--accent); }
.sidebar-foot { margin-top: auto; display: flex; gap: 8px; flex-wrap: wrap; }
+.instance-list { display: flex; flex-direction: column; gap: 3px; max-height: 90px; overflow: auto; }
+.instance-item { width: 100%; display: flex; align-items: center; gap: 5px; padding: 4px 5px; border: 1px solid transparent; border-radius: 6px; background: transparent; text-align: left; }
+.instance-item:hover { border-color: var(--line); background: rgba(255,255,255,.05); }
+.agent-dot { width: 6px; height: 6px; flex: 0 0 6px; border-radius: 50%; background: #6d737d; }
+.instance-item.working .agent-dot { background: #8cf0ae; box-shadow: 0 0 8px rgba(140,240,174,.8); animation: agent-pulse 1s ease-in-out infinite alternate; }
+.instance-copy { min-width: 0; overflow: hidden; text-overflow: ellipsis; white-space: nowrap; font-size: 8px; }
+.session-item.working { border-left: 2px solid #8cf0ae; }
+@keyframes agent-pulse { to { transform: scale(1.45); opacity: .65; } }
.main {
display: flex;
flex-direction: column;
min-width: 0;
- margin: 16px;
+ margin: 8px;
background: var(--bg-2);
border: 1px solid var(--line);
- border-radius: 24px;
- backdrop-filter: blur(24px) saturate(140%);
- -webkit-backdrop-filter: blur(24px) saturate(140%);
+ border-radius: 8px;
box-shadow: var(--shadow);
overflow: hidden;
+ position: relative;
}
.topbar {
display: flex;
justify-content: space-between;
align-items: center;
- padding: 18px 22px 12px;
+ padding: 9px 11px 6px;
border-bottom: 1px solid var(--line);
}
-.topbar h1 { margin: 0; font-size: 22px; font-weight: 600; }
+.topbar h1 { margin: 0; font-size: 14px; font-weight: 600; }
+.top-actions { display: flex; align-items: center; gap: 4px; flex-wrap: wrap; justify-content: flex-end; }
.usage-chip {
display: inline-flex;
gap: 8px;
- margin-top: 8px;
- padding: 5px 10px;
+ margin-top: 4px;
+ padding: 3px 6px;
border-radius: 999px;
border: 1px solid var(--line);
background: rgba(244, 247, 255, 0.06);
- font-size: 12px;
+ font-size: 8px;
color: var(--muted);
}
.transcript {
flex: 1;
overflow: auto;
- padding: 22px;
+ padding: 11px;
display: flex;
flex-direction: column;
- gap: 14px;
+ gap: 7px;
}
.bubble {
- max-width: min(760px, 100%);
- padding: 13px 15px;
- border-radius: 16px;
+ max-width: min(680px, 100%);
+ padding: 7px 8px;
+ border-radius: 9px;
border: 1px solid var(--line);
background: var(--panel);
- backdrop-filter: blur(18px);
white-space: pre-wrap;
word-break: break-word;
- box-shadow: 0 10px 30px rgba(0, 0, 0, 0.18);
+ box-shadow: none;
}
.bubble.user { align-self: flex-end; background: rgba(244, 247, 255, 0.1); }
.bubble.assistant { align-self: flex-start; }
+.bubble.assistant, .bubble.user { line-height: 1.5; }
.bubble.system { align-self: center; color: var(--muted); background: transparent; border: 0; box-shadow: none; }
.tool-card {
align-self: flex-start;
border-left: 3px solid var(--accent);
- font-size: 13px;
+ font-size: 9px;
}
.tool-card .summary { color: var(--muted); }
.bubble pre, .md-pre {
@@ -209,23 +220,24 @@ button { cursor: pointer; }
}
.composer {
position: relative;
- padding: 14px 18px 18px;
+ padding: 7px 9px 9px;
border-top: 1px solid var(--line);
background: rgba(12, 13, 16, 0.35);
}
#prompt {
width: 100%;
resize: vertical;
- min-height: 76px;
- border-radius: 16px;
+ min-height: 42px;
+ border-radius: 9px;
border: 1px solid var(--line);
background: rgba(255, 255, 255, 0.05);
- padding: 12px 14px;
- backdrop-filter: blur(12px);
+ padding: 6px 7px;
}
-.composer-row, .row { display: flex; gap: 8px; align-items: center; flex-wrap: wrap; }
+#prompt:hover, #prompt:focus { border-color: rgba(185, 215, 255, .55); background: rgba(255, 255, 255, 0.065); }
+.composer-row, .row { display: flex; gap: 4px; align-items: center; flex-wrap: wrap; }
+.worker-controls select { width: auto; min-width: 70px; padding: 5px 6px; border: 1px solid var(--line); border-radius: 7px; background: var(--panel); }
#usage-line { margin: 4px 0 0; }
-.composer-row { justify-content: space-between; margin-top: 8px; }
+.composer-row { justify-content: space-between; margin-top: 4px; }
.end { justify-content: flex-end; }
.chips { display: flex; flex-wrap: wrap; gap: 6px; margin-bottom: 8px; }
.chip {
@@ -298,6 +310,45 @@ button { cursor: pointer; }
backdrop-filter: blur(22px);
}
.dialog.wide { width: min(640px, calc(100vw - 32px)); max-height: calc(100vh - 48px); overflow: auto; }
+.dialog.connector-dialog { width: min(820px, calc(100vw - 24px)); padding: 14px; overflow: hidden; }
+.connector-titlebar { display: flex; align-items: flex-start; justify-content: space-between; gap: 12px; }
+.connector-titlebar h2 { margin: 0 0 2px; }
+.icon-button {
+ width: 28px; height: 28px; border: 1px solid var(--line); border-radius: 8px;
+ background: rgba(255,255,255,.05); font-size: 18px; line-height: 1;
+}
+.connector-grid, .connector-mini-grid {
+ display: grid; grid-template-columns: repeat(auto-fit, minmax(130px, 1fr)); gap: 7px; margin-top: 12px;
+}
+.connector-card, .connector-mini-card {
+ min-height: 92px; display: flex; flex-direction: column; align-items: flex-start; gap: 3px;
+ text-align: left; border: 1px solid var(--line); border-radius: 11px; padding: 9px;
+ color: var(--text); background: rgba(255,255,255,.04);
+}
+.connector-card:hover, .connector-mini-card:hover, .connector-mini-card.active {
+ transform: translateY(-1px); border-color: rgba(244,247,255,.4); background: rgba(255,255,255,.08);
+}
+.connector-card small { color: var(--muted); min-height: 22px; }
+.connector-card strong, .connector-mini-card strong { letter-spacing: .01em; }
+.connector-icon {
+ display: grid; place-items: center; min-width: 26px; height: 22px; padding: 0 5px;
+ border-radius: 6px; background: rgba(244,247,255,.1); font-weight: 700; font-size: 9px;
+}
+.card-state { margin-top: auto; color: #9be7b2; font-size: 8px; }
+.card-state::before { content: ""; display: inline-block; width: 4px; height: 4px; margin-right: 4px; border-radius: 50%; background: currentColor; vertical-align: 1px; box-shadow: 0 0 6px currentColor; }
+.connector-detail {
+ position: relative; margin-top: 10px; max-height: min(590px, calc(100vh - 155px)); overflow: auto;
+ border: 1px solid var(--line); border-radius: 12px; padding: 12px; background: rgba(8,9,12,.38);
+}
+.detail-close { position: sticky; z-index: 2; top: 0; float: right; }
+.connector-panel h3 { margin: 2px 34px 5px 0; }
+.form-grid { display: grid; gap: 0 8px; }
+.form-grid.two { grid-template-columns: repeat(2, minmax(0, 1fr)); }
+.form-grid .span-two { grid-column: 1 / -1; }
+.compact-list .scan-item { padding: 6px 8px; }
+.danger { border: 1px solid rgba(224,122,106,.55); border-radius: 7px; padding: 5px 7px; background: rgba(224,122,106,.12); color: #ffb0a4; }
+.tool-image-preview, .direct-image-preview { display: block; max-width: min(420px, 100%); max-height: 300px; margin-top: 8px; border-radius: 9px; border: 1px solid var(--line); }
+@media (max-width: 620px) { .form-grid.two { grid-template-columns: 1fr; } .form-grid .span-two { grid-column: auto; } }
.connector { margin: 10px 0 4px; }
.scan-list { display: flex; flex-direction: column; gap: 6px; margin-top: 8px; }
.scan-item {
@@ -317,9 +368,106 @@ button { cursor: pointer; }
border: 1px solid var(--line);
background: rgba(0, 0, 0, 0.28);
}
+.dialog input:hover, .dialog select:hover, .dialog textarea:hover { border-color: rgba(185, 215, 255, .38); }
code { font-family: ui-monospace, SFMono-Regular, Menlo, Consolas, monospace; }
+.working-robot {
+ position: absolute;
+ left: 10px;
+ bottom: 78px;
+ z-index: 6;
+ display: flex;
+ align-items: center;
+ gap: 4px;
+ padding: 3px 6px;
+ border: 1px solid var(--line);
+ border-radius: 999px;
+ background: rgba(15, 16, 20, 0.88);
+ box-shadow: 0 0 16px rgba(244, 247, 255, 0.28);
+ font-size: 8px;
+ animation: robot-roam 2.2s ease-in-out infinite alternate;
+}
+.working-robot img { width: 22px; height: 22px; border-radius: 6px; object-fit: cover; }
+@keyframes robot-roam {
+ 0% { transform: translate(0, 0) rotate(-2deg); }
+ 45% { transform: translate(28px, -7px) rotate(3deg); }
+ 100% { transform: translate(55px, 0) rotate(-1deg); }
+}
+
+#app.preview-open { grid-template-columns: 190px minmax(0, 1fr) minmax(250px, 32vw); }
+.preview-pane {
+ min-width: 0;
+ margin: 8px 8px 8px 0;
+ padding: 8px;
+ overflow: auto;
+ background: var(--bg-2);
+ border: 1px solid var(--line);
+ border-radius: 13px;
+ box-shadow: var(--shadow);
+ backdrop-filter: blur(22px) saturate(140%);
+}
+.preview-head { display: flex; align-items: center; justify-content: space-between; gap: 6px; margin-bottom: 7px; }
+.preview-frame { width: 100%; height: 50%; min-height: 260px; border: 1px solid var(--line); border-radius: 8px; background: #fff; }
+.preview-empty { padding: 16px 7px; text-align: center; border: 1px dashed var(--line); border-radius: 8px; margin-bottom: 8px; }
+.preview-files, .artifact-list { display: flex; flex-direction: column; gap: 3px; margin: 5px 0 10px; }
+.preview-file { padding: 3px 5px; overflow: hidden; text-overflow: ellipsis; white-space: nowrap; border-radius: 4px; background: rgba(255,255,255,.035); font: 8px ui-monospace, Consolas, monospace; }
+.artifact-item { display: flex; align-items: center; justify-content: space-between; gap: 5px; padding: 5px; border: 1px solid var(--line); border-radius: 6px; }
+.artifact-item a { color: var(--text); text-decoration: none; }
+.artifact-item a:hover { text-decoration: underline; }
+.tool-download { display: block; width: fit-content; margin-top: 5px; color: var(--accent); }
+.tool-image-preview { display: block; max-width: min(100%, 520px); max-height: 320px; margin-top: 6px; border: 1px solid var(--line); border-radius: 8px; object-fit: contain; }
+.media-bubble { min-width: min(280px, 80%); }
+.conversation-image { margin: 8px 0 0; overflow: hidden; border: 1px solid var(--line); border-radius: 9px; background: rgba(0, 0, 0, 0.18); }
+.conversation-image > a { display: block; line-height: 0; }
+.conversation-image-preview { display: block; width: auto; max-width: min(100%, 560px); max-height: 420px; margin: 0 auto; object-fit: contain; background: rgba(0, 0, 0, 0.24); }
+.conversation-image figcaption { display: flex; align-items: center; justify-content: space-between; gap: 12px; padding: 7px 9px; font-size: 0.78rem; color: var(--muted); line-height: 1.3; }
+.conversation-image figcaption span { overflow: hidden; text-overflow: ellipsis; white-space: nowrap; }
+.conversation-image-download { flex: 0 0 auto; color: var(--accent); font-weight: 650; text-decoration: none; }
+.conversation-image-download:hover { text-decoration: underline; }
+.media-engine { display: flex; flex-direction: column; gap: 2px; }
+.control-dialog { max-height: min(760px, calc(100vh - 30px)); overflow: auto; }
+.dialog-head { display: flex; justify-content: space-between; align-items: flex-start; gap: 10px; }
+.dialog-head h2 { margin-bottom: 3px; }
+.control-panel > h3 { margin-top: 0; }
+.control-panel textarea { width: 100%; resize: vertical; }
+.profile-grid { display: grid; grid-template-columns: repeat(auto-fill, minmax(210px, 1fr)); gap: 7px; margin-top: 10px; }
+.profile-card { display: flex; flex-direction: column; gap: 5px; min-height: 126px; padding: 9px; border: 1px solid var(--line); border-radius: 10px; background: rgba(255,255,255,.04); }
+.profile-card .profile-actions { display: flex; flex-wrap: wrap; gap: 4px; margin-top: auto; }
+.profile-card .profile-actions button { flex: 1; }
+.agent-floor { display: grid; grid-template-columns: repeat(auto-fit,minmax(145px,1fr)); gap: 12px; margin: 10px 0 14px; padding: 22px 14px 14px; border: 1px solid var(--line); border-radius: 10px; background: linear-gradient(150deg,#1c2025,#101317); perspective: 700px; overflow: hidden; }
+.agent-desk { position: relative; display: flex; flex-direction: column; align-items: flex-start; min-height: 80px; padding: 12px; color: var(--text); text-align: left; border: 1px solid #414851; border-radius: 7px; background: #242a30; box-shadow: 0 7px 0 #0a0c0e; transform: rotateX(2deg); }
+.agent-desk small { color: var(--muted); margin-top: 5px; }
+.agent-desk .desk-light { position: absolute; top: 8px; right: 8px; width: 7px; height: 7px; border-radius: 50%; background: #77808a; }
+.agent-desk.working .desk-light { background: #61d095; box-shadow: 0 0 8px #61d095; animation: pulse 1.1s infinite; }
+.agent-desk.command-desk { border-color: #65717e; background: #2b3239; }
+.agent-desk.empty-desk { opacity: .62; border-style: dashed; }
+.search-hit { cursor: pointer; }
+.search-hit mark { background: rgba(244,247,255,.22); color: inherit; border-radius: 2px; }
+.skills-dialog { width: min(980px, calc(100vw - 30px)); max-height: min(780px, calc(100vh - 30px)); overflow: auto; }
+.skills-toolbar { display: flex; align-items: end; justify-content: space-between; gap: 10px; margin: 8px 0 12px; }
+.skills-layout { display: grid; grid-template-columns: minmax(170px, 28%) minmax(0, 1fr); gap: 10px; min-height: 440px; }
+.ecc-manager { margin: 0 0 10px; padding: 8px; border: 1px solid var(--line); border-radius: 9px; background: rgba(255,255,255,.025); }
+.ecc-manager summary { cursor: pointer; }
+.ecc-manager .row { margin: 7px 0; }
+.ecc-manager input { min-width: 150px; flex: 1; }
+.ecc-skill-list { display: grid; grid-template-columns: repeat(auto-fill, minmax(190px, 1fr)); gap: 5px; max-height: 210px; overflow: auto; }
+.ecc-skill-card { display: grid; grid-template-columns: minmax(0, 1fr) auto; gap: 5px; align-items: center; padding: 6px; border: 1px solid var(--line); border-radius: 8px; }
+.ecc-skill-card span { overflow: hidden; text-overflow: ellipsis; white-space: nowrap; }
+.ecc-skill-card button { padding: 4px 7px; }
+.skills-list { display: flex; flex-direction: column; gap: 5px; overflow: auto; padding-right: 3px; }
+.skill-card { display: flex; flex-direction: column; gap: 3px; padding: 8px; text-align: left; border: 1px solid var(--line); border-radius: 9px; background: rgba(255,255,255,.04); color: var(--text); }
+.skill-card:hover, .skill-card.active { border-color: var(--accent); background: rgba(255,255,255,.08); }
+.skill-card small { color: var(--muted); }
+.skill-editor textarea { width: 100%; min-height: 360px; resize: vertical; padding: 9px; border-radius: 9px; border: 1px solid var(--line); background: rgba(0,0,0,.28); color: var(--text); font: 10px/1.45 ui-monospace, Consolas, monospace; }
+@media (max-width: 720px) { .skills-layout { grid-template-columns: 1fr; } .skills-list { max-height: 150px; } }
+
@media (max-width: 860px) {
#app { grid-template-columns: 1fr; }
.sidebar { display: none; }
+ .preview-pane { display: none !important; }
+}
+
+@media (prefers-reduced-motion: reduce) {
+ *, *::before, *::after { animation-duration: .01ms !important; animation-iteration-count: 1 !important; scroll-behavior: auto !important; transition-duration: .01ms !important; }
+ .working-robot { animation: none; }
}
diff --git a/src/device_control.py b/src/device_control.py
new file mode 100644
index 0000000..1bf95cb
--- /dev/null
+++ b/src/device_control.py
@@ -0,0 +1,158 @@
+"""Installed application, browser, and command discovery for device control."""
+
+from __future__ import annotations
+
+import os
+import shutil
+import sys
+from pathlib import Path
+from typing import Any
+
+
+def _clean_executable(value: str) -> Path | None:
+ text = os.path.expandvars(str(value or "").strip()).strip('"')
+ if not text:
+ return None
+ # Registry DisplayIcon values may end with an icon index.
+ if "," in text:
+ possible, suffix = text.rsplit(",", 1)
+ if suffix.strip().lstrip("-").isdigit():
+ text = possible.strip('" ')
+ path = Path(text).expanduser()
+ return path.resolve() if path.is_file() else None
+
+
+def discover_browsers() -> list[dict[str, str]]:
+ candidates: dict[str, list[Path]] = {}
+ if os.name == "nt":
+ program_files = Path(os.environ.get("ProgramFiles", r"C:\Program Files"))
+ program_files_x86 = Path(os.environ.get("ProgramFiles(x86)", r"C:\Program Files (x86)"))
+ local = Path(os.environ.get("LocalAppData", ""))
+ candidates = {
+ "Microsoft Edge": [program_files_x86 / "Microsoft/Edge/Application/msedge.exe", program_files / "Microsoft/Edge/Application/msedge.exe"],
+ "Google Chrome": [program_files / "Google/Chrome/Application/chrome.exe", program_files_x86 / "Google/Chrome/Application/chrome.exe", local / "Google/Chrome/Application/chrome.exe"],
+ "Mozilla Firefox": [program_files / "Mozilla Firefox/firefox.exe", program_files_x86 / "Mozilla Firefox/firefox.exe"],
+ "Brave": [program_files / "BraveSoftware/Brave-Browser/Application/brave.exe", local / "BraveSoftware/Brave-Browser/Application/brave.exe"],
+ "Vivaldi": [local / "Vivaldi/Application/vivaldi.exe", program_files / "Vivaldi/Application/vivaldi.exe"],
+ "Opera": [local / "Programs/Opera/opera.exe"],
+ }
+ elif sys.platform == "darwin":
+ candidates = {
+ "Safari": [Path("/Applications/Safari.app/Contents/MacOS/Safari")],
+ "Google Chrome": [Path("/Applications/Google Chrome.app/Contents/MacOS/Google Chrome")],
+ "Mozilla Firefox": [Path("/Applications/Firefox.app/Contents/MacOS/firefox")],
+ }
+ else:
+ candidates = {
+ "Google Chrome": [Path(shutil.which("google-chrome") or "")],
+ "Chromium": [Path(shutil.which("chromium") or shutil.which("chromium-browser") or "")],
+ "Mozilla Firefox": [Path(shutil.which("firefox") or "")],
+ "Microsoft Edge": [Path(shutil.which("microsoft-edge") or "")],
+ }
+ rows: list[dict[str, str]] = []
+ seen: set[Path] = set()
+ for name, paths in candidates.items():
+ match = next((path.resolve() for path in paths if str(path) not in {"", "."} and path.is_file()), None)
+ if match and match not in seen:
+ seen.add(match)
+ rows.append({"name": name, "path": str(match)})
+ return rows
+
+
+def _windows_registered_apps() -> list[dict[str, str]]:
+ if os.name != "nt":
+ return []
+ try:
+ import winreg
+ except ImportError:
+ return []
+ rows: dict[str, dict[str, str]] = {}
+ keys = [
+ (winreg.HKEY_CURRENT_USER, r"Software\Microsoft\Windows\CurrentVersion\App Paths"),
+ (winreg.HKEY_LOCAL_MACHINE, r"Software\Microsoft\Windows\CurrentVersion\App Paths"),
+ (winreg.HKEY_LOCAL_MACHINE, r"Software\WOW6432Node\Microsoft\Windows\CurrentVersion\App Paths"),
+ ]
+ for root, key_name in keys:
+ try:
+ with winreg.OpenKey(root, key_name) as key:
+ count = winreg.QueryInfoKey(key)[0]
+ for index in range(count):
+ sub_name = winreg.EnumKey(key, index)
+ try:
+ with winreg.OpenKey(key, sub_name) as sub:
+ value = winreg.QueryValue(sub, None)
+ except OSError:
+ continue
+ path = _clean_executable(value)
+ if path:
+ rows[str(path).lower()] = {"name": path.stem, "path": str(path), "source": "Windows App Paths"}
+ except OSError:
+ continue
+ return sorted(rows.values(), key=lambda item: item["name"].lower())
+
+
+def discover_path_tools(*, limit: int = 2000) -> list[dict[str, str]]:
+ extensions = {item.lower() for item in os.environ.get("PATHEXT", ".EXE;.CMD;.BAT;.COM").split(";") if item}
+ rows: dict[str, dict[str, str]] = {}
+ for raw in os.environ.get("PATH", "").split(os.pathsep):
+ folder = Path(raw.strip('" ')).expanduser()
+ try:
+ is_directory = folder.is_dir()
+ except OSError:
+ is_directory = False
+ if not is_directory:
+ continue
+ try:
+ for path in folder.iterdir():
+ if not path.is_file():
+ continue
+ if os.name == "nt" and path.suffix.lower() not in extensions and path.suffix.lower() != ".ps1":
+ continue
+ if os.name != "nt" and not os.access(path, os.X_OK):
+ continue
+ rows.setdefault(path.name.lower(), {"name": path.stem, "path": str(path.resolve()), "source": "PATH"})
+ if len(rows) >= limit:
+ break
+ except OSError:
+ continue
+ if len(rows) >= limit:
+ break
+ return sorted(rows.values(), key=lambda item: item["name"].lower())
+
+
+def discover_device_controls(*, limit: int = 2000) -> dict[str, Any]:
+ browsers = discover_browsers()
+ applications = _windows_registered_apps()
+ known = {item["path"].lower() for item in applications}
+ for browser in browsers:
+ if browser["path"].lower() not in known:
+ applications.append({**browser, "source": "browser"})
+ tools = discover_path_tools(limit=limit)
+ return {
+ "platform": sys.platform,
+ "applications": sorted(applications, key=lambda item: item["name"].lower()),
+ "browsers": browsers,
+ "command_line_tools": tools,
+ "counts": {"applications": len(applications), "browsers": len(browsers), "command_line_tools": len(tools)},
+ }
+
+
+def resolve_application(identifier: str) -> Path:
+ value = str(identifier or "").strip()
+ if not value:
+ raise ValueError("application or executable is required")
+ direct = _clean_executable(value)
+ if direct:
+ return direct
+ on_path = shutil.which(value)
+ if on_path:
+ return Path(on_path).resolve()
+ wanted = value.lower()
+ inventory = discover_device_controls()
+ candidates = [*inventory["applications"], *inventory["command_line_tools"]]
+ for item in candidates:
+ name = str(item["name"]).lower()
+ path = Path(str(item["path"]))
+ if wanted in {name, path.name.lower(), path.stem.lower()}:
+ return path.resolve()
+ raise ValueError(f"installed application or executable was not found: {identifier}")
diff --git a/src/doctor.py b/src/doctor.py
new file mode 100644
index 0000000..1d19537
--- /dev/null
+++ b/src/doctor.py
@@ -0,0 +1,62 @@
+"""Fast, side-effect-free diagnostics plus safe local repairs."""
+
+from __future__ import annotations
+
+import importlib.util
+import os
+import shutil
+import sqlite3
+import sys
+from pathlib import Path
+from typing import Any
+
+from src.agent.checkpoints import checkpoint_dir
+from src.agent.history_search import rebuild_history_index, search_database_path
+from src.agent.memory import memory_dir
+from src.agent.session import session_dir
+from src.config import get_default_provider, has_configured_provider
+from src.tool_system.tools.terminal import discover_terminals
+
+
+def _check(name: str, ok: bool, detail: str, repairable: bool = False) -> dict[str, Any]:
+ return {"name": name, "ok": bool(ok), "detail": detail, "repairable": repairable}
+
+
+def run_doctor(*, repair: bool = False) -> dict[str, Any]:
+ repaired: list[str] = []
+ if repair:
+ for folder in (session_dir(), memory_dir(), checkpoint_dir()):
+ folder.mkdir(parents=True, exist_ok=True)
+ rebuild_history_index()
+ repaired.extend(["state directories", "history search index"])
+ required = ["anthropic", "openai", "PIL", "psutil", "trimesh"]
+ missing = [name for name in required if importlib.util.find_spec(name) is None]
+ state_root = Path.home() / ".clawd"
+ writable = state_root.exists() and os.access(state_root, os.W_OK)
+ try:
+ with sqlite3.connect(":memory:") as db:
+ db.execute("CREATE VIRTUAL TABLE probe USING fts5(text)")
+ fts = True
+ except sqlite3.OperationalError:
+ fts = False
+ terminals = discover_terminals()
+ checks = [
+ _check("Python", sys.version_info >= (3, 10), sys.version.split()[0]),
+ _check("Python dependencies", not missing, "ready" if not missing else "missing: " + ", ".join(missing), bool(missing)),
+ _check("Persistent state", writable, str(state_root), True),
+ _check("Conversation search", fts, f"FTS5 · {search_database_path()}" if fts else "SQLite FTS5 unavailable"),
+ _check("Model provider", has_configured_provider(), get_default_provider(), False),
+ _check("Git", bool(shutil.which("git")), shutil.which("git") or "not found"),
+ _check("Node.js", bool(shutil.which("node")), shutil.which("node") or "optional / not found"),
+ _check("Ripgrep", bool(shutil.which("rg")), shutil.which("rg") or "optional / not found"),
+ _check("FFmpeg", bool(shutil.which("ffmpeg")), shutil.which("ffmpeg") or "optional / not found"),
+ _check("Terminal backends", bool(terminals), ", ".join(str(row.get("name") or row.get("shell") or "terminal") for row in terminals) or "none"),
+ ]
+ required_checks = checks[:6]
+ return {
+ "ok": all(row["ok"] for row in required_checks),
+ "checks": checks,
+ "repaired": repaired,
+ "next_action": "Connect a model provider." if not has_configured_provider() else ("Run repair." if missing or not writable else "Ready."),
+ }
+
diff --git a/src/file_content.py b/src/file_content.py
new file mode 100644
index 0000000..ad4048e
--- /dev/null
+++ b/src/file_content.py
@@ -0,0 +1,86 @@
+"""Safe text extraction for user-selected local files.
+
+The desktop stages a private copy before calling these helpers. Extraction is
+bounded so an uploaded document cannot silently consume the model context.
+"""
+
+from __future__ import annotations
+
+from pathlib import Path
+from typing import Any
+import zipfile
+
+
+RICH_DOCUMENT_SUFFIXES = {".pdf", ".docx", ".xlsx", ".xlsm", ".pptx", ".zip"}
+
+
+def _bounded(parts: list[str], max_chars: int) -> tuple[str, bool]:
+ text = "\n".join(part for part in parts if str(part).strip())
+ if len(text) <= max_chars:
+ return text, False
+ return text[:max_chars].rstrip() + "\n[content truncated]", True
+
+
+def extract_rich_text(path: str | Path, *, max_chars: int = 256_000) -> tuple[bool, str, str]:
+ """Return ``(supported, text, note)`` for common document containers."""
+ target = Path(path)
+ suffix = target.suffix.lower()
+ if suffix not in RICH_DOCUMENT_SUFFIXES:
+ return False, "", ""
+ try:
+ parts: list[str] = []
+ kind = suffix.lstrip(".").upper()
+ if suffix == ".pdf":
+ from pypdf import PdfReader
+
+ reader = PdfReader(str(target))
+ for index, page in enumerate(reader.pages, 1):
+ parts.append(f"--- Page {index} ---\n{page.extract_text() or ''}")
+ elif suffix == ".docx":
+ from docx import Document
+
+ document = Document(str(target))
+ parts.extend(paragraph.text for paragraph in document.paragraphs)
+ for table_index, table in enumerate(document.tables, 1):
+ parts.append(f"--- Table {table_index} ---")
+ parts.extend("\t".join(cell.text for cell in row.cells) for row in table.rows)
+ elif suffix in {".xlsx", ".xlsm"}:
+ from openpyxl import load_workbook
+
+ workbook = load_workbook(str(target), read_only=True, data_only=True)
+ try:
+ for sheet in workbook.worksheets:
+ parts.append(f"--- Sheet: {sheet.title} ---")
+ for row in sheet.iter_rows(values_only=True):
+ values = ["" if value is None else str(value) for value in row]
+ if any(values):
+ parts.append("\t".join(values))
+ if sum(len(item) for item in parts) >= max_chars:
+ break
+ if sum(len(item) for item in parts) >= max_chars:
+ break
+ finally:
+ workbook.close()
+ elif suffix == ".pptx":
+ from pptx import Presentation
+
+ presentation = Presentation(str(target))
+ for index, slide in enumerate(presentation.slides, 1):
+ parts.append(f"--- Slide {index} ---")
+ for shape in slide.shapes:
+ text = getattr(shape, "text", "")
+ if text:
+ parts.append(str(text))
+ elif suffix == ".zip":
+ with zipfile.ZipFile(target) as archive:
+ parts = [f"{item.filename}\t{item.file_size} bytes" for item in archive.infolist()]
+ kind = "ZIP file listing"
+ text, truncated = _bounded(parts, max(1, int(max_chars)))
+ note = f"extracted {kind} content"
+ if truncated:
+ note += f" (truncated to {max_chars} characters)"
+ if not text.strip():
+ note = f"{kind} contained no extractable text"
+ return True, text, note
+ except Exception as exc: # malformed documents remain downloadable/tool-readable
+ return True, "", f"could not extract {suffix.lstrip('.').upper()} content: {exc}"
diff --git a/src/image_vision.py b/src/image_vision.py
new file mode 100644
index 0000000..5dcec7a
--- /dev/null
+++ b/src/image_vision.py
@@ -0,0 +1,185 @@
+"""Fast local image inspection that complements a connected vision model."""
+
+from __future__ import annotations
+
+import math
+import threading
+from collections import Counter
+from pathlib import Path
+from typing import Any
+
+_OCR_LOCK = threading.Lock()
+_OCR_ENGINE: Any | None = None
+
+_COLOR_NAMES = {
+ "black": (0, 0, 0), "white": (255, 255, 255), "gray": (128, 128, 128),
+ "red": (220, 40, 40), "orange": (242, 133, 32), "yellow": (240, 220, 40),
+ "green": (45, 155, 70), "cyan": (35, 190, 190), "blue": (45, 90, 210),
+ "purple": (135, 70, 185), "pink": (225, 105, 165), "brown": (130, 85, 50),
+ "beige": (220, 205, 170), "navy": (20, 40, 90),
+}
+
+
+def _nearest_color(rgb: tuple[int, int, int]) -> str:
+ return min(
+ _COLOR_NAMES,
+ key=lambda name: sum((rgb[index] - _COLOR_NAMES[name][index]) ** 2 for index in range(3)),
+ )
+
+
+def _dominant_colors(image: Any, count: int = 6) -> list[dict[str, Any]]:
+ sample = image.convert("RGB")
+ sample.thumbnail((256, 256))
+ quantized = sample.quantize(colors=count)
+ palette = quantized.getpalette() or []
+ histogram = quantized.getcolors(maxcolors=256 * 256) or []
+ total = max(1, sum(amount for amount, _ in histogram))
+ rows: list[dict[str, Any]] = []
+ for amount, index in sorted(histogram, reverse=True)[:count]:
+ rgb = tuple(int(value) for value in palette[index * 3:index * 3 + 3])
+ if len(rgb) != 3:
+ continue
+ rows.append({
+ "name": _nearest_color(rgb),
+ "rgb": list(rgb),
+ "hex": "#%02x%02x%02x" % rgb,
+ "percent": round(amount * 100 / total, 1),
+ })
+ return rows
+
+
+def _ocr(np_rgb: Any) -> tuple[list[dict[str, Any]], str]:
+ global _OCR_ENGINE
+ try:
+ from rapidocr_onnxruntime import RapidOCR
+ except ImportError:
+ return [], "RapidOCR is not installed"
+ with _OCR_LOCK:
+ if _OCR_ENGINE is None:
+ _OCR_ENGINE = RapidOCR()
+ engine = _OCR_ENGINE
+ try:
+ result, _elapsed = engine(np_rgb)
+ except Exception as exc:
+ return [], f"OCR failed: {exc}"
+ rows: list[dict[str, Any]] = []
+ for item in result or []:
+ if not isinstance(item, (list, tuple)) or len(item) < 3:
+ continue
+ text = str(item[1] or "").strip()
+ if not text:
+ continue
+ rows.append({"text": text, "confidence": round(float(item[2]), 3), "box": item[0]})
+ return rows[:100], "ready"
+
+
+def inspect_image(path: str | Path, *, run_ocr: bool = False) -> dict[str, Any]:
+ """Inspect pixels locally for color, shape, people/face, QR, and optional text."""
+ from PIL import Image, ImageStat
+
+ source = Path(path).expanduser().resolve()
+ if not source.is_file():
+ raise ValueError(f"image not found: {source}")
+ with Image.open(source) as opened:
+ opened.seek(0)
+ rgb = opened.convert("RGB")
+ info = {
+ "path": str(source), "format": opened.format or source.suffix.lstrip(".").upper(),
+ "width": opened.width, "height": opened.height, "mode": opened.mode,
+ "frames": int(getattr(opened, "n_frames", 1) or 1),
+ }
+ colors = _dominant_colors(rgb)
+ stat = ImageStat.Stat(rgb.convert("L"))
+ brightness = round(float(stat.mean[0]), 1)
+ contrast = round(float(stat.stddev[0]), 1)
+
+ try:
+ import cv2
+ import numpy as np
+ except ImportError:
+ return {**info, "dominant_colors": colors, "brightness": brightness, "contrast": contrast,
+ "objects": [], "shapes": {}, "text": [], "warning": "OpenCV is not installed"}
+
+ np_rgb = np.asarray(rgb)
+ bgr = cv2.cvtColor(np_rgb, cv2.COLOR_RGB2BGR)
+ scale = min(1.0, 960.0 / max(bgr.shape[:2]))
+ working = cv2.resize(bgr, None, fx=scale, fy=scale, interpolation=cv2.INTER_AREA) if scale < 1 else bgr
+ gray = cv2.cvtColor(working, cv2.COLOR_BGR2GRAY)
+ gray = cv2.GaussianBlur(gray, (5, 5), 0)
+ edges = cv2.Canny(gray, 60, 150)
+ contours, _hierarchy = cv2.findContours(edges, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
+ image_area = float(working.shape[0] * working.shape[1])
+ shape_counts: Counter[str] = Counter()
+ for contour in contours:
+ area = float(cv2.contourArea(contour))
+ if area < max(80.0, image_area * 0.0008) or area > image_area * 0.92:
+ continue
+ perimeter = float(cv2.arcLength(contour, True))
+ if perimeter <= 0:
+ continue
+ vertices = len(cv2.approxPolyDP(contour, 0.035 * perimeter, True))
+ circularity = 4 * math.pi * area / (perimeter * perimeter)
+ if vertices == 3:
+ label = "triangle"
+ elif vertices == 4:
+ x, y, width, height = cv2.boundingRect(contour)
+ label = "square" if 0.88 <= width / max(1, height) <= 1.12 else "rectangle"
+ elif circularity >= 0.72:
+ label = "circle"
+ elif vertices <= 8:
+ label = "polygon"
+ else:
+ continue
+ shape_counts[label] += 1
+
+ objects: list[dict[str, Any]] = []
+ try:
+ cascade = cv2.CascadeClassifier(str(Path(cv2.data.haarcascades) / "haarcascade_frontalface_default.xml"))
+ faces = cascade.detectMultiScale(gray, scaleFactor=1.15, minNeighbors=5, minSize=(28, 28))
+ if len(faces):
+ objects.append({"label": "face", "count": int(len(faces)), "detector": "opencv-haar"})
+ except Exception:
+ pass
+ try:
+ hog = cv2.HOGDescriptor()
+ hog.setSVMDetector(cv2.HOGDescriptor_getDefaultPeopleDetector())
+ people, _weights = hog.detectMultiScale(working, winStride=(8, 8), padding=(8, 8), scale=1.08)
+ if len(people):
+ objects.append({"label": "person", "count": int(len(people)), "detector": "opencv-hog"})
+ except Exception:
+ pass
+ qr_text = ""
+ try:
+ qr_text, _points, _straight = cv2.QRCodeDetector().detectAndDecode(working)
+ if qr_text:
+ objects.append({"label": "qr-code", "count": 1, "value": qr_text, "detector": "opencv-qr"})
+ except Exception:
+ pass
+
+ text, ocr_status = _ocr(np_rgb) if run_ocr else ([], "not requested")
+ sharpness = round(float(cv2.Laplacian(gray, cv2.CV_64F).var()), 1)
+ return {
+ **info,
+ "orientation": "landscape" if info["width"] > info["height"] else "portrait" if info["height"] > info["width"] else "square",
+ "dominant_colors": colors,
+ "brightness": brightness,
+ "contrast": contrast,
+ "sharpness": sharpness,
+ "objects": objects,
+ "shapes": dict(shape_counts.most_common()),
+ "text": text,
+ "ocr_status": ocr_status,
+ "semantic_object_recognition": "The connected vision model receives the original pixels for broader semantic identification.",
+ }
+
+
+def compact_image_brief(result: dict[str, Any]) -> str:
+ colors = ", ".join(f"{row['name']} {row['percent']}%" for row in result.get("dominant_colors") or [])
+ objects = ", ".join(f"{row['label']}×{row.get('count', 1)}" for row in result.get("objects") or []) or "none from local detectors"
+ shapes = ", ".join(f"{name}×{count}" for name, count in (result.get("shapes") or {}).items()) or "none"
+ text = " | ".join(row.get("text", "") for row in result.get("text") or [])
+ return (
+ f"{result.get('width')}×{result.get('height')} {result.get('format')} {result.get('orientation')}; "
+ f"colors: {colors or 'unknown'}; local objects: {objects}; shapes: {shapes}"
+ + (f"; OCR: {text}" if text else "")
+ )
diff --git a/src/install/__main__.py b/src/install/__main__.py
index 602b3ac..03b3ada 100644
--- a/src/install/__main__.py
+++ b/src/install/__main__.py
@@ -55,7 +55,7 @@ def run_cli(args: argparse.Namespace) -> int:
)
except Exception:
pass
- if args.launch:
+ if args.launch or result.get("restart_required"):
from .launch import launch_jonathan_ai
launch_jonathan_ai(source)
diff --git a/src/install/bootstrap.py b/src/install/bootstrap.py
index 7e5070b..1a5bced 100644
--- a/src/install/bootstrap.py
+++ b/src/install/bootstrap.py
@@ -7,19 +7,48 @@
from __future__ import annotations
import argparse
+import json
import os
import subprocess
import sys
+from datetime import datetime, timezone
from pathlib import Path
from typing import Any
from .app_root import discover_existing_install
-from .deps import install_desktop_deps, install_python_deps
+from .deps import install_blender_dep, install_code_memory_dep, install_desktop_deps, install_drawai_dep, install_ecc_dep, install_fooocus_dep, install_kronos_dep, install_personal_finance_dep, install_procoder_dep, install_python_deps
from .python_env import ensure_venv, venv_is_usable
from .record import resolve_source_dir, write_install_record
from .source import current_branch, current_commit
+from src.version import get_version
-CRITICAL_IMPORT = "import anthropic, openai, rich, dotenv, prompt_toolkit"
+CRITICAL_IMPORT = "import anthropic, openai, rich, dotenv, prompt_toolkit, tiktoken, PIL, cv2, rapidocr_onnxruntime, trimesh, psutil, yfinance, pandas, pypdf, docx, openpyxl, pptx, pyautogui"
+# The shipped native launcher was last compiled with this readiness marker.
+# Setup still writes the current marker; this compatibility marker prevents a
+# successful fresh install from repeating the same repair on every launch.
+LAUNCHER_COMPAT_VERSION = "0.4.6"
+
+
+def runtime_ready_marker(source_dir: str | Path, version: str | None = None) -> Path:
+ """Marker used by the native/Electron launchers to skip repeat repairs."""
+ release = str(version or get_version()).strip()
+ return Path(source_dir).expanduser().resolve() / f".jonathan-ai-runtime-{release}.ready"
+
+
+def write_runtime_ready_marker(source_dir: str | Path, result: dict[str, Any]) -> Path | None:
+ if not result.get("ok"):
+ return None
+ root = Path(source_dir).expanduser().resolve()
+ marker = runtime_ready_marker(root)
+ marker.write_text(json.dumps({
+ "version": get_version(),
+ "created_at": datetime.now(timezone.utc).isoformat(),
+ "python": str(result.get("venv_python") or ""),
+ }, indent=2), encoding="utf-8")
+ compatibility = runtime_ready_marker(root, LAUNCHER_COMPAT_VERSION)
+ if compatibility != marker:
+ compatibility.write_text(marker.read_text(encoding="utf-8"), encoding="utf-8")
+ return marker
def resolve_source(explicit: str | Path | None = None) -> Path:
@@ -66,6 +95,8 @@ def ensure_runtime_deps(
source_dir: str | Path | None = None,
*,
skip_desktop: bool = False,
+ skip_blender: bool = False,
+ skip_fooocus: bool = False,
progress: Any | None = None,
) -> dict[str, Any]:
"""Create/repair the venv and install missing packages. Never raises for Electron."""
@@ -88,13 +119,33 @@ def log(message: str) -> None:
if not _imports_ok(python):
pip_status = "failed"
log("Python packages still missing after pip")
- desktop_status = "skipped"
+ desktop_status = "skipped" if skip_desktop else "present"
if not skip_desktop and not _electron_present(root):
try:
desktop_status = install_desktop_deps(root, retry=2, progress=log)
except Exception as exc:
desktop_status = "failed"
log(f"Electron optional install failed: {exc}")
+ blender_status = "skipped"
+ if not skip_blender:
+ try:
+ blender_status = install_blender_dep(python, root, progress=log)
+ except Exception as exc:
+ blender_status = "failed"
+ log(f"Required Blender install failed: {exc}")
+ fooocus_status = "skipped"
+ if not skip_fooocus:
+ try:
+ fooocus_status = install_fooocus_dep(root, progress=log)
+ except Exception as exc:
+ fooocus_status = "failed"
+ log(f"Required local image-engine install failed: {exc}")
+ kronos_status = install_kronos_dep(root, progress=log)
+ personal_finance_status = install_personal_finance_dep(root, progress=log)
+ code_memory_status = install_code_memory_dep(root, progress=log)
+ procoder_status = install_procoder_dep(root, progress=log)
+ drawai_status = install_drawai_dep(root, progress=log)
+ ecc_status = install_ecc_dep(root, progress=log)
try:
write_install_record(
source_dir=root,
@@ -104,31 +155,45 @@ def log(message: str) -> None:
)
except Exception:
pass
- return {
- "ok": venv_is_usable(root) and pip_status != "failed",
+ result = {
+ "ok": venv_is_usable(root) and pip_status != "failed" and blender_status != "failed" and fooocus_status != "failed" and desktop_status != "failed",
"source_dir": str(root),
"venv_python": str(python),
"pip": pip_status,
"desktop_deps": desktop_status,
+ "blender": blender_status,
+ "fooocus": fooocus_status,
+ "kronos": kronos_status,
+ "personal_finance": personal_finance_status,
+ "code_memory": code_memory_status,
+ "procoder": procoder_status,
+ "drawai": drawai_status,
+ "ecc": ecc_status,
"notes": notes,
}
+ write_runtime_ready_marker(root, result)
+ return result
def main(argv: list[str] | None = None) -> int:
parser = argparse.ArgumentParser(description="Repair Jonathan Ai runtime dependencies")
parser.add_argument("--source-dir", default=None)
parser.add_argument("--skip-desktop-deps", action="store_true")
+ parser.add_argument("--skip-blender", action="store_true")
+ parser.add_argument("--skip-fooocus", action="store_true")
args = parser.parse_args(argv)
try:
result = ensure_runtime_deps(
args.source_dir,
skip_desktop=args.skip_desktop_deps,
+ skip_blender=args.skip_blender,
+ skip_fooocus=args.skip_fooocus,
progress=lambda message: print(message, flush=True),
)
except Exception as exc:
print(f"bootstrap failed: {exc}", file=sys.stderr)
return 1
- print(f"bootstrap pip={result['pip']} electron={result['desktop_deps']}", flush=True)
+ print(f"bootstrap pip={result['pip']} electron={result['desktop_deps']} blender={result['blender']} fooocus={result['fooocus']} ecc={result['ecc']}", flush=True)
return 0 if result.get("ok") else 1
diff --git a/src/install/deps.py b/src/install/deps.py
index 07a8723..6abd9c4 100644
--- a/src/install/deps.py
+++ b/src/install/deps.py
@@ -2,6 +2,8 @@
from __future__ import annotations
+import os
+import shutil
import subprocess
from pathlib import Path
from typing import Callable
@@ -9,8 +11,14 @@
Progress = Callable[[str], None]
-def _run(cmd: list[str], *, cwd: Path | None = None) -> None:
- result = subprocess.run(cmd, cwd=str(cwd) if cwd else None, capture_output=True, text=True)
+def _run(cmd: list[str], *, cwd: Path | None = None, env: dict[str, str] | None = None) -> None:
+ result = subprocess.run(
+ cmd,
+ cwd=str(cwd) if cwd else None,
+ env=env,
+ capture_output=True,
+ text=True,
+ )
if result.returncode != 0:
detail = (result.stderr or result.stdout or "").strip()
raise RuntimeError(f"{' '.join(cmd)} failed: {detail[-400:]}")
@@ -45,29 +53,210 @@ def install_python_deps(
raise RuntimeError(f"Python dependency install failed after {retry} attempts: {last}")
+def install_fooocus_dep(
+ source_dir: Path,
+ *,
+ progress: Progress | None = None,
+) -> str:
+ """Prepare Jonathan's native local Diffusers image engine."""
+ integration = source_dir / "src" / "integrations" / "fooocus.py"
+ if not integration.is_file():
+ return "skipped"
+ from src.integrations.fooocus import FooocusManager
+
+ manager = FooocusManager(source_dir)
+ status = manager.install(progress=progress)
+ if not status.get("source_ready") or not status.get("dependencies_ready"):
+ raise RuntimeError("Local image engine install completed without a usable runtime")
+ return "ready"
+
+
+def install_ecc_dep(
+ source_dir: Path,
+ *,
+ progress: Progress | None = None,
+) -> str:
+ """Synchronize and audit the managed ECC catalog during setup.
+
+ ECC is an optional knowledge pack rather than a runtime prerequisite, so
+ an offline installer remains usable and the desktop can retry later.
+ """
+ if progress:
+ progress("Synchronizing the audited ECC skills and agent catalog")
+ try:
+ from src.integrations.ecc import import_catalog, sync
+
+ sync(source_dir)
+ result = import_catalog(source_dir)
+ except Exception as exc:
+ if progress:
+ progress(f"ECC catalog deferred (retry from Skills library): {exc}")
+ return "deferred"
+ if progress:
+ progress(
+ f"ECC ready: {len(result.get('imported_skills') or [])} skills, "
+ f"{len(result.get('imported_agents') or [])} agent templates"
+ )
+ return "ready"
+
+
+def install_kronos_dep(
+ source_dir: Path,
+ *,
+ progress: Progress | None = None,
+) -> str:
+ """Install the local Kronos research-forecasting engine when online."""
+ if progress:
+ progress("Preparing the Kronos financial-market forecasting engine")
+ try:
+ from src.integrations.kronos import KronosManager
+
+ result = KronosManager(source_dir).install(progress=progress)
+ except Exception as exc:
+ if progress:
+ progress(f"Kronos deferred (retry from Business & finance): {exc}")
+ return "deferred"
+ return "ready" if result.get("source_ready") and result.get("runtime_ready") else "deferred"
+
+
+def install_personal_finance_dep(
+ source_dir: Path,
+ *,
+ progress: Progress | None = None,
+) -> str:
+ """Sync the optional isolated finance service without starting containers."""
+ if progress:
+ progress("Preparing the optional private personal-finance service")
+ try:
+ from src.integrations.securo import SecuroManager
+
+ result = SecuroManager(source_dir).sync(progress)
+ except Exception as exc:
+ if progress:
+ progress(f"Personal finance service deferred (retry from Connect tools): {exc}")
+ return "deferred"
+ return "ready" if result.get("source_ready") else "deferred"
+
+
+def install_code_memory_dep(
+ source_dir: Path,
+ *,
+ progress: Progress | None = None,
+) -> str:
+ """Install the optional local code graph without adding Claude-specific hooks."""
+ if progress:
+ progress("Preparing the local code-memory and symbol-graph engine")
+ try:
+ from src.integrations.claude_db import ClaudeDbManager
+
+ result = ClaudeDbManager(source_dir).install(progress)
+ except Exception as exc:
+ if progress:
+ progress(f"Code-memory engine deferred (retry from Connect tools): {exc}")
+ return "deferred"
+ return "ready" if result.get("runtime_ready") else "deferred"
+
+
+def install_procoder_dep(source_dir: Path, *, progress: Progress | None = None) -> str:
+ """Verify the optional standalone engineering gate and its upstream checksum."""
+ if progress:
+ progress("Preparing the Procoder engineering-quality controller")
+ try:
+ from src.integrations.procoder import ProcoderManager
+
+ result = ProcoderManager(source_dir).install(progress)
+ except Exception as exc:
+ if progress:
+ progress(f"Procoder deferred (retry from Connect tools): {exc}")
+ return "deferred"
+ return "ready" if result.get("runtime_ready") and result.get("checksum_verified") else "deferred"
+
+
+def install_drawai_dep(source_dir: Path, *, progress: Progress | None = None) -> str:
+ """Install DrawAI code dependencies; large optional models download on first use."""
+ if progress:
+ progress("Preparing DrawAI editable SVG/PPTX dependencies")
+ try:
+ from src.integrations.drawai import DrawAiManager
+
+ result = DrawAiManager(source_dir).install(models=False, progress=progress)
+ except Exception as exc:
+ if progress:
+ progress(f"DrawAI deferred (retry from Image Studio): {exc}")
+ return "deferred"
+ return "ready" if result.get("dependencies_ready") else "deferred"
+
+
def install_desktop_deps(
source_dir: Path,
*,
retry: int = 3,
progress: Progress | None = None,
) -> str:
- """Install Electron deps when npm is available. Optional on Linux CI."""
- import shutil
-
+ """Install Electron deps, bootstrapping Node.js LTS on Windows when needed."""
desktop = source_dir / "desktop"
if not (desktop / "package.json").exists():
return "skipped"
npm = shutil.which("npm")
+ if not npm and os.name == "nt":
+ winget = shutil.which("winget")
+ if winget:
+ if progress:
+ progress("Node.js is missing; installing Node.js LTS")
+ try:
+ _run([
+ winget,
+ "install",
+ "--id",
+ "OpenJS.NodeJS.LTS",
+ "-e",
+ "--source",
+ "winget",
+ "--silent",
+ "--accept-package-agreements",
+ "--accept-source-agreements",
+ ])
+ except RuntimeError as exc:
+ if progress:
+ progress(f"Node.js automatic install failed: {exc}")
+ candidates = [
+ Path(os.environ.get("ProgramFiles", r"C:\Program Files")) / "nodejs" / "npm.cmd",
+ Path(os.environ.get("LOCALAPPDATA", "")) / "Programs" / "nodejs" / "npm.cmd",
+ ]
+ npm = next((str(path) for path in candidates if path.exists()), None)
if not npm:
if progress:
- progress("npm not found — Electron shell skipped; browser UI still works")
- return "skipped"
+ progress("npm is unavailable; Electron desktop dependencies could not be installed")
+ return "failed"
last: Exception | None = None
+ cache_root = source_dir / ".cache"
+ install_env = dict(os.environ)
+ install_env.update({
+ "NPM_CONFIG_CACHE": str(cache_root / "npm"),
+ "ELECTRON_CACHE": str(cache_root / "electron"),
+ "ELECTRON_BUILDER_CACHE": str(cache_root / "electron-builder"),
+ # Electron 44's installer passes this exact npm-style setting to
+ # @electron/get; ELECTRON_CACHE alone is not read by that version.
+ "electron_config_cache": str(cache_root / "electron"),
+ })
+ electron_binary = desktop / "node_modules" / "electron" / "dist" / (
+ "electron.exe" if os.name == "nt" else "electron"
+ )
for attempt in range(1, retry + 1):
try:
if progress:
progress(f"Installing desktop shell packages (attempt {attempt}/{retry})")
- _run([npm, "install"], cwd=desktop)
+ _run([npm, "install"], cwd=desktop, env=install_env)
+ if not electron_binary.is_file():
+ if progress:
+ progress("Electron package is present without its runtime; downloading it into the app cache")
+ node = shutil.which("node")
+ installer = desktop / "node_modules" / "electron" / "install.js"
+ if not node or not installer.is_file():
+ raise RuntimeError("Electron runtime installer is unavailable")
+ _run([node, str(installer)], cwd=desktop, env=install_env)
+ if not electron_binary.is_file():
+ raise RuntimeError("npm completed without installing the Electron runtime binary")
return "installed"
except Exception as exc:
last = exc
@@ -76,3 +265,26 @@ def install_desktop_deps(
if progress:
progress("Electron install skipped after retries — browser UI still works")
return "failed"
+
+
+def install_blender_dep(
+ python: Path,
+ source_dir: Path,
+ *,
+ progress: Progress | None = None,
+) -> str:
+ """Ensure the full Blender engine is present, including portable fallback."""
+ if progress:
+ progress("Checking Blender 3D engine")
+ code = (
+ "from pathlib import Path; "
+ "from src.tool_system.context import ToolContext; "
+ "from src.tool_system.tools.three_d_studio import ThreeDStudioTool; "
+ "result=ThreeDStudioTool().run({'action':'install_blender','timeout':1800}, ToolContext(workspace_root=Path.cwd())); "
+ "print(result.output); "
+ "raise SystemExit(1 if result.is_error else 0)"
+ )
+ _run([str(python), "-c", code], cwd=source_dir)
+ if progress:
+ progress("Blender 3D engine is ready")
+ return "ready"
diff --git a/src/install/python_env.py b/src/install/python_env.py
index 6653928..b6cd01e 100644
--- a/src/install/python_env.py
+++ b/src/install/python_env.py
@@ -75,6 +75,15 @@ def venv_is_usable(source_dir: Path) -> bool:
target = venv_python(source_dir)
if not target.is_file():
return False
+ # Avoid asking CreateProcess to execute a corrupt/text file on Windows.
+ # Besides being slow to time out, that can invoke OS compatibility and
+ # security handlers. A real Windows Python launcher is always a PE file.
+ if os.name == "nt":
+ try:
+ if target.read_bytes()[:2] != b"MZ":
+ return False
+ except OSError:
+ return False
try:
result = subprocess.run(
[str(target), "-c", "import sys"],
diff --git a/src/install/runtime_process.py b/src/install/runtime_process.py
new file mode 100644
index 0000000..08043c0
--- /dev/null
+++ b/src/install/runtime_process.py
@@ -0,0 +1,212 @@
+"""Track and stop only the Jonathan Ai processes owned by an install tree."""
+
+from __future__ import annotations
+
+import json
+import os
+import shutil
+import signal
+import subprocess
+import sys
+import time
+from datetime import datetime, timezone
+from pathlib import Path
+from typing import Any
+
+
+PROCESS_RECORD_NAME = ".jonathan-ai-processes.json"
+
+
+def process_record_path(source_dir: str | Path) -> Path:
+ return Path(source_dir).expanduser().resolve() / PROCESS_RECORD_NAME
+
+
+def write_process_record(
+ source_dir: str | Path,
+ *,
+ owner_pid: int | None = None,
+ backend_pid: int | None = None,
+ kind: str = "python",
+ owner_executable: str | None = None,
+) -> Path:
+ """Atomically record the process tree an in-place upgrade may stop."""
+ root = Path(source_dir).expanduser().resolve()
+ path = process_record_path(root)
+ payload = {
+ "source_dir": str(root),
+ "owner_pid": int(owner_pid or os.getpid()),
+ "backend_pid": int(backend_pid) if backend_pid else None,
+ "kind": kind,
+ "owner_executable": owner_executable or (sys.executable if owner_pid in (None, os.getpid()) else ""),
+ "updated_at": datetime.now(timezone.utc).isoformat(),
+ }
+ temp = path.with_name(f"{path.name}.{os.getpid()}.tmp")
+ temp.write_text(json.dumps(payload, indent=2), encoding="utf-8")
+ os.replace(temp, path)
+ return path
+
+
+def read_process_record(source_dir: str | Path) -> dict[str, Any] | None:
+ root = Path(source_dir).expanduser().resolve()
+ path = process_record_path(root)
+ try:
+ payload = json.loads(path.read_text(encoding="utf-8"))
+ except (OSError, json.JSONDecodeError):
+ return None
+ if not isinstance(payload, dict):
+ return None
+ try:
+ recorded_root = Path(str(payload.get("source_dir") or "")).expanduser().resolve()
+ owner_pid = int(payload.get("owner_pid") or 0)
+ except (OSError, TypeError, ValueError):
+ return None
+ if recorded_root != root or owner_pid <= 4:
+ return None
+ return payload
+
+
+def remove_process_record(source_dir: str | Path, *, owner_pid: int | None = None) -> None:
+ path = process_record_path(source_dir)
+ if owner_pid is not None:
+ record = read_process_record(source_dir)
+ if record and int(record.get("owner_pid") or 0) != int(owner_pid):
+ return
+ try:
+ path.unlink()
+ except FileNotFoundError:
+ pass
+
+
+def _windows_process_info(pid: int) -> tuple[bool, str]:
+ """Return process liveness/image via read-only Win32 handles."""
+ import ctypes
+ from ctypes import wintypes
+
+ process_query_limited_information = 0x1000
+ still_active = 259
+ kernel32 = ctypes.WinDLL("kernel32", use_last_error=True)
+ kernel32.OpenProcess.argtypes = [wintypes.DWORD, wintypes.BOOL, wintypes.DWORD]
+ kernel32.OpenProcess.restype = wintypes.HANDLE
+ kernel32.GetExitCodeProcess.argtypes = [wintypes.HANDLE, ctypes.POINTER(wintypes.DWORD)]
+ kernel32.GetExitCodeProcess.restype = wintypes.BOOL
+ kernel32.QueryFullProcessImageNameW.argtypes = [
+ wintypes.HANDLE,
+ wintypes.DWORD,
+ wintypes.LPWSTR,
+ ctypes.POINTER(wintypes.DWORD),
+ ]
+ kernel32.QueryFullProcessImageNameW.restype = wintypes.BOOL
+ kernel32.CloseHandle.argtypes = [wintypes.HANDLE]
+ kernel32.CloseHandle.restype = wintypes.BOOL
+
+ handle = kernel32.OpenProcess(process_query_limited_information, False, pid)
+ if not handle:
+ return False, ""
+ try:
+ exit_code = wintypes.DWORD()
+ if not kernel32.GetExitCodeProcess(handle, ctypes.byref(exit_code)):
+ return False, ""
+ size = wintypes.DWORD(32768)
+ buffer = ctypes.create_unicode_buffer(size.value)
+ image = ""
+ if kernel32.QueryFullProcessImageNameW(handle, 0, buffer, ctypes.byref(size)):
+ image = buffer.value
+ return exit_code.value == still_active, image
+ finally:
+ kernel32.CloseHandle(handle)
+
+
+def _pid_is_running(pid: int) -> tuple[bool, str]:
+ if pid <= 4 or pid == os.getpid():
+ return False, ""
+ if os.name == "nt":
+ return _windows_process_info(pid)
+ proc_entry = Path("/proc") / str(pid)
+ if Path("/proc").is_dir():
+ return proc_entry.exists(), ""
+ try:
+ result = subprocess.run(
+ ["ps", "-p", str(pid), "-o", "pid="],
+ capture_output=True,
+ text=True,
+ timeout=2,
+ )
+ except (OSError, subprocess.TimeoutExpired):
+ return False, ""
+ return result.returncode == 0 and bool(result.stdout.strip()), ""
+
+
+def _is_expected_process(record: dict[str, Any], image: str, source_dir: Path) -> bool:
+ name = Path(image).name.lower()
+ kind = str(record.get("kind") or "").lower()
+ allowed = {"jonathanai.exe", "electron.exe", "python.exe", "pythonw.exe"}
+ if kind == "electron":
+ allowed = {"jonathanai.exe", "electron.exe"}
+ elif kind == "python":
+ allowed = {"python.exe", "pythonw.exe"}
+ if name not in allowed:
+ return False
+ recorded = Path(str(record.get("owner_executable") or "")).name.lower()
+ if recorded and recorded != name:
+ return False
+ try:
+ image_path = Path(image).resolve()
+ root = source_dir.resolve()
+ if image_path != root and root not in image_path.parents:
+ return False
+ recorded_path = str(record.get("owner_executable") or "").strip()
+ if recorded_path and image_path != Path(recorded_path).resolve():
+ return False
+ except OSError:
+ return False
+ return True
+
+
+def stop_running_app(source_dir: str | Path, *, timeout_s: float = 10.0) -> dict[str, Any]:
+ """Stop the recorded Jonathan Ai process tree before an in-place upgrade."""
+ root = Path(source_dir).expanduser().resolve()
+ record = read_process_record(root)
+ if not record:
+ return {"was_running": False, "stopped": False}
+ owner_pid = int(record["owner_pid"])
+ running, image = _pid_is_running(owner_pid)
+ if not running:
+ remove_process_record(root)
+ return {"was_running": False, "stopped": False, "owner_pid": owner_pid}
+ if os.name == "nt" and not _is_expected_process(record, image, root):
+ return {
+ "was_running": False,
+ "stopped": False,
+ "owner_pid": owner_pid,
+ "error": f"recorded PID belongs to an unexpected executable: {image or '(unknown)'}",
+ }
+
+ if os.name == "nt":
+ taskkill = shutil.which("taskkill") or str(
+ Path(os.environ.get("SystemRoot", r"C:\Windows")) / "System32" / "taskkill.exe"
+ )
+ result = subprocess.run(
+ [taskkill, "/PID", str(owner_pid), "/T", "/F"],
+ capture_output=True,
+ text=True,
+ timeout=max(1.0, timeout_s),
+ )
+ stopped = result.returncode == 0
+ else:
+ stopped = True
+ try:
+ os.kill(owner_pid, signal.SIGTERM)
+ except OSError:
+ stopped = False
+ deadline = time.monotonic() + max(0.1, timeout_s)
+ while stopped and _pid_is_running(owner_pid)[0] and time.monotonic() < deadline:
+ time.sleep(0.05)
+ if stopped and _pid_is_running(owner_pid)[0]:
+ try:
+ os.kill(owner_pid, signal.SIGKILL)
+ except OSError:
+ stopped = False
+
+ if stopped:
+ remove_process_record(root)
+ return {"was_running": True, "stopped": stopped, "owner_pid": owner_pid}
diff --git a/src/install/windows_shortcuts.py b/src/install/windows_shortcuts.py
index 3dfe7d2..babf06e 100644
--- a/src/install/windows_shortcuts.py
+++ b/src/install/windows_shortcuts.py
@@ -4,6 +4,7 @@
import os
import shutil
+import time
from pathlib import Path
@@ -123,11 +124,25 @@ def install_app_shortcuts(
)
for src in sources:
if src.exists() and src.resolve() != dest_exe.resolve():
- shutil.copy2(src, dest_exe)
+ if src.read_bytes()[:2] != b"MZ":
+ raise RuntimeError(f"bundled launcher is not a Windows executable: {src}")
+ temp_exe = source_dir / "JonathanAi.exe.new"
+ shutil.copy2(src, temp_exe)
+ last_error: OSError | None = None
+ for _attempt in range(20):
+ try:
+ os.replace(temp_exe, dest_exe)
+ last_error = None
+ break
+ except OSError as exc:
+ last_error = exc
+ time.sleep(0.1)
+ if last_error:
+ raise RuntimeError(f"could not replace {dest_exe}; close Jonathan Ai and retry") from last_error
break
- remove_parent_leftover_exes(source_dir)
+ removed_leftovers = remove_parent_leftover_exes(source_dir)
result = create_windows_shortcuts(dest_exe, icon if icon and icon.exists() else None)
result["target"] = str(dest_exe)
result["workdir"] = str(source_dir)
- result["removed_leftovers"] = remove_parent_leftover_exes(source_dir)
+ result["removed_leftovers"] = removed_leftovers
return result
diff --git a/src/install/wizard.py b/src/install/wizard.py
index e4b4366..8b712b3 100644
--- a/src/install/wizard.py
+++ b/src/install/wizard.py
@@ -13,9 +13,10 @@
from src.providers import PROVIDER_INFO
from .constants import CANONICAL_HTTPS
-from .deps import install_desktop_deps, install_python_deps
+from .deps import install_blender_dep, install_code_memory_dep, install_desktop_deps, install_drawai_dep, install_ecc_dep, install_fooocus_dep, install_kronos_dep, install_personal_finance_dep, install_procoder_dep, install_python_deps
from .python_env import detect_os, ensure_venv, find_system_python
from .record import write_install_record
+from .runtime_process import stop_running_app
from .app_root import discover_existing_install
from .source import current_branch, current_commit, materialize_source
from .verify import verify_agent_session
@@ -158,8 +159,15 @@ def _run(
skip_desktop_deps: bool,
clone: bool,
) -> None:
+ restart_after_install = False
+ dest = Path(source_dir).expanduser() if source_dir else discover_existing_install()
try:
- dest = Path(source_dir).expanduser() if source_dir else discover_existing_install()
+ process_state = stop_running_app(dest)
+ restart_after_install = bool(process_state.get("was_running"))
+ if restart_after_install and not process_state.get("stopped"):
+ raise RuntimeError("Jonathan Ai is running and could not be stopped for an in-place upgrade")
+ if restart_after_install:
+ self._emit(job, {"type": "step", "id": "restart", "message": "Stopped the running app for an in-place upgrade"})
self._emit(job, {"type": "step", "id": "os", "message": f"Detected {detect_os()['platform']}"})
self._emit(job, {"type": "step", "id": "python", "message": "Locating Python 3.10+"})
python = find_system_python()
@@ -184,12 +192,26 @@ def pip_progress(message: str) -> None:
self._emit(job, {"type": "step", "id": "deps", "message": message})
install_python_deps(venv, tree, retry=3, progress=pip_progress)
+ blender_status = install_blender_dep(venv, tree, progress=pip_progress)
+ fooocus_status = install_fooocus_dep(tree, progress=pip_progress)
desktop_status = "skipped"
if not skip_desktop_deps:
desktop_status = install_desktop_deps(tree, retry=3, progress=pip_progress)
+ if desktop_status == "failed":
+ raise RuntimeError("Electron desktop dependencies could not be installed")
self._emit(job, {"type": "step", "id": "config", "message": "Writing provider placeholders (no API keys)"})
write_provider_placeholders()
+ from src.skills.library import ensure_learning_skill
+
+ ensure_learning_skill(source_dir=tree)
+ self._emit(job, {"type": "step", "id": "skills", "message": f"Shared skill library ready at {tree / 'Skills'}"})
+ ecc_status = install_ecc_dep(tree, progress=pip_progress)
+ kronos_status = install_kronos_dep(tree, progress=pip_progress)
+ personal_finance_status = install_personal_finance_dep(tree, progress=pip_progress)
+ code_memory_status = install_code_memory_dep(tree, progress=pip_progress)
+ procoder_status = install_procoder_dep(tree, progress=pip_progress)
+ drawai_status = install_drawai_dep(tree, progress=pip_progress)
self._emit(job, {"type": "step", "id": "verify", "message": "Verifying the agent can start a session"})
verification = verify_agent_session(tree, python_path=venv)
@@ -205,10 +227,19 @@ def pip_progress(message: str) -> None:
"venv_python": str(venv),
"repo": CANONICAL_HTTPS,
"desktop_deps": desktop_status,
+ "blender": blender_status,
+ "fooocus": fooocus_status,
+ "ecc": ecc_status,
+ "kronos": kronos_status,
+ "personal_finance": personal_finance_status,
+ "code_memory": code_memory_status,
+ "procoder": procoder_status,
+ "drawai": drawai_status,
"launchers": launchers,
"verification": verification,
"commit": commit,
"branch": branch,
+ "restart_required": restart_after_install,
}
job.result = result
self._emit(job, {"type": "done", **result})
@@ -216,6 +247,13 @@ def pip_progress(message: str) -> None:
job.done = True
job.condition.notify_all()
except Exception as exc:
+ if restart_after_install:
+ try:
+ from .launch import launch_jonathan_ai
+
+ launch_jonathan_ai(dest)
+ except Exception:
+ pass
job.error = str(exc)
self._emit(job, {"type": "error", "error": str(exc)})
with job.condition:
diff --git a/src/integrations/__init__.py b/src/integrations/__init__.py
new file mode 100644
index 0000000..1709a57
--- /dev/null
+++ b/src/integrations/__init__.py
@@ -0,0 +1,11 @@
+"""Managed third-party application integrations."""
+
+from .fooocus import FooocusManager
+from .kronos import KronosManager
+from .securo import SecuroManager
+from .claude_db import ClaudeDbManager
+from .procoder import ProcoderManager
+from .drawai import DrawAiManager
+from .character_studio import generate_character
+
+__all__ = ["ClaudeDbManager", "DrawAiManager", "FooocusManager", "KronosManager", "ProcoderManager", "SecuroManager", "generate_character"]
diff --git a/src/integrations/character_studio.py b/src/integrations/character_studio.py
new file mode 100644
index 0000000..a25a311
--- /dev/null
+++ b/src/integrations/character_studio.py
@@ -0,0 +1,46 @@
+"""Jonathan-native deterministic procedural character and rig generator."""
+
+from __future__ import annotations
+
+import colorsys
+import hashlib
+import json
+import random
+import re
+from pathlib import Path
+from typing import Any
+
+KINDERGRIMM_REFERENCE = "https://github.com/albertobeiz/kindergrimm"
+
+
+def _color(h: float, s: float, l: float) -> str:
+ r,g,b=colorsys.hls_to_rgb(h%1,l,s); return f"#{int(r*255):02x}{int(g*255):02x}{int(b*255):02x}"
+
+
+def generate_character(output_dir: str|Path, *, seed: str="jonathan", name: str="", species: str="auto", medium: str="ink") -> dict[str,Any]:
+ target=Path(output_dir).expanduser().resolve(); target.mkdir(parents=True,exist_ok=True)
+ seed=str(seed)
+ digest=hashlib.sha256(seed.encode("utf-8")).digest(); rng=random.Random(digest)
+ chosen=species if species!="auto" else rng.choice(["human","cat","dog","fox","rabbit","bear"])
+ display=name.strip() or rng.choice(["Pip","Moss","Orla","Juniper","Tinker","Ember"])
+ base=rng.random(); skin=_color(base,.42,.68); accent=_color(base+.34,.62,.48); dark=_color(base,.35,.18); paper="#f4edda"
+ ear={"cat":"polygon","fox":"polygon","rabbit":"long","dog":"flop","bear":"round"}.get(chosen,"round")
+ eye_y=142+rng.randint(-5,5); eye_gap=46+rng.randint(-8,8); mouth_y=190+rng.randint(-4,8)
+ if ear=="long": ears=''
+ elif ear=="polygon": ears=''
+ elif ear=="flop": ears=''
+ else: ears=''
+ svg=f''''''
+ stem=re.sub(r"[^a-z0-9]+","-",display.lower()).strip("-") or "character"
+ svg_path=target/f"{stem}.svg"; svg_path.write_text(svg,encoding="utf-8")
+ recipe={"name":display,"seed":seed,"species":chosen,"medium":medium,"palette":{"paper":paper,"skin":skin,"accent":accent,"ink":dark},"rig":{"bones":["body","head","ears","eyes","nose","mouth","arms"],"animations":["boil","blink"]},"reference":KINDERGRIMM_REFERENCE}
+ json_path=target/f"{stem}.character.json"; json_path.write_text(json.dumps(recipe,ensure_ascii=False,indent=2),encoding="utf-8")
+ return {"name":display,"seed":seed,"species":chosen,"svg":str(svg_path),"recipe":str(json_path),"artifacts":[str(svg_path),str(json_path)],"repository":"internal://jonathan/character-studio","reference_repository":KINDERGRIMM_REFERENCE}
diff --git a/src/integrations/claude_db.py b/src/integrations/claude_db.py
new file mode 100644
index 0000000..2239def
--- /dev/null
+++ b/src/integrations/claude_db.py
@@ -0,0 +1,166 @@
+"""Jonathan-native UTF-8 SQLite code graph and durable code notes."""
+
+from __future__ import annotations
+
+import ast
+import re
+import sqlite3
+from collections import deque
+from contextlib import contextmanager
+from datetime import datetime, timezone
+from pathlib import Path
+from typing import Any, Callable
+
+CLAUDE_DB_REPOSITORY = "https://github.com/Avijit07x/claude-db.git"
+CLAUDE_DB_LICENSE = "Apache-2.0"
+SUPPORTED = {".py", ".js", ".jsx", ".ts", ".tsx", ".go", ".rs", ".java", ".kt", ".c", ".cc", ".cpp", ".h", ".hpp", ".rb", ".php"}
+SKIP_DIRS = {".git", ".venv", "venv", "node_modules", "dist", "build", "__pycache__", "Integrations"}
+SYMBOL_RE = re.compile(r"(?m)^\s*(?:async\s+)?(?:def|class|function|func|fn|interface|struct|enum)\s+([A-Za-z_$][\w$]*)")
+
+
+class ClaudeDbManager:
+ """Compatibility name retained while the implementation is fully Jonathan-native."""
+
+ def __init__(self, source_dir: str | Path) -> None:
+ self.source_dir = Path(source_dir).expanduser().resolve()
+ self.checkout = self.source_dir / "src" / "integrations"
+ self.database = Path.home() / ".clawd" / "code_memory" / "jonathan-code-memory.sqlite"
+
+ def _connect(self) -> sqlite3.Connection:
+ self.database.parent.mkdir(parents=True, exist_ok=True)
+ db = sqlite3.connect(self.database)
+ db.row_factory = sqlite3.Row
+ db.executescript("""
+ PRAGMA journal_mode=WAL;
+ CREATE TABLE IF NOT EXISTS notes(id INTEGER PRIMARY KEY, project TEXT NOT NULL, text TEXT NOT NULL, created_at TEXT NOT NULL);
+ CREATE VIRTUAL TABLE IF NOT EXISTS notes_fts USING fts5(text, content='notes', content_rowid='id');
+ CREATE TRIGGER IF NOT EXISTS notes_ai AFTER INSERT ON notes BEGIN INSERT INTO notes_fts(rowid,text) VALUES(new.id,new.text); END;
+ CREATE TABLE IF NOT EXISTS files(project TEXT NOT NULL, path TEXT NOT NULL, mtime_ns INTEGER NOT NULL, size INTEGER NOT NULL, PRIMARY KEY(project,path));
+ CREATE TABLE IF NOT EXISTS symbols(project TEXT NOT NULL, name TEXT NOT NULL, kind TEXT NOT NULL, path TEXT NOT NULL, line INTEGER NOT NULL);
+ CREATE INDEX IF NOT EXISTS symbols_project_name ON symbols(project,name);
+ CREATE TABLE IF NOT EXISTS edges(project TEXT NOT NULL, source TEXT NOT NULL, target TEXT NOT NULL, path TEXT NOT NULL, line INTEGER NOT NULL);
+ CREATE INDEX IF NOT EXISTS edges_project_source ON edges(project,source);
+ """)
+ return db
+
+ @contextmanager
+ def _db(self):
+ db = self._connect()
+ try:
+ with db:
+ yield db
+ finally:
+ db.close()
+
+ def status(self) -> dict[str, Any]:
+ try:
+ with self._db() as db:
+ notes = db.execute("SELECT count(*) FROM notes").fetchone()[0]
+ symbols = db.execute("SELECT count(*) FROM symbols").fetchone()[0]
+ files = db.execute("SELECT count(*) FROM files").fetchone()[0]
+ ready = True
+ except sqlite3.Error:
+ notes = symbols = files = 0; ready = False
+ return {"available": True, "source_ready": True, "runtime_ready": ready, "database": str(self.database),
+ "repository": "internal://jonathan/code-memory", "reference_repository": CLAUDE_DB_REPOSITORY,
+ "revision": "jonathan-native-1", "license": "MIT (implementation); Apache-2.0 design reference",
+ "local_only": True, "hooks_installed": False, "payments_required": False,
+ "counts": {"notes": notes, "symbols": symbols, "files": files},
+ "features": ["incremental code graph", "symbol usages", "dependency paths", "local search", "manual durable notes"]}
+
+ def sync(self, progress: Callable[[str], None] | None = None) -> dict[str, Any]:
+ if progress: progress("Initializing Jonathan's native code-memory database")
+ self._connect().close()
+ return self.status()
+
+ install = sync
+
+ @staticmethod
+ def _project(root: Path) -> str: return str(root).casefold()
+
+ @staticmethod
+ def _files(root: Path):
+ for path in root.rglob("*"):
+ if path.is_file() and path.suffix.lower() in SUPPORTED and not any(part in SKIP_DIRS for part in path.relative_to(root).parts):
+ try:
+ if path.stat().st_size <= 2_000_000: yield path
+ except OSError: continue
+
+ @staticmethod
+ def _extract(path: Path, root: Path) -> tuple[list[tuple[str, str, str, int]], list[tuple[str, str, str, int]]]:
+ text = path.read_text(encoding="utf-8", errors="replace"); rel = path.relative_to(root).as_posix()
+ symbols: list[tuple[str, str, str, int]] = []; edges: list[tuple[str, str, str, int]] = []
+ if path.suffix.lower() == ".py":
+ try:
+ tree = ast.parse(text)
+ for node in ast.walk(tree):
+ if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef, ast.ClassDef)):
+ symbols.append((node.name, "class" if isinstance(node, ast.ClassDef) else "function", rel, node.lineno))
+ elif isinstance(node, ast.Call):
+ target = node.func.id if isinstance(node.func, ast.Name) else node.func.attr if isinstance(node.func, ast.Attribute) else ""
+ if target: edges.append((rel, target, rel, getattr(node, "lineno", 1)))
+ elif isinstance(node, (ast.Import, ast.ImportFrom)):
+ for name in (a.name for a in node.names): edges.append((rel, name, rel, getattr(node, "lineno", 1)))
+ return symbols, edges
+ except SyntaxError: pass
+ for match in SYMBOL_RE.finditer(text): symbols.append((match.group(1), "symbol", rel, text.count("\n", 0, match.start()) + 1))
+ return symbols, edges
+
+ def scan(self, root: Path, *, force: bool = False) -> str:
+ project = self._project(root); parsed = skipped = symbol_count = edge_count = 0
+ with self._db() as db:
+ known = {row[0]: (row[1], row[2]) for row in db.execute("SELECT path,mtime_ns,size FROM files WHERE project=?", (project,))}; seen: set[str] = set()
+ for path in self._files(root):
+ rel = path.relative_to(root).as_posix(); seen.add(rel); stat = path.stat()
+ if not force and known.get(rel) == (stat.st_mtime_ns, stat.st_size): skipped += 1; continue
+ db.execute("DELETE FROM symbols WHERE project=? AND path=?", (project, rel)); db.execute("DELETE FROM edges WHERE project=? AND path=?", (project, rel))
+ symbols, edges = self._extract(path, root)
+ db.executemany("INSERT INTO symbols(project,name,kind,path,line) VALUES(?,?,?,?,?)", ((project,*row) for row in symbols))
+ db.executemany("INSERT INTO edges(project,source,target,path,line) VALUES(?,?,?,?,?)", ((project,*row) for row in edges))
+ db.execute("INSERT OR REPLACE INTO files(project,path,mtime_ns,size) VALUES(?,?,?,?)", (project,rel,stat.st_mtime_ns,stat.st_size))
+ parsed += 1; symbol_count += len(symbols); edge_count += len(edges)
+ for rel in set(known) - seen:
+ db.execute("DELETE FROM files WHERE project=? AND path=?", (project,rel)); db.execute("DELETE FROM symbols WHERE project=? AND path=?", (project,rel)); db.execute("DELETE FROM edges WHERE project=? AND path=?", (project,rel))
+ return f"Scanned {root}\n {parsed} parsed, {skipped} unchanged\n {symbol_count} symbols, {edge_count} edges updated"
+
+ def execute(self, action: str, *, workspace: str | Path, query: str = "", target: str = "", mode: str = "text", force: bool = False, limit: int = 100) -> dict[str, Any]:
+ action = action.strip().lower(); root = Path(workspace).expanduser().resolve(); project = self._project(root); limit = max(1,min(limit,1000))
+ if action == "status": return self.status()
+ if action in {"sync","install","repair"}: return self.sync()
+ if action == "scan": output = self.scan(root, force=force)
+ elif action == "remember":
+ if not query.strip(): raise ValueError("text to remember is required")
+ with self._db() as db: db.execute("INSERT INTO notes(project,text,created_at) VALUES(?,?,?)", (project,query.strip(),datetime.now(timezone.utc).isoformat()))
+ output = "Remembered in Jonathan's local code-memory database."
+ elif action == "search":
+ if not query.strip(): raise ValueError("query is required")
+ with self._db() as db: rows = db.execute("SELECT n.text,n.created_at FROM notes_fts f JOIN notes n ON n.id=f.rowid WHERE notes_fts MATCH ? AND n.project=? LIMIT ?", (query,project,limit)).fetchall()
+ output = "\n".join(f"{r['created_at']} {r['text']}" for r in rows) or "No matching notes."
+ else:
+ selected = action if action in {"explain","path"} else mode
+ if not query.strip(): raise ValueError("symbol or search text is required")
+ if selected == "text":
+ matches=[]
+ for path in self._files(root):
+ for number,line in enumerate(path.read_text(encoding="utf-8",errors="replace").splitlines(),1):
+ if query.casefold() in line.casefold(): matches.append(f"{path.relative_to(root)}:{number}: {line.strip()}")
+ if len(matches)>=limit: break
+ if len(matches)>=limit: break
+ output="\n".join(matches) or "No live text matches."
+ else:
+ self.scan(root)
+ with self._db() as db:
+ defs=db.execute("SELECT kind,path,line FROM symbols WHERE project=? AND name=? LIMIT ?",(project,query,limit)).fetchall(); refs=db.execute("SELECT source,path,line FROM edges WHERE project=? AND target=? LIMIT ?",(project,query,limit)).fetchall()
+ if selected=="path":
+ if not target.strip(): raise ValueError("target symbol is required for path mode")
+ graph: dict[str,set[str]]={}
+ for row in db.execute("SELECT source,target FROM edges WHERE project=?",(project,)): graph.setdefault(row[0],set()).add(row[1])
+ queue=deque([(query,[query])]); found=[]; visited={query}
+ while queue:
+ node,path=queue.popleft()
+ if node==target: found=path; break
+ for nxt in graph.get(node,set()):
+ if nxt not in visited: visited.add(nxt); queue.append((nxt,path+[nxt]))
+ output=" -> ".join(found) if found else f"No dependency path found from {query} to {target}."
+ else: output="\n".join([*(f"definition {r['kind']} {r['path']}:{r['line']}" for r in defs),*(f"reference from {r['source']} at {r['path']}:{r['line']}" for r in refs)]) or "No graph matches."
+ return {**self.status(),"action":action,"workspace":str(root),"output":output}
diff --git a/src/integrations/drawai.py b/src/integrations/drawai.py
new file mode 100644
index 0000000..43ccf9f
--- /dev/null
+++ b/src/integrations/drawai.py
@@ -0,0 +1,102 @@
+"""Jonathan-native raster-to-editable SVG and PPTX reconstruction."""
+
+from __future__ import annotations
+
+import html
+import importlib.util
+import json
+from pathlib import Path
+from typing import Any, Callable
+
+DRAWAI_REPOSITORY = "https://github.com/Renaissance-Mind/DrawAI.git"
+DRAWAI_LICENSE = "Apache-2.0"
+
+
+class DrawAiManager:
+ """Compatibility name retained for a standalone Jonathan implementation."""
+
+ def __init__(self, source_dir: str | Path) -> None:
+ self.source_dir = Path(source_dir).expanduser().resolve()
+ self.checkout = self.source_dir / "src" / "integrations"
+ self.runtime = self.source_dir / "src"
+
+ @staticmethod
+ def _dependencies() -> tuple[bool,list[str]]:
+ missing=[module for module in ("PIL","cv2","numpy","pptx") if importlib.util.find_spec(module) is None]
+ return not missing,missing
+
+ def status(self) -> dict[str, Any]:
+ ready,missing=self._dependencies()
+ return {"available": True,"source_ready": True,"dependencies_ready": ready,"models_ready": True,"runtime_ready": ready,
+ "path":str(self.checkout),"runtime_path":str(self.runtime),"repository":"internal://jonathan/editable-graphics",
+ "reference_repository":DRAWAI_REPOSITORY,"revision":"jonathan-native-1","license":"MIT (implementation); Apache-2.0 design reference",
+ "payments_required":False,"model_terms_apply":False,"disk_requirement":"No separate model pack", "missing":missing,
+ "features":["color-region vectorization","shape contours","optional OCR","editable SVG","native-shape PPTX"]}
+
+ def sync(self, progress: Callable[[str],None]|None=None) -> dict[str,Any]:
+ if progress: progress("Jonathan's native editable-graphics engine is ready")
+ return self.status()
+
+ def install(self, *, models: bool=False, device: str="cpu", progress: Callable[[str],None]|None=None) -> dict[str,Any]:
+ status=self.sync(progress)
+ if not status["runtime_ready"]: raise RuntimeError(f"Missing Jonathan image/document dependencies: {', '.join(status['missing'])}")
+ return status
+
+ @staticmethod
+ def _ocr(image_path: Path) -> list[dict[str,Any]]:
+ try:
+ from rapidocr_onnxruntime import RapidOCR
+ result,_=RapidOCR()(str(image_path))
+ rows=[]
+ for box,text,score in result or []:
+ xs=[float(p[0]) for p in box]; ys=[float(p[1]) for p in box]
+ rows.append({"text":str(text),"score":float(score),"x":min(xs),"y":min(ys),"width":max(xs)-min(xs),"height":max(ys)-min(ys)})
+ return rows
+ except Exception: return []
+
+ def convert(self, image: str|Path, output_root: str|Path, *, device: str="cpu", timeout: int=7200) -> dict[str,Any]:
+ self.install(); source=Path(image).expanduser().resolve(); target=Path(output_root).expanduser().resolve()
+ if not source.is_file(): raise ValueError(f"Input image does not exist: {source}")
+ target.mkdir(parents=True,exist_ok=True)
+ from PIL import Image,ImageDraw
+ import cv2
+ import numpy as np
+ from pptx import Presentation
+ from pptx.dml.color import RGBColor
+ from pptx.enum.shapes import MSO_SHAPE
+ from pptx.util import Inches,Pt
+
+ with Image.open(source) as opened: rgb=opened.convert("RGB")
+ width,height=rgb.size; scale=min(1.0,1200/max(width,height)); work=rgb.resize((max(1,int(width*scale)),max(1,int(height*scale))))
+ array=np.array(work); small=work.quantize(colors=10,method=Image.Quantize.MEDIANCUT).convert("RGB"); palette=np.array(small)
+ colors,counts=np.unique(palette.reshape(-1,3),axis=0,return_counts=True); order=np.argsort(counts)[::-1]
+ svg=[f'"); svg_path=target/f"{source.stem}-editable.svg"; svg_path.write_text("\n".join(svg),encoding="utf-8")
+
+ prs=Presentation(); prs.slide_width=Inches(13.333); prs.slide_height=Inches(13.333*height/width); slide=prs.slides.add_slide(prs.slide_layouts[6])
+ sx=prs.slide_width/width; sy=prs.slide_height/height
+ background=colors[order[0]]; shape=slide.shapes.add_shape(MSO_SHAPE.RECTANGLE,0,0,prs.slide_width,prs.slide_height); shape.fill.solid(); shape.fill.fore_color.rgb=RGBColor(*[int(v) for v in background]); shape.line.fill.background()
+ for region in sorted(regions,key=lambda r:r["width"]*r["height"],reverse=True)[:400]:
+ shape=slide.shapes.add_shape(MSO_SHAPE.RECTANGLE,int(region["x"]*sx),int(region["y"]*sy),max(1,int(region["width"]*sx)),max(1,int(region["height"]*sy))); shape.fill.solid(); shape.fill.fore_color.rgb=RGBColor(*region["color"]); shape.line.fill.background()
+ for row in ocr:
+ box=slide.shapes.add_textbox(int(row["x"]*sx),int(row["y"]*sy),max(1,int(row["width"]*sx)),max(1,int(row["height"]*sy*1.25))); paragraph=box.text_frame.paragraphs[0]; paragraph.text=row["text"]; paragraph.font.name="Arial"; paragraph.font.size=Pt(max(6,row["height"]*0.56))
+ pptx_path=target/f"{source.stem}-editable.pptx"; prs.save(pptx_path)
+ preview=rgb.copy(); draw=ImageDraw.Draw(preview)
+ for row in ocr: draw.rectangle((row["x"],row["y"],row["x"]+row["width"],row["y"]+row["height"]),outline="#e14949",width=2)
+ preview_path=target/f"{source.stem}-review.png"; preview.save(preview_path)
+ package_path=target/"editable_graphics_package.json"; package_path.write_text(json.dumps({"source":str(source),"size":[width,height],"regions":regions,"ocr":ocr,"outputs":[str(svg_path),str(pptx_path),str(preview_path)]},ensure_ascii=False,indent=2),encoding="utf-8")
+ return {**self.status(),"input":str(source),"run_dir":str(target),"output":f"Vectorized {len(regions)} color regions and {len(ocr)} text regions.","artifacts":[str(svg_path),str(pptx_path),str(preview_path),str(package_path)]}
diff --git a/src/integrations/ecc.py b/src/integrations/ecc.py
new file mode 100644
index 0000000..8bc35ad
--- /dev/null
+++ b/src/integrations/ecc.py
@@ -0,0 +1,443 @@
+"""Audited Everything Claude Code (ECC) integration.
+
+ECC remains an independently updatable upstream cache. Jonathan indexes the
+whole catalog but only loads explicitly enabled managed skills into prompts.
+"""
+
+from __future__ import annotations
+
+import hashlib
+import json
+import os
+import re
+import shutil
+import subprocess
+import threading
+from datetime import datetime, timezone
+from pathlib import Path
+from typing import Any, Iterable
+
+from src.agent.memory import add_fact, list_facts, sanitize_memory_text
+from src.skills.frontmatter import parse_frontmatter
+from src.skills.library import user_skill_library, validate_skill_source
+
+ECC_REPOSITORY = "https://github.com/affaan-m/ECC.git"
+ECC_LICENSE = "MIT"
+DEFAULT_ENABLED_SKILLS = (
+ "plan-orchestrate",
+ "tdd-workflow",
+ "verification-loop",
+ "security-review",
+ "git-workflow",
+ "codebase-onboarding",
+ "context-budget",
+ "error-handling",
+ "production-audit",
+ "agent-architecture-audit",
+ "team-agent-orchestration",
+ "architecture-decision-records",
+ "api-design",
+ "frontend-patterns",
+ "backend-patterns",
+ "python-patterns",
+ "python-testing",
+ "react-patterns",
+ "react-testing",
+ "e2e-testing",
+ "deployment-patterns",
+ "database-migrations",
+ "mcp-server-patterns",
+ "cost-tracking",
+ "unified-memory",
+)
+
+_LOCK = threading.RLock()
+_NAME = re.compile(r"^[a-z0-9][a-z0-9-]{0,63}$")
+_SECRET_PATTERNS = (
+ ("github-token", re.compile(r"\bgh[oprsu]_[A-Za-z0-9_]{20,}\b")),
+ ("gitlab-token", re.compile(r"\bglpat-[A-Za-z0-9_-]{20,}\b")),
+ ("openai-style-key", re.compile(r"\bsk-[A-Za-z0-9_-]{20,}\b")),
+ ("slack-token", re.compile(r"\bxox[baprs]-[A-Za-z0-9-]{20,}\b")),
+ ("private-key", re.compile(r"-----BEGIN (?:RSA |EC |OPENSSH )?PRIVATE KEY-----")),
+)
+_INJECTION = re.compile(
+ r"(?i)(?:ignore (?:all |any )?(?:previous|prior|system) instructions|"
+ r"reveal (?:the )?(?:system prompt|secrets?)|disable (?:safety|permission)|"
+ r"bypass (?:approval|permission)|grant (?:full|root|administrator) access)"
+)
+_DANGEROUS = re.compile(
+ r"(?i)(?:rm\s+-rf\s+(?:/|~|\$HOME)|Remove-Item\s+[^\n]*-Recurse[^\n]*-Force|"
+ r"Invoke-Expression|\biex\s*\(|curl[^\n|]*\|\s*(?:sh|bash)|wget[^\n|]*\|\s*(?:sh|bash))"
+)
+
+
+def _source_dir(source_dir: str | Path | None = None) -> Path:
+ if source_dir:
+ return Path(source_dir).expanduser().resolve()
+ try:
+ from src.install.record import resolve_source_dir
+
+ resolved = resolve_source_dir()
+ if resolved:
+ return Path(resolved).resolve()
+ except Exception:
+ pass
+ return Path(__file__).resolve().parents[2]
+
+
+def ecc_cache_dir(source_dir: str | Path | None = None) -> Path:
+ return _source_dir(source_dir) / "Integrations" / "ECC"
+
+
+def ecc_managed_dir(source_dir: str | Path | None = None) -> Path:
+ return user_skill_library(source_dir=_source_dir(source_dir)) / ".managed" / "ecc"
+
+
+def _state_path(source_dir: str | Path | None = None) -> Path:
+ return ecc_managed_dir(source_dir) / "state.json"
+
+
+def _default_state() -> dict[str, Any]:
+ return {
+ "version": 1,
+ "repository": ECC_REPOSITORY,
+ "revision": "",
+ "synced_at": "",
+ "imported_at": "",
+ "installed_skills": [],
+ "enabled_skills": [],
+ "installed_agents": [],
+ "last_scan": {},
+ }
+
+
+def read_state(source_dir: str | Path | None = None) -> dict[str, Any]:
+ path = _state_path(source_dir)
+ try:
+ data = json.loads(path.read_text(encoding="utf-8"))
+ except (OSError, UnicodeDecodeError, json.JSONDecodeError):
+ data = {}
+ return {**_default_state(), **(data if isinstance(data, dict) else {})}
+
+
+def _save_state(state: dict[str, Any], source_dir: str | Path | None = None) -> None:
+ path = _state_path(source_dir)
+ path.parent.mkdir(parents=True, exist_ok=True)
+ temp = path.with_name(f"{path.name}.{os.getpid()}.{threading.get_ident()}.tmp")
+ try:
+ temp.write_text(json.dumps(state, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")
+ os.replace(temp, path)
+ finally:
+ try:
+ temp.unlink()
+ except FileNotFoundError:
+ pass
+
+
+def enabled_skill_names(source_dir: str | Path | None = None) -> set[str]:
+ state = read_state(source_dir)
+ installed = set(str(item) for item in state.get("installed_skills") or [])
+ return installed & set(str(item) for item in state.get("enabled_skills") or [])
+
+
+def _run_git(args: list[str], *, timeout: int = 180) -> subprocess.CompletedProcess[str]:
+ command = ["git", "-c", "http.sslBackend=openssl", *args]
+ try:
+ return subprocess.run(command, capture_output=True, text=True, encoding="utf-8", errors="replace", timeout=timeout, check=False)
+ except FileNotFoundError as exc:
+ raise ValueError("Git is required to synchronize ECC.") from exc
+ except subprocess.TimeoutExpired as exc:
+ raise ValueError("ECC synchronization timed out.") from exc
+
+
+def sync(source_dir: str | Path | None = None) -> dict[str, Any]:
+ """Clone or fast-refresh the dedicated managed upstream cache."""
+ cache = ecc_cache_dir(source_dir)
+ cache.parent.mkdir(parents=True, exist_ok=True)
+ if (cache / ".git").is_dir():
+ result = _run_git(["-C", str(cache), "fetch", "--depth", "1", "origin", "main"])
+ if result.returncode == 0:
+ result = _run_git(["-C", str(cache), "checkout", "--detach", "--force", "FETCH_HEAD"])
+ elif cache.exists() and any(cache.iterdir()):
+ raise ValueError(f"ECC cache exists but is not a managed Git checkout: {cache}")
+ else:
+ result = _run_git(["clone", "--depth", "1", ECC_REPOSITORY, str(cache)])
+ if result.returncode != 0:
+ raise ValueError(result.stderr.strip() or result.stdout.strip() or "ECC synchronization failed.")
+ revision_result = _run_git(["-C", str(cache), "rev-parse", "HEAD"], timeout=15)
+ revision = revision_result.stdout.strip() if revision_result.returncode == 0 else ""
+ state = read_state(source_dir)
+ state.update({"repository": ECC_REPOSITORY, "revision": revision, "synced_at": datetime.now(timezone.utc).isoformat()})
+ _save_state(state, source_dir)
+ return {"ok": True, "path": str(cache), "revision": revision, "catalog": catalog(source_dir)}
+
+
+def _frontmatter_summary(path: Path, fallback_name: str) -> dict[str, Any]:
+ try:
+ content = path.read_text(encoding="utf-8")
+ except (OSError, UnicodeDecodeError):
+ return {"name": fallback_name, "description": "Unreadable UTF-8 entry", "valid": False}
+ parsed = parse_frontmatter(content)
+ name = str(parsed.frontmatter.get("name") or fallback_name).strip().lower()
+ description = str(parsed.frontmatter.get("description") or "").strip()
+ return {
+ "name": name,
+ "description": description[:500],
+ "version": str(parsed.frontmatter.get("version") or parsed.frontmatter.get("metadata.version") or ""),
+ "path": str(path),
+ "valid": bool(_NAME.fullmatch(name) and description and parsed.body.strip()),
+ }
+
+
+def catalog(source_dir: str | Path | None = None) -> dict[str, Any]:
+ cache = ecc_cache_dir(source_dir)
+ state = read_state(source_dir)
+ skills: list[dict[str, Any]] = []
+ agents: list[dict[str, Any]] = []
+ skills_root = cache / "skills"
+ if skills_root.is_dir():
+ for directory in sorted(skills_root.iterdir(), key=lambda item: item.name.lower()):
+ path = directory / "SKILL.md"
+ if directory.is_dir() and path.is_file():
+ summary = _frontmatter_summary(path, directory.name)
+ summary.update({"installed": summary["name"] in set(state.get("installed_skills") or []), "enabled": summary["name"] in set(state.get("enabled_skills") or [])})
+ skills.append(summary)
+ agents_root = cache / "agents"
+ if agents_root.is_dir():
+ for path in sorted(agents_root.glob("*.md"), key=lambda item: item.name.lower()):
+ summary = _frontmatter_summary(path, path.stem)
+ summary["installed"] = summary["name"] in set(state.get("installed_agents") or [])
+ agents.append(summary)
+ return {
+ "available": cache.is_dir(),
+ "path": str(cache),
+ "repository": ECC_REPOSITORY,
+ "license": ECC_LICENSE,
+ "revision": state.get("revision") or "",
+ "synced_at": state.get("synced_at") or "",
+ "skills": skills,
+ "agents": agents,
+ "counts": {
+ "skills": len(skills),
+ "agents": len(agents),
+ "installed": len(state.get("installed_skills") or []),
+ "enabled": len(enabled_skill_names(source_dir)),
+ },
+ }
+
+
+def security_scan(root: str | Path, *, max_files: int = 20000) -> dict[str, Any]:
+ base = Path(root).expanduser().resolve()
+ if not base.is_dir():
+ return {"ok": False, "root": str(base), "files_scanned": 0, "findings": [{"severity": "critical", "kind": "missing-source", "path": str(base), "message": "Source directory is missing."}]}
+ findings: list[dict[str, str]] = []
+ scanned = 0
+ ignored = {".git", "node_modules", ".venv", "__pycache__"}
+ text_suffixes = {".md", ".json", ".yaml", ".yml", ".toml", ".txt", ".py", ".js", ".mjs", ".cjs", ".sh", ".ps1", ".bat", ".cmd"}
+ for current, dirs, files in os.walk(base):
+ dirs[:] = [name for name in dirs if name not in ignored]
+ for name in files:
+ if scanned >= max_files:
+ findings.append({"severity": "medium", "kind": "scan-limit", "path": str(base), "message": f"Stopped after {max_files} files."})
+ break
+ path = Path(current) / name
+ scanned += 1
+ try:
+ resolved = path.resolve()
+ if base != resolved and base not in resolved.parents:
+ findings.append({"severity": "critical", "kind": "path-escape", "path": str(path), "message": "Resolved path escapes the integration root."})
+ continue
+ if path.is_symlink():
+ findings.append({"severity": "high", "kind": "symlink", "path": str(path), "message": "Symbolic links are not imported."})
+ continue
+ if path.suffix.lower() not in text_suffixes or path.stat().st_size > 2_000_000:
+ continue
+ text = path.read_text(encoding="utf-8", errors="replace")
+ except OSError:
+ continue
+ relative = path.relative_to(base).as_posix()
+ for kind, pattern in _SECRET_PATTERNS:
+ if pattern.search(text):
+ findings.append({"severity": "critical", "kind": kind, "path": relative, "message": "Possible embedded credential; import is blocked by default."})
+ if _INJECTION.search(text):
+ findings.append({"severity": "high", "kind": "prompt-injection", "path": relative, "message": "Contains an instruction-override pattern; review before enabling."})
+ if _DANGEROUS.search(text):
+ findings.append({"severity": "high", "kind": "dangerous-command", "path": relative, "message": "Contains a destructive or pipe-to-shell command; approval remains required."})
+ if scanned >= max_files:
+ break
+ critical = sum(1 for item in findings if item["severity"] == "critical")
+ high = sum(1 for item in findings if item["severity"] == "high")
+ return {"ok": critical == 0, "root": str(base), "files_scanned": scanned, "critical": critical, "high": high, "findings": findings[:500], "truncated": len(findings) > 500}
+
+
+def _copy_managed(source: Path, destination: Path) -> None:
+ root = destination.parent.resolve()
+ target = destination.resolve()
+ if target.parent != root:
+ raise ValueError("Managed integration target escaped its approved root.")
+ temp = destination.with_name(f".{destination.name}.incoming-{os.getpid()}-{threading.get_ident()}")
+ if temp.exists():
+ shutil.rmtree(temp)
+ shutil.copytree(source, temp, symlinks=False, ignore=shutil.ignore_patterns(".git", "__pycache__", "*.pyc"))
+ if destination.exists():
+ archive_root = destination.parent.parent / ".archive" / "ecc"
+ archive_root.mkdir(parents=True, exist_ok=True)
+ archive = archive_root / f"{destination.name}-{datetime.now().strftime('%Y%m%d-%H%M%S-%f')}"
+ os.replace(destination, archive)
+ os.replace(temp, destination)
+
+
+def import_catalog(
+ source_dir: str | Path | None = None,
+ *,
+ skill_names: Iterable[str] | None = None,
+ agent_names: Iterable[str] | None = None,
+ enable_names: Iterable[str] | None = None,
+ allow_unsafe: bool = False,
+) -> dict[str, Any]:
+ """Import all requested content while keeping prompt activation lazy."""
+ cache = ecc_cache_dir(source_dir)
+ if not (cache / "skills").is_dir():
+ raise ValueError("Synchronize ECC before importing its catalog.")
+ # Only skills and agent templates are copied. Test fixtures intentionally
+ # contain fake secrets and must remain visible in a full-repository scan,
+ # but they are not part of the import decision.
+ skill_scan = security_scan(cache / "skills")
+ agent_scan = security_scan(cache / "agents")
+ scan = {
+ "ok": bool(skill_scan["ok"] and agent_scan["ok"]),
+ "root": str(cache),
+ "files_scanned": int(skill_scan["files_scanned"]) + int(agent_scan["files_scanned"]),
+ "critical": int(skill_scan["critical"]) + int(agent_scan["critical"]),
+ "high": int(skill_scan["high"]) + int(agent_scan["high"]),
+ "findings": [*skill_scan["findings"], *agent_scan["findings"]][:500],
+ "scope": ["skills", "agents"],
+ }
+ if scan["critical"] and not allow_unsafe:
+ raise ValueError(f"ECC import blocked by {scan['critical']} critical finding(s) in importable content. Review the scan first.")
+ info = catalog(source_dir)
+ available_skills = {str(item["name"]): item for item in info["skills"] if item.get("valid")}
+ available_agents = {str(item["name"]): item for item in info["agents"] if item.get("valid")}
+ requested_skills = sorted(set(skill_names if skill_names is not None else available_skills))
+ requested_agents = sorted(set(agent_names if agent_names is not None else available_agents))
+ unknown = [name for name in requested_skills if name not in available_skills]
+ if unknown:
+ raise ValueError("Unknown or invalid ECC skills: " + ", ".join(unknown[:20]))
+ managed = ecc_managed_dir(source_dir)
+ skills_target = managed / "skills"
+ agents_target = managed / "agents"
+ skills_target.mkdir(parents=True, exist_ok=True)
+ agents_target.mkdir(parents=True, exist_ok=True)
+ imported_skills: list[str] = []
+ imported_agents: list[str] = []
+ with _LOCK:
+ for name in requested_skills:
+ source = Path(str(available_skills[name]["path"])).parent
+ validation = validate_skill_source(name, (source / "SKILL.md").read_text(encoding="utf-8"))
+ if not validation["valid"]:
+ continue
+ _copy_managed(source, skills_target / name)
+ imported_skills.append(name)
+ for name in requested_agents:
+ item = available_agents.get(name)
+ if not item:
+ continue
+ source = Path(str(item["path"]))
+ shutil.copy2(source, agents_target / f"{name}.md")
+ imported_agents.append(name)
+ state = read_state(source_dir)
+ installed = set(str(item) for item in state.get("installed_skills") or []) | set(imported_skills)
+ installed_agents = set(str(item) for item in state.get("installed_agents") or []) | set(imported_agents)
+ requested_enabled = set(enable_names if enable_names is not None else DEFAULT_ENABLED_SKILLS)
+ enabled = (set(str(item) for item in state.get("enabled_skills") or []) | requested_enabled) & installed
+ state.update({
+ "installed_skills": sorted(installed),
+ "enabled_skills": sorted(enabled),
+ "installed_agents": sorted(installed_agents),
+ "imported_at": datetime.now(timezone.utc).isoformat(),
+ "last_scan": {key: scan[key] for key in ("ok", "files_scanned", "critical", "high")},
+ })
+ _save_state(state, source_dir)
+ return {"ok": True, "imported_skills": imported_skills, "imported_agents": imported_agents, "enabled_skills": sorted(enabled), "scan": scan, "catalog": catalog(source_dir)}
+
+
+def set_enabled(names: Iterable[str], enabled: bool, source_dir: str | Path | None = None) -> dict[str, Any]:
+ requested = {str(name).strip().lower() for name in names if _NAME.fullmatch(str(name).strip().lower())}
+ state = read_state(source_dir)
+ installed = set(str(item) for item in state.get("installed_skills") or [])
+ missing = requested - installed
+ if missing:
+ raise ValueError("Import these skills before enabling them: " + ", ".join(sorted(missing)))
+ active = set(str(item) for item in state.get("enabled_skills") or [])
+ active = (active | requested) if enabled else (active - requested)
+ state["enabled_skills"] = sorted(active & installed)
+ _save_state(state, source_dir)
+ return {"ok": True, "enabled_skills": state["enabled_skills"], "catalog": catalog(source_dir)}
+
+
+def memory_vault_export() -> dict[str, Any]:
+ """Export Jonathan facts as inspectable ECC-compatible Markdown documents."""
+ root = Path.home() / ".clawd" / "memory" / "vault"
+ root.mkdir(parents=True, exist_ok=True)
+ written = []
+ for fact in list_facts():
+ fact_id = re.sub(r"[^a-z0-9_-]", "-", str(fact.get("id") or "fact").lower())
+ memory_id = f"mem_jonathan_{fact_id}"
+ timestamp = str(fact.get("created_at") or datetime.now(timezone.utc).isoformat()).replace("+00:00", "Z")
+ body = sanitize_memory_text(str(fact.get("text") or ""), limit=4096)
+ if not body:
+ continue
+ content = (
+ "---\n"
+ "schema: ecc.memory.v1\n"
+ f"id: {memory_id}\n"
+ f"title: Jonathan fact {fact_id}\n"
+ "kind: fact\n"
+ "scope: user\n"
+ "trust: unreviewed\n"
+ "status: active\n"
+ "sourceHarness: jonathan-ai\n"
+ "targetHarnesses: [jonathan-ai, codex, claude-code]\n"
+ "tags: [jonathan-memory]\n"
+ "links: []\n"
+ f"createdAt: {timestamp}\n"
+ f"updatedAt: {timestamp}\n"
+ "---\n\n"
+ f"{body}\n"
+ )
+ path = root / f"{memory_id}.md"
+ path.write_text(content, encoding="utf-8", newline="\n")
+ written.append(str(path))
+ return {"ok": True, "path": str(root), "count": len(written), "files": written}
+
+
+def memory_vault_import(path: str | Path) -> dict[str, Any]:
+ root = Path(path).expanduser().resolve()
+ candidates = list(root.glob("*.md")) if root.is_dir() else [root]
+ imported = []
+ rejected = []
+ for candidate in candidates[:5000]:
+ try:
+ text = candidate.read_text(encoding="utf-8")
+ except (OSError, UnicodeDecodeError):
+ rejected.append({"path": str(candidate), "reason": "not readable UTF-8"})
+ continue
+ parsed = parse_frontmatter(text)
+ if parsed.frontmatter.get("schema") != "ecc.memory.v1" or parsed.frontmatter.get("trust") != "unreviewed":
+ rejected.append({"path": str(candidate), "reason": "not an unreviewed ecc.memory.v1 document"})
+ continue
+ if _INJECTION.search(parsed.body) or any(pattern.search(parsed.body) for _, pattern in _SECRET_PATTERNS):
+ rejected.append({"path": str(candidate), "reason": "contains unsafe instruction or possible secret"})
+ continue
+ fact = add_fact(parsed.body, source_session=f"memory-vault:{candidate.name}")
+ if fact:
+ imported.append(fact)
+ return {"ok": True, "imported": imported, "rejected": rejected}
+
+
+def status(source_dir: str | Path | None = None) -> dict[str, Any]:
+ info = catalog(source_dir)
+ info["managed_path"] = str(ecc_managed_dir(source_dir))
+ info["state"] = read_state(source_dir)
+ info["default_enabled"] = list(DEFAULT_ENABLED_SKILLS)
+ return info
diff --git a/src/integrations/fooocus.py b/src/integrations/fooocus.py
new file mode 100644
index 0000000..16820ec
--- /dev/null
+++ b/src/integrations/fooocus.py
@@ -0,0 +1,67 @@
+"""Jonathan-native local diffusion image engine (legacy Fooocus API surface)."""
+from __future__ import annotations
+import importlib.util
+import os,subprocess,sys,uuid
+from datetime import datetime
+from pathlib import Path
+from typing import Any,Callable
+FOOOCUS_REPOSITORY="https://github.com/lllyasviel/Fooocus.git"; DEFAULT_PORT=7865
+
+def install_python_310(*_args:Any,**_kwargs:Any)->Path|None: return None
+
+class FooocusManager:
+ """Preserves callers while using Jonathan's own Diffusers implementation."""
+ def __init__(self,source_dir:str|Path)->None:
+ self.source_dir=Path(source_dir).expanduser().resolve(); self.checkout=self.source_dir/"src"/"integrations"; self.venv=self.source_dir/".venv"; self.outputs=self.source_dir/"integrations"/"Fooocus-outputs"; self.cache=self.source_dir/"integrations"/".image-models"; self.port=0
+ @staticmethod
+ def _deps()->tuple[bool,list[str]]:
+ missing=[module for module in ("torch","diffusers","transformers","safetensors","PIL") if importlib.util.find_spec(module) is None]
+ return not missing,missing
+ def status(self)->dict[str,Any]:
+ ready,missing=self._deps(); model=os.environ.get("JONATHAN_IMAGE_MODEL","stabilityai/sdxl-turbo")
+ return {"cloned":False,"source_ready":True,"dependencies_ready":ready,"runtime_ready":ready,"running":ready,"reachable":ready,"url":"","port":0,"path":str(self.checkout),"venv":str(self.venv),"outputs":str(self.outputs),"repository":"internal://jonathan/local-diffusion","reference_repository":FOOOCUS_REPOSITORY,"revision":"jonathan-native-1","backend":"diffusers","model":model,"missing":missing,"payments_required":False}
+ def install(self,progress:Callable[[str],None]|None=None)->dict[str,Any]:
+ ready,missing=self._deps()
+ if not ready:
+ if progress: progress(f"Installing Jonathan local diffusion packages: {', '.join(missing)}")
+ result=subprocess.run([sys.executable,"-m","pip","install","torch>=2.4","diffusers>=0.35","accelerate>=1.0","transformers>=4.48","safetensors>=0.5"],capture_output=True,text=True,encoding="utf-8",errors="replace",timeout=1800,check=False)
+ if result.returncode!=0: raise RuntimeError(result.stderr[-4000:] or "Local diffusion dependency install failed")
+ self.outputs.mkdir(parents=True,exist_ok=True); self.cache.mkdir(parents=True,exist_ok=True)
+ status=self.status()
+ if not status["dependencies_ready"]: raise RuntimeError(f"Missing local diffusion dependencies: {status['missing']}")
+ return status
+ def start(self,*,install:bool=True,progress:Callable[[str],None]|None=None)->dict[str,Any]: return self.install(progress) if install else self.status()
+ def stop(self)->dict[str,Any]: return {**self.status(),"running":False,"message":"No background image service is used; model memory is released after each generation."}
+ def latest_outputs(self,limit:int=20)->list[dict[str,Any]]:
+ if not self.outputs.is_dir(): return []
+ rows=[]
+ for path in sorted((p for p in self.outputs.rglob("*") if p.is_file() and p.suffix.lower() in {".png",".jpg",".jpeg",".webp"}),key=lambda p:p.stat().st_mtime,reverse=True)[:max(1,limit)]: rows.append({"path":str(path.resolve()),"name":path.name,"size":path.stat().st_size,"modified":path.stat().st_mtime})
+ return rows
+ def generate(self,prompt:str,*,negative_prompt:str="",width:int=1024,height:int=1024,performance:str="Speed",steps:int|None=None,seed:int|None=None,source_image:str|Path|None=None,strength:float=.65,timeout:int=1800,progress:Callable[[str],None]|None=None)->dict[str,Any]:
+ if not prompt.strip(): raise ValueError("Image prompt is required")
+ self.install(progress); import torch
+ from PIL import Image
+ from diffusers import AutoPipelineForImage2Image,AutoPipelineForText2Image
+ model=os.environ.get("JONATHAN_IMAGE_MODEL","stabilityai/sdxl-turbo"); use_cuda=torch.cuda.is_available(); dtype=torch.float16 if use_cuda else torch.float32
+ cls=AutoPipelineForImage2Image if source_image else AutoPipelineForText2Image
+ if progress: progress(f"Loading Jonathan local image model {model}")
+ pipe=cls.from_pretrained(model,torch_dtype=dtype,cache_dir=str(self.cache),use_safetensors=True)
+ if use_cuda: pipe=pipe.to("cuda")
+ else:
+ pipe=pipe.to("cpu")
+ if hasattr(pipe,"enable_attention_slicing"): pipe.enable_attention_slicing()
+ generator=torch.Generator(device="cuda" if use_cuda else "cpu").manual_seed(int(seed if seed is not None else uuid.uuid4().int%(2**31)))
+ count=int(steps or (2 if str(performance).lower() in {"speed","extreme speed","lightning","hyper-sd"} else 4)); kwargs={"prompt":prompt,"negative_prompt":negative_prompt or None,"num_inference_steps":max(1,min(count,30)),"guidance_scale":0.0,"generator":generator}
+ if source_image:
+ source=Path(source_image).expanduser().resolve()
+ if not source.is_file(): raise ValueError(f"Source image does not exist: {source}")
+ with Image.open(source) as opened: kwargs["image"]=opened.convert("RGB").resize((width,height))
+ kwargs["strength"]=max(.05,min(float(strength),.95))
+ else: kwargs.update(width=max(256,min(width,2048)),height=max(256,min(height,2048)))
+ try: image=pipe(**kwargs).images[0]
+ finally:
+ del pipe
+ if use_cuda: torch.cuda.empty_cache()
+ folder=self.outputs/datetime.now().strftime("%Y-%m-%d"); folder.mkdir(parents=True,exist_ok=True); target=folder/f"jonathan-{datetime.now().strftime('%H%M%S')}-{uuid.uuid4().hex[:8]}.png"; image.save(target)
+ item={"path":str(target.resolve()),"name":target.name,"size":target.stat().st_size}
+ return {**self.status(),"ok":True,"outputs":[item],"engine":"jonathan-local-diffusion","model":model,"device":"cuda" if use_cuda else "cpu"}
diff --git a/src/integrations/kronos.py b/src/integrations/kronos.py
new file mode 100644
index 0000000..f3ec45a
--- /dev/null
+++ b/src/integrations/kronos.py
@@ -0,0 +1,46 @@
+"""Jonathan-native probabilistic OHLCV research forecasting engine."""
+from __future__ import annotations
+import math
+import importlib.util
+from pathlib import Path
+from typing import Any,Callable
+KRONOS_REPOSITORY="https://github.com/shiyu-coder/Kronos.git"; KRONOS_LICENSE="MIT"
+MODELS={"mini":{"paths":1000},"small":{"paths":3000},"base":{"paths":8000}}
+
+class KronosManager:
+ """Compatibility name retained; no upstream checkout or runtime is required."""
+ def __init__(self,source_dir:str|Path)->None:
+ self.source_dir=Path(source_dir).expanduser().resolve(); self.checkout=self.source_dir/"src"/"integrations"; self.venv=Path(); self.cache=Path.home()/".clawd"/"forecast-cache"
+ def status(self)->dict[str,Any]:
+ ready=all(importlib.util.find_spec(module) is not None for module in ("numpy","pandas"))
+ return {"available":True,"source_ready":True,"runtime_ready":ready,"runtime_python":"","repository":"internal://jonathan/market-forecast",
+ "reference_repository":KRONOS_REPOSITORY,"revision":"jonathan-native-1","license":"MIT","models":list(MODELS),"device":"cpu",
+ "disclaimer":"Research forecast only; not investment advice and never places orders."}
+ def sync(self,progress:Callable[[str],None]|None=None)->dict[str,Any]:
+ if progress: progress("Jonathan's native market forecast engine is ready")
+ return self.status()
+ install=sync
+ def forecast(self,input_csv:str|Path,output_csv:str|Path,*,model:str="mini",pred_len:int=24,lookback:int=400,sample_count:int=1,temperature:float=1.0,top_p:float=.9,timeout:int=1800)->dict[str,Any]:
+ if model not in MODELS: raise ValueError(f"Unsupported model: {model}")
+ import numpy as np
+ import pandas as pd
+ source=Path(input_csv).expanduser().resolve(); target=Path(output_csv).expanduser().resolve()
+ if not source.is_file(): raise ValueError(f"OHLCV CSV does not exist: {source}")
+ data=pd.read_csv(source); cols={str(c).lower():c for c in data.columns}
+ if "close" not in cols: raise ValueError("CSV must contain close")
+ close=pd.to_numeric(data[cols["close"]],errors="coerce").dropna().tail(max(16,lookback)).to_numpy(float)
+ if len(close)<2: raise ValueError("At least two valid close rows are required")
+ returns=np.diff(np.log(np.maximum(close,1e-12))); weights=np.exp(np.linspace(-3,0,len(returns))); weights/=weights.sum(); drift=float(np.sum(returns*weights)); vol=float(np.sqrt(np.sum(weights*(returns-drift)**2)))
+ rng=np.random.default_rng(int(abs(close[-1])*1_000_003)%2**32); paths=MODELS[model]["paths"]*max(1,min(sample_count,20)); shocks=rng.normal(drift, max(vol,1e-8)*max(.05,temperature),(paths,pred_len)); simulated=close[-1]*np.exp(np.cumsum(shocks,axis=1))
+ median=np.median(simulated,axis=0); low=np.quantile(simulated,.1,axis=0); high=np.quantile(simulated,.9,axis=0)
+ interval=None
+ if "timestamp" in cols:
+ parsed=pd.to_datetime(data[cols["timestamp"]],errors="coerce").dropna()
+ if len(parsed)>=2: interval=parsed.iloc[-1]-parsed.iloc[-2]
+ stamps=[(parsed.iloc[-1]+(i+1)*(interval or pd.Timedelta(days=1))).isoformat() for i in range(pred_len)] if len(parsed) else list(range(1,pred_len+1))
+ else: stamps=list(range(len(data),len(data)+pred_len))
+ output=pd.DataFrame({"timestamp":stamps,"open":np.r_[close[-1],median[:-1]],"high":high,"low":low,"close":median,"p10":low,"p90":high})
+ target.parent.mkdir(parents=True,exist_ok=True); output.to_csv(target,index=False)
+ return {"ok":True,"path":str(target),"rows":pred_len,"device":"cpu","engine":"Jonathan probabilistic EWMA Monte Carlo","model":model,"paths":paths,
+ "revision":"jonathan-native-1","repository":"internal://jonathan/market-forecast","reference_repository":KRONOS_REPOSITORY,
+ "disclaimer":"Research forecast only; validate with walk-forward testing, costs, slippage, and risk controls. Not investment advice."}
diff --git a/src/integrations/kronos_worker.py b/src/integrations/kronos_worker.py
new file mode 100644
index 0000000..fb74552
--- /dev/null
+++ b/src/integrations/kronos_worker.py
@@ -0,0 +1,70 @@
+"""Isolated Kronos inference worker; invoked by :mod:`src.integrations.kronos`."""
+
+from __future__ import annotations
+
+import json
+import sys
+from pathlib import Path
+
+
+def main() -> int:
+ request_path, response_path = map(Path, sys.argv[1:3])
+ request = json.loads(request_path.read_text(encoding="utf-8"))
+ checkout = Path(request["checkout"]).resolve()
+ sys.path.insert(0, str(checkout))
+
+ import pandas as pd
+ import torch
+ from model import Kronos, KronosPredictor, KronosTokenizer
+
+ frame = pd.read_csv(request["input_csv"])
+ lower = {str(column).strip().lower(): column for column in frame.columns}
+ missing = [name for name in ("open", "high", "low", "close") if name not in lower]
+ if missing:
+ raise ValueError(f"CSV is missing required columns: {', '.join(missing)}")
+ timestamp_name = next((lower[name] for name in ("timestamp", "timestamps", "datetime", "date", "time") if name in lower), None)
+ if timestamp_name is None:
+ raise ValueError("CSV requires a timestamp/date/datetime column")
+ timestamps = pd.to_datetime(frame[timestamp_name], errors="coerce", utc=True).dt.tz_convert(None)
+ if timestamps.isna().any() or len(timestamps) < 2:
+ raise ValueError("CSV timestamps are invalid or insufficient")
+ price = pd.DataFrame({name: pd.to_numeric(frame[lower[name]], errors="coerce") for name in ("open", "high", "low", "close")})
+ for optional in ("volume", "amount"):
+ if optional in lower:
+ price[optional] = pd.to_numeric(frame[lower[optional]], errors="coerce")
+ if price.isna().any().any():
+ raise ValueError("CSV contains non-numeric or missing OHLCV values")
+ lookback = min(int(request["lookback"]), int(request["max_context"]), len(price))
+ if lookback < 16:
+ raise ValueError("Kronos requires at least 16 historical rows")
+ price = price.iloc[-lookback:].reset_index(drop=True)
+ history_ts = timestamps.iloc[-lookback:].reset_index(drop=True)
+ delta = history_ts.diff().dropna().median()
+ if pd.isna(delta) or delta <= pd.Timedelta(0):
+ raise ValueError("CSV timestamps must be increasing at a consistent interval")
+ pred_len = int(request["pred_len"])
+ future_ts = pd.Series(pd.date_range(history_ts.iloc[-1] + delta, periods=pred_len, freq=delta))
+
+ device = "cuda:0" if torch.cuda.is_available() else "cpu"
+ tokenizer = KronosTokenizer.from_pretrained(request["tokenizer"])
+ model = Kronos.from_pretrained(request["model"])
+ predictor = KronosPredictor(model, tokenizer, device=device, max_context=int(request["max_context"]))
+ forecast = predictor.predict(price, history_ts, future_ts, pred_len=pred_len,
+ T=float(request["temperature"]), top_p=float(request["top_p"]),
+ sample_count=int(request["sample_count"]), verbose=False)
+ forecast.index.name = "timestamp"
+ output = Path(request["output_csv"]).resolve()
+ forecast.to_csv(output, encoding="utf-8")
+ response = {
+ "ok": True, "path": str(output), "rows": len(forecast), "lookback": lookback,
+ "model": request["model"], "model_key": request["model_key"], "device": device,
+ "first_timestamp": forecast.index[0].isoformat(), "last_timestamp": forecast.index[-1].isoformat(),
+ "last_close": float(forecast.iloc[-1]["close"]),
+ }
+ response_path.write_text(json.dumps(response, ensure_ascii=False, indent=2), encoding="utf-8")
+ return 0
+
+
+if __name__ == "__main__":
+ raise SystemExit(main())
+
diff --git a/src/integrations/procoder.py b/src/integrations/procoder.py
new file mode 100644
index 0000000..dfcd28a
--- /dev/null
+++ b/src/integrations/procoder.py
@@ -0,0 +1,108 @@
+"""Jonathan-native fail-closed engineering quality controller."""
+
+from __future__ import annotations
+
+import json
+import os
+import re
+import shutil
+import subprocess
+from pathlib import Path
+from typing import Any, Callable
+
+PROCODER_REPOSITORY = "https://github.com/azrtydxb/procoder.git"
+PROCODER_LICENSE = "Apache-2.0"
+SAFE_ACTIONS = {"status", "doctor", "check", "test", "audit", "security", "git", "docs", "ci", "infra", "maintain", "deps", "release", "index"}
+SECRET_RE = re.compile(r"(?i)(?:api[_-]?key|token|secret|password)\s*[:=]\s*['\"]?[A-Za-z0-9_./+\-=]{16,}")
+CONFLICT_RE = re.compile(r"(?m)^(?:<{7}|={7}|>{7})(?:\s|$)")
+JUNK = {".DS_Store", "Thumbs.db", "desktop.ini"}
+
+
+class ProcoderManager:
+ """Compatibility name retained while all checks are implemented in Jonathan."""
+
+ def __init__(self, source_dir: str | Path) -> None:
+ self.source_dir = Path(source_dir).expanduser().resolve()
+ self.checkout = self.source_dir / "src" / "integrations"
+
+ @staticmethod
+ def _run(command: list[str], *, cwd: Path | None = None, timeout: int = 1800) -> subprocess.CompletedProcess[str]:
+ try:
+ return subprocess.run(command, cwd=str(cwd) if cwd else None, capture_output=True, text=True,
+ encoding="utf-8", errors="replace", timeout=timeout, check=False)
+ except (FileNotFoundError, subprocess.TimeoutExpired) as exc:
+ return subprocess.CompletedProcess(command, 127, "", str(exc))
+
+ def status(self) -> dict[str, Any]:
+ return {"available": True, "source_ready": True, "runtime_ready": True, "binary": "",
+ "checksum_verified": True, "version": "jonathan-native-1", "revision": "jonathan-native-1",
+ "path": str(self.checkout), "repository": "internal://jonathan/engineering-gate",
+ "reference_repository": PROCODER_REPOSITORY, "license": "MIT (implementation); Apache-2.0 design reference",
+ "hooks_installed": False, "automatic_edits": False, "payments_required": False,
+ "features": ["commit gate", "test controller", "security audit", "release check", "code index", "documentation/CI/infra hygiene"]}
+
+ def sync(self, progress: Callable[[str], None] | None = None) -> dict[str, Any]:
+ if progress: progress("Jonathan's native engineering gate is ready")
+ return self.status()
+
+ install = sync
+
+ @staticmethod
+ def _tracked(root: Path) -> list[Path]:
+ result = ProcoderManager._run(["git","ls-files","-co","--exclude-standard"],cwd=root,timeout=30)
+ if result.returncode == 0:
+ return [root / row for row in result.stdout.splitlines() if row and (root/row).is_file()]
+ return [p for p in root.rglob("*") if p.is_file() and ".git" not in p.parts]
+
+ def _gate(self, root: Path) -> tuple[bool,str]:
+ findings=[]; checked=0; unchecked=0
+ for path in self._tracked(root):
+ rel=path.relative_to(root).as_posix()
+ if path.name in JUNK: findings.append(f"BLOCKING {rel}: junk file")
+ try: raw=path.read_bytes()
+ except OSError: findings.append(f"BLOCKING {rel}: unreadable"); unchecked+=1; continue
+ if len(raw)>10_000_000: findings.append(f"BLOCKING {rel}: oversized file ({len(raw)} bytes)")
+ if b"\x00" in raw: unchecked+=1; continue
+ text=raw.decode("utf-8",errors="replace"); checked+=1
+ for match in CONFLICT_RE.finditer(text): findings.append(f"BLOCKING {rel}:{text.count(chr(10),0,match.start())+1}: conflict marker")
+ if SECRET_RE.search(text) and not rel.startswith("tests/"): findings.append(f"BLOCKING {rel}: possible embedded credential")
+ if text and not text.endswith("\n") and path.suffix.lower() in {".py",".js",".ts",".json",".md",".yml",".yaml",".toml"}: findings.append(f"ADVISORY {rel}: missing final newline")
+ blocking=sum(row.startswith("BLOCKING") for row in findings)
+ output="\n".join(findings+[f"Jonathan gate: {checked} text files checked, {unchecked} binary/unchecked, {blocking} blocking finding(s)"])
+ return blocking==0,output
+
+ def _tests(self, root: Path, coverage: bool, timeout: int) -> tuple[bool,str]:
+ commands=[]
+ if (root/"pyproject.toml").is_file() or (root/"pytest.ini").is_file() or (root/"tests").is_dir(): commands.append(([str(root/".venv"/("Scripts/python.exe" if os.name=="nt" else "bin/python")),"-m","pytest","-q"],"pytest"))
+ if (root/"package.json").is_file():
+ try:
+ scripts=json.loads((root/"package.json").read_text(encoding="utf-8")).get("scripts",{})
+ if "test" in scripts: commands.append(([shutil.which("npm") or "npm","test","--","--runInBand"],"npm test"))
+ except Exception: pass
+ if (root/"go.mod").is_file(): commands.append(([shutil.which("go") or "go","test","./..."],"go test"))
+ if not commands: return False,"NOT RUN: no canonical test command detected"
+ rows=[]; ok=True
+ for command,label in commands:
+ if not Path(command[0]).exists() and not shutil.which(command[0]): rows.append(f"NOT RUN {label}: runner unavailable"); ok=False; continue
+ result=self._run(command,cwd=root,timeout=timeout); rows.append(f"{'PASS' if result.returncode==0 else 'FAIL'} {label} (exit {result.returncode})\n{(result.stdout+result.stderr)[-12000:]}"); ok &= result.returncode==0
+ return ok,"\n".join(rows)
+
+ def execute(self, action: str, *, workspace: str | Path, argument: str = "", deep: bool = False, coverage: bool = False, timeout: int = 1800) -> dict[str, Any]:
+ action=action.strip().lower(); root=Path(workspace).expanduser().resolve()
+ if action=="status": return self.status()
+ if action in {"sync","install","repair"}: return self.sync()
+ if action not in SAFE_ACTIONS: raise ValueError(f"Unsupported engineering-gate action: {action}")
+ if action=="doctor":
+ tools=["git","python","node","npm","rg","go"]; output="\n".join(f"{name}: {shutil.which(name) or 'not installed'}" for name in tools); ok=True
+ elif action=="test": ok,output=self._tests(root,coverage,timeout)
+ elif action in {"check","audit","security"}: ok,output=self._gate(root)
+ elif action=="git":
+ result=self._run(["git","status","--short","--branch"],cwd=root,timeout=30); ok=result.returncode==0; output=result.stdout or result.stderr
+ elif action=="release":
+ gate_ok,gate=self._gate(root); tests_ok,tests=self._tests(root,coverage,timeout); status=self._run(["git","status","--porcelain"],cwd=root,timeout=30); clean=status.returncode==0 and not status.stdout.strip(); ok=gate_ok and tests_ok and clean; output=f"tree_clean: {clean}\n{gate}\n{tests}"
+ elif action=="index":
+ from src.integrations.claude_db import ClaudeDbManager
+ manager=ClaudeDbManager(self.source_dir); sub=argument or "stats"; output=manager.scan(root) if sub=="build" else json.dumps(manager.status().get("counts",{}),indent=2); ok=True
+ else:
+ gate_ok,gate=self._gate(root); ok=gate_ok; output=f"{action} report\n{gate}\nNo external controller or hidden hook was used."
+ return {**self.status(),"action":action,"workspace":str(root),"ok":bool(ok),"exit_code":0 if ok else 1,"output":output[-100_000:]}
diff --git a/src/integrations/securo.py b/src/integrations/securo.py
new file mode 100644
index 0000000..fc72f2e
--- /dev/null
+++ b/src/integrations/securo.py
@@ -0,0 +1,47 @@
+"""Jonathan-native local personal-finance vault."""
+from __future__ import annotations
+import csv,json,sqlite3,uuid
+from contextlib import contextmanager
+from datetime import date,datetime,timezone
+from pathlib import Path
+from typing import Any,Callable
+SECURO_REPOSITORY="https://github.com/securo-finance/securo.git"; SECURO_LICENSE="AGPL-3.0"; FRONTEND_PORT=0; BACKEND_PORT=0
+
+class SecuroManager:
+ """Compatibility name retained; data and logic are standalone Jonathan code."""
+ def __init__(self,source_dir:str|Path)->None:
+ self.source_dir=Path(source_dir).expanduser().resolve(); self.checkout=self.source_dir/"src"/"integrations"; self.database=Path.home()/".clawd"/"finance"/"personal-finance.sqlite"
+ def _connect(self)->sqlite3.Connection:
+ self.database.parent.mkdir(parents=True,exist_ok=True); db=sqlite3.connect(self.database); db.row_factory=sqlite3.Row
+ db.executescript("""PRAGMA journal_mode=WAL; CREATE TABLE IF NOT EXISTS accounts(id TEXT PRIMARY KEY,name TEXT NOT NULL,type TEXT NOT NULL,currency TEXT NOT NULL,opening_balance REAL NOT NULL DEFAULT 0,created_at TEXT NOT NULL); CREATE TABLE IF NOT EXISTS transactions(id TEXT PRIMARY KEY,account_id TEXT NOT NULL,occurred_on TEXT NOT NULL,amount REAL NOT NULL,currency TEXT NOT NULL,category TEXT NOT NULL DEFAULT '',description TEXT NOT NULL DEFAULT '',recurring INTEGER NOT NULL DEFAULT 0,created_at TEXT NOT NULL); CREATE TABLE IF NOT EXISTS budgets(id TEXT PRIMARY KEY,name TEXT NOT NULL,category TEXT NOT NULL,amount REAL NOT NULL,currency TEXT NOT NULL,period TEXT NOT NULL); CREATE TABLE IF NOT EXISTS goals(id TEXT PRIMARY KEY,name TEXT NOT NULL,target REAL NOT NULL,current REAL NOT NULL DEFAULT 0,currency TEXT NOT NULL,due_date TEXT NOT NULL DEFAULT ''); CREATE TABLE IF NOT EXISTS assets(id TEXT PRIMARY KEY,name TEXT NOT NULL,kind TEXT NOT NULL,value REAL NOT NULL,currency TEXT NOT NULL,updated_at TEXT NOT NULL);"""); return db
+ @contextmanager
+ def _db(self):
+ db=self._connect()
+ try:
+ with db: yield db
+ finally: db.close()
+ def status(self)->dict[str,Any]:
+ with self._db() as db:
+ counts={table:db.execute(f"SELECT count(*) FROM {table}").fetchone()[0] for table in ("accounts","transactions","budgets","goals","assets")}
+ return {"available":True,"source_ready":True,"container_runtime":"","runtime_ready":True,"running":True,"url":"","backend_url":"","path":str(self.database),"database":str(self.database),"repository":"internal://jonathan/personal-finance","reference_repository":SECURO_REPOSITORY,"revision":"jonathan-native-1","license":"MIT (implementation); AGPL design reference only","isolation":"Local Jonathan SQLite vault; no containers or external service.","features":["accounts","CSV transactions/import","categories","recurring transactions","budgets","goals","assets","reports","multi-currency"],"payments_required":False,"counts":counts}
+ def sync(self,progress:Callable[[str],None]|None=None)->dict[str,Any]:
+ if progress: progress("Jonathan's native personal-finance vault is ready")
+ self._connect().close(); return self.status()
+ install=sync
+ def start(self)->dict[str,Any]: return self.status()
+ def stop(self)->dict[str,Any]: return {**self.status(),"running":True,"message":"Embedded vault remains available; no external service is running."}
+ def logs(self,limit:int=200)->dict[str,Any]: return {**self.status(),"logs":"Embedded local vault is healthy. Actions are recorded in Jonathan's audit/event logs."}
+ def action(self,action:str,**data:Any)->dict[str,Any]:
+ now=datetime.now(timezone.utc).isoformat(); action=action.lower()
+ with self._db() as db:
+ if action=="add_account":
+ ident=uuid.uuid4().hex[:12]; db.execute("INSERT INTO accounts VALUES(?,?,?,?,?,?)",(ident,data.get("name") or "Account",data.get("type") or "checking",data.get("currency") or "USD",float(data.get("amount") or 0),now)); db.commit(); return {"account":dict(db.execute("SELECT * FROM accounts WHERE id=?",(ident,)).fetchone())}
+ if action=="add_transaction":
+ ident=uuid.uuid4().hex[:12]; db.execute("INSERT INTO transactions VALUES(?,?,?,?,?,?,?,?,?)",(ident,data.get("account_id") or "",data.get("occurred_on") or date.today().isoformat(),float(data.get("amount") or 0),data.get("currency") or "USD",data.get("category") or "",data.get("description") or "",1 if data.get("recurring") else 0,now)); db.commit(); return {"transaction":dict(db.execute("SELECT * FROM transactions WHERE id=?",(ident,)).fetchone())}
+ if action=="list":
+ entity=str(data.get("entity") or "transactions"); allowed={"accounts","transactions","budgets","goals","assets"}
+ if entity not in allowed: raise ValueError("Unsupported personal-finance entity")
+ return {"entity":entity,"items":[dict(r) for r in db.execute(f"SELECT * FROM {entity} LIMIT ?",(max(1,min(int(data.get('limit') or 100),1000)),))]}
+ if action=="dashboard":
+ rows=db.execute("SELECT currency,COALESCE(SUM(amount),0) total FROM transactions GROUP BY currency").fetchall(); return {"totals_by_currency":{r[0]:r[1] for r in rows},**self.status()}
+ raise ValueError(f"Unsupported personal finance action: {action}")
diff --git a/src/providers/anthropic_provider.py b/src/providers/anthropic_provider.py
index b3b7b30..bf67642 100644
--- a/src/providers/anthropic_provider.py
+++ b/src/providers/anthropic_provider.py
@@ -70,7 +70,10 @@ def _request_kwargs(
"messages": messages,
}
if system:
- payload["system"] = system
+ # Keep Jonathan's stable prefix eligible for Anthropic prompt
+ # caching without inserting a proxy or rewriting any content.
+ payload["system"] = ([{"type": "text", "text": system, "cache_control": {"type": "ephemeral"}}]
+ if isinstance(system, str) else system)
if tools:
payload["tools"] = tools
safe_extra = {
@@ -132,11 +135,6 @@ def _call_create(
except Exception as exc:
if prepared and is_input_schema_type_error(exc):
return client.messages.create(**self._retry_without_tools(request, prepared, exc))
- if prepared:
- try:
- return client.messages.create(**self._retry_without_tools(request, prepared, exc))
- except Exception:
- raise exc
raise
def _stream_request(
@@ -269,8 +267,7 @@ def chat_stream(
if not prepared:
raise
if not is_input_schema_type_error(exc):
- # Prefer a no-tools answer over failing the first streamed message.
- pass
+ raise
active = self._retry_without_tools(request, prepared, exc)
with client.messages.stream(**active) as stream:
for text in stream.text_stream:
@@ -335,6 +332,8 @@ def _read_stream(req: dict[str, Any]) -> tuple[str, Any]:
)
if not prepared:
raise
+ if not is_input_schema_type_error(exc):
+ raise
retry = self._retry_without_tools(request, prepared, exc)
streamed_text, final_message = _read_stream(retry)
@@ -350,7 +349,7 @@ def _read_stream(req: dict[str, Any]) -> tuple[str, Any]:
)
def _prepare_messages(self, messages: list[MessageInput]) -> list[dict[str, Any]]:
- """Convert history and stringify object-shaped tool_result.content."""
+ """Convert history and normalize Anthropic tool-result content."""
return sanitize_anthropic_messages(super()._prepare_messages(messages))
def get_available_models(self) -> list[str]:
diff --git a/src/providers/local_endpoints.py b/src/providers/local_endpoints.py
index b7e8809..f8e9510 100644
--- a/src/providers/local_endpoints.py
+++ b/src/providers/local_endpoints.py
@@ -8,8 +8,13 @@
import ipaddress
import json
+import os
+import shutil
import socket
+import subprocess
+import time
from concurrent.futures import ThreadPoolExecutor, as_completed
+from pathlib import Path
from typing import Any, Callable
from urllib.error import HTTPError, URLError
from urllib.parse import urlparse
@@ -49,6 +54,24 @@
"urls": ("http://127.0.0.1:3000/v1", "http://127.0.0.1:8080"),
"default_url": "http://127.0.0.1:3000/v1",
},
+ {
+ "id": "localai",
+ "label": "LocalAI",
+ "urls": ("http://127.0.0.1:8080/v1",),
+ "default_url": "http://127.0.0.1:8080/v1",
+ },
+ {
+ "id": "koboldcpp",
+ "label": "KoboldCpp",
+ "urls": ("http://127.0.0.1:5001/v1",),
+ "default_url": "http://127.0.0.1:5001/v1",
+ },
+ {
+ "id": "jan",
+ "label": "Jan",
+ "urls": ("http://127.0.0.1:1337/v1",),
+ "default_url": "http://127.0.0.1:1337/v1",
+ },
)
@@ -280,3 +303,296 @@ def scan_local_endpoints(
"error": str(exc),
})
return found
+
+
+def _windows_hidden() -> dict[str, Any]:
+ if os.name != "nt":
+ return {}
+ startup = subprocess.STARTUPINFO()
+ startup.dwFlags |= subprocess.STARTF_USESHOWWINDOW
+ startup.wShowWindow = 0
+ return {"creationflags": getattr(subprocess, "CREATE_NO_WINDOW", 0), "startupinfo": startup}
+
+
+def _known_executable(name: str, candidates: list[Path]) -> Path | None:
+ found = shutil.which(name)
+ if found:
+ return Path(found)
+ for candidate in candidates:
+ if candidate.is_file():
+ return candidate
+ return None
+
+
+def _command_json(command: list[str], *, timeout: int = 20) -> Any:
+ result = subprocess.run(
+ command, capture_output=True, text=True, encoding="utf-8", errors="replace",
+ timeout=timeout, **_windows_hidden(),
+ )
+ if result.returncode != 0:
+ raise RuntimeError((result.stderr or result.stdout or "command failed").strip())
+ raw = result.stdout.strip()
+ return json.loads(raw) if raw else None
+
+
+def _model_files(root: Path, *, limit: int = 100) -> list[str]:
+ if not root.is_dir():
+ return []
+ results: list[str] = []
+ extensions = {".gguf", ".bin", ".safetensors", ".onnx", ".pth", ".pt"}
+ try:
+ for path in root.rglob("*"):
+ if path.is_file() and path.suffix.lower() in extensions:
+ results.append(str(path))
+ if len(results) >= limit:
+ break
+ except OSError:
+ pass
+ return results
+
+
+def _memory_fit_label(model_bytes: int, available_bytes: int) -> str:
+ """Return conservative, informational fit guidance for a model artifact."""
+ if model_bytes <= 0 or available_bytes <= 0:
+ return "unknown"
+ if model_bytes > available_bytes:
+ return "insufficient"
+ if model_bytes > available_bytes * 0.65:
+ return "tight"
+ return "comfortable"
+
+
+def _runtime_resource_snapshot(
+ runtime_id: str,
+ model_files: list[str],
+ cli_models: list[dict[str, Any]],
+) -> dict[str, Any]:
+ """Measure local model storage and resident runtime memory without loading a model."""
+ file_sizes: list[int] = []
+ for value in model_files:
+ try:
+ file_sizes.append(Path(value).stat().st_size)
+ except OSError:
+ continue
+ reported_sizes: list[int] = []
+ for model in cli_models:
+ for key in ("size", "sizeBytes", "size_bytes", "file_size"):
+ try:
+ size = int(model.get(key) or 0)
+ except (TypeError, ValueError):
+ size = 0
+ if size > 0:
+ reported_sizes.append(size)
+ break
+
+ sizes = file_sizes + reported_sizes
+
+ result: dict[str, Any] = {
+ # A runtime may report the same artifacts that were found on disk.
+ "model_storage_bytes": max(sum(file_sizes), sum(reported_sizes)),
+ "largest_model_bytes": max(sizes, default=0),
+ "resident_memory_bytes": 0,
+ "running_processes": [],
+ "system_available_memory_bytes": 0,
+ "system_total_memory_bytes": 0,
+ "fit": "unknown",
+ "fit_is_estimate": True,
+ }
+ try:
+ import psutil
+ except ImportError:
+ return result
+
+ memory = psutil.virtual_memory()
+ result["system_available_memory_bytes"] = int(memory.available)
+ result["system_total_memory_bytes"] = int(memory.total)
+ tokens = {
+ "ollama": ("ollama",),
+ "lmstudio": ("lm studio", "lmstudio", "lms"),
+ "jan": ("jan",),
+ "gpt4all": ("gpt4all",),
+ "llamacpp": ("llama-server", "llama server"),
+ "koboldcpp": ("koboldcpp",),
+ }.get(runtime_id, ())
+ for process in psutil.process_iter(["pid", "name", "cmdline", "memory_info"]):
+ try:
+ info = process.info
+ name = str(info.get("name") or "").lower()
+ command = " ".join(str(value) for value in (info.get("cmdline") or [])).lower()
+ exact_name = Path(name).stem
+ matched = any(
+ token == exact_name if len(token) <= 3
+ else (token in command or token in name or token == exact_name)
+ for token in tokens
+ )
+ if not matched:
+ continue
+ rss = int(getattr(info.get("memory_info"), "rss", 0) or 0)
+ result["resident_memory_bytes"] += rss
+ result["running_processes"].append({"pid": int(info["pid"]), "name": info.get("name"), "rss_bytes": rss})
+ except (psutil.Error, OSError, TypeError, ValueError):
+ continue
+ result["running_processes"] = result["running_processes"][:50]
+ result["fit"] = _memory_fit_label(result["largest_model_bytes"], int(memory.available))
+ return result
+
+
+def discover_installed_runtimes() -> list[dict[str, Any]]:
+ """Inventory known local model applications and on-disk model stores.
+
+ This does not execute model files or contact the internet. CLI JSON is used
+ when a known local application provides it; otherwise only paths are read.
+ """
+ home = Path.home()
+ local = Path(os.environ.get("LOCALAPPDATA", ""))
+ programs = Path(os.environ.get("ProgramFiles", r"C:\Program Files"))
+ specs = [
+ {
+ "id": "ollama", "label": "Ollama", "command": "ollama",
+ "executables": [local / "Programs" / "Ollama" / "ollama.exe", programs / "Ollama" / "ollama.exe"],
+ "model_dirs": [home / ".ollama" / "models"],
+ },
+ {
+ "id": "lmstudio", "label": "LM Studio", "command": "lms",
+ "executables": [home / ".lmstudio" / "bin" / "lms.exe", local / "LM-Studio" / "bin" / "lms.exe"],
+ "model_dirs": [home / ".lmstudio" / "models", home / ".cache" / "lm-studio" / "models"],
+ },
+ {
+ "id": "jan", "label": "Jan", "command": "jan",
+ "executables": [local / "Programs" / "jan" / "Jan.exe", local / "Programs" / "Jan" / "Jan.exe"],
+ "model_dirs": [home / ".jan" / "models", home / "jan" / "models"],
+ },
+ {
+ "id": "gpt4all", "label": "GPT4All", "command": "gpt4all",
+ "executables": [local / "nomic.ai" / "GPT4All" / "GPT4All.exe", programs / "GPT4All" / "GPT4All.exe"],
+ "model_dirs": [home / ".cache" / "gpt4all", local / "nomic.ai" / "GPT4All"],
+ },
+ {
+ "id": "llamacpp", "label": "llama.cpp", "command": "llama-server",
+ "executables": [home / "llama.cpp" / "llama-server.exe", local / "llama.cpp" / "llama-server.exe"],
+ "model_dirs": [],
+ },
+ {
+ "id": "koboldcpp", "label": "KoboldCpp", "command": "koboldcpp",
+ "executables": [home / "KoboldCpp" / "koboldcpp.exe", local / "KoboldCpp" / "koboldcpp.exe"],
+ "model_dirs": [],
+ },
+ ]
+ inventory: list[dict[str, Any]] = []
+ for spec in specs:
+ executable = _known_executable(str(spec["command"]), list(spec["executables"]))
+ model_files: list[str] = []
+ cli_models: list[dict[str, Any]] = []
+ if spec["id"] == "lmstudio" and executable:
+ try:
+ payload = _command_json([str(executable), "ls", "--json"])
+ if isinstance(payload, list):
+ cli_models = [item for item in payload if isinstance(item, dict)]
+ except Exception:
+ pass
+ elif spec["id"] == "ollama" and executable:
+ try:
+ payload = _command_json([str(executable), "list", "--json"])
+ if isinstance(payload, list):
+ cli_models = [item for item in payload if isinstance(item, dict)]
+ except Exception:
+ pass
+ for directory in spec["model_dirs"]:
+ model_files.extend(_model_files(Path(directory), limit=max(0, 100 - len(model_files))))
+ if len(model_files) >= 100:
+ break
+ installed = bool(executable or cli_models or model_files or any(Path(path).exists() for path in spec["model_dirs"]))
+ if not installed:
+ continue
+ inventory.append({
+ "id": spec["id"], "label": spec["label"], "installed": True,
+ "executable": str(executable) if executable else "",
+ "models": cli_models,
+ "model_files": model_files,
+ "resources": _runtime_resource_snapshot(str(spec["id"]), model_files, cli_models),
+ })
+
+ # Hugging Face caches can be consumed by several runtimes, so report them
+ # independently without pretending a cache is itself a reachable server.
+ hf_root = home / ".cache" / "huggingface" / "hub"
+ hf_models = []
+ if hf_root.is_dir():
+ try:
+ hf_models = [path.name for path in hf_root.iterdir() if path.is_dir() and path.name.startswith("models--")][:100]
+ except OSError:
+ pass
+ if hf_models:
+ inventory.append({
+ "id": "huggingface-cache", "label": "Hugging Face model cache", "installed": True,
+ "executable": "", "models": [{"modelKey": name.replace("models--", "").replace("--", "/")} for name in hf_models],
+ "model_files": [],
+ "resources": _runtime_resource_snapshot("huggingface-cache", [], []),
+ })
+ return inventory
+
+
+def _runtime_model_key(runtime: dict[str, Any]) -> str:
+ for model in runtime.get("models") or []:
+ if not isinstance(model, dict) or model.get("type") == "embedding":
+ continue
+ value = model.get("modelKey") or model.get("key") or model.get("name") or model.get("model")
+ if value:
+ return str(value)
+ return ""
+
+
+def _start_known_runtime(runtime: dict[str, Any]) -> None:
+ executable = str(runtime.get("executable") or "")
+ if not executable:
+ return
+ runtime_id = runtime.get("id")
+ if runtime_id == "lmstudio":
+ subprocess.run([executable, "server", "start"], capture_output=True, timeout=45, **_windows_hidden())
+ model = _runtime_model_key(runtime)
+ if model:
+ subprocess.run(
+ [executable, "load", model, "--yes"], capture_output=True, timeout=300,
+ **_windows_hidden(),
+ )
+ elif runtime_id == "ollama":
+ subprocess.Popen(
+ [executable, "serve"], stdin=subprocess.DEVNULL, stdout=subprocess.DEVNULL,
+ stderr=subprocess.DEVNULL, **_windows_hidden(),
+ )
+
+
+def discover_local_environment(
+ extra_urls: list[str] | None = None,
+ *,
+ auto_start: bool = False,
+ timeout: float = 0.6,
+) -> dict[str, Any]:
+ """Find installed runtimes and connectable local endpoints.
+
+ When ``auto_start`` is true, only recognized Ollama/LM Studio executables
+ are launched, and both remain bound to their own localhost defaults.
+ """
+ runtimes = discover_installed_runtimes()
+ endpoints = scan_local_endpoints(extra_urls, timeout=timeout)
+ if auto_start and not any(item.get("reachable") and item.get("models") for item in endpoints):
+ for runtime in runtimes:
+ if runtime.get("id") not in {"ollama", "lmstudio"}:
+ continue
+ try:
+ _start_known_runtime(runtime)
+ except Exception:
+ continue
+ for _ in range(20):
+ time.sleep(0.25)
+ endpoints = scan_local_endpoints(extra_urls, timeout=max(timeout, 0.8))
+ if any(item.get("reachable") and item.get("models") for item in endpoints):
+ break
+ if any(item.get("reachable") and item.get("models") for item in endpoints):
+ break
+ selected = next((item for item in endpoints if item.get("reachable") and item.get("models")), None)
+ return {
+ "runtimes": runtimes,
+ "endpoints": endpoints,
+ "selected": selected,
+ "auto_started": bool(auto_start and selected),
+ }
diff --git a/src/providers/openai_compatible.py b/src/providers/openai_compatible.py
index c9d21fc..bd53841 100644
--- a/src/providers/openai_compatible.py
+++ b/src/providers/openai_compatible.py
@@ -14,6 +14,27 @@
from src.tool_system.schema_sanitize import sanitize_input_schema, sanitize_tools_for_api
+def _openai_multimodal_messages(messages: list[dict[str, Any]]) -> list[dict[str, Any]]:
+ """Translate persisted Anthropic-style image blocks to OpenAI image_url blocks."""
+ converted: list[dict[str, Any]] = []
+ for message in messages:
+ content = message.get("content") if isinstance(message, dict) else None
+ if not isinstance(content, list):
+ converted.append(message)
+ continue
+ blocks: list[Any] = []
+ for block in content:
+ if isinstance(block, dict) and block.get("type") == "image":
+ source = block.get("source") or {}
+ if source.get("type") == "base64" and source.get("data"):
+ media = source.get("media_type") or "image/png"
+ blocks.append({"type": "image_url", "image_url": {"url": f"data:{media};base64,{source['data']}"}})
+ continue
+ blocks.append(block)
+ converted.append({**message, "content": blocks})
+ return converted
+
+
def _convert_to_openai_tool_schema(anthropic_tool: dict[str, Any]) -> dict[str, Any] | None:
"""Convert Anthropic tool schema to OpenAI/GLM/Minimax function format.
@@ -87,6 +108,9 @@ def _build_usage_dict(self, usage: Any) -> dict[str, Any]:
"total_tokens": getattr(usage, "total_tokens", 0),
}
+ def _prepare_messages(self, messages: list[MessageInput]) -> list[dict[str, Any]]:
+ return _openai_multimodal_messages(super()._prepare_messages(messages))
+
def chat(
self,
messages: list[MessageInput],
diff --git a/src/providers/resilient.py b/src/providers/resilient.py
new file mode 100644
index 0000000..a945203
--- /dev/null
+++ b/src/providers/resilient.py
@@ -0,0 +1,133 @@
+"""Small in-process provider resilience layer.
+
+OmniRoute was reviewed as a design reference. Jonathan keeps its own provider
+classes and does not run a proxy or silently move prompts between providers.
+"""
+from __future__ import annotations
+
+import threading
+import time
+from typing import Any, Generator, Optional
+
+from .base import BaseProvider, ChatResponse, MessageInput, TextChunkCallback
+
+
+def _status_code(exc: BaseException) -> int | None:
+ for source in (exc, getattr(exc, "response", None)):
+ value = getattr(source, "status_code", None) or getattr(source, "status", None)
+ try:
+ return int(value) if value is not None else None
+ except (TypeError, ValueError):
+ pass
+ return None
+
+
+def is_transient_provider_error(exc: BaseException) -> bool:
+ """Retry only transport, throttling and server failures—not auth or 400s."""
+ code = _status_code(exc)
+ if code is not None:
+ return code in {408, 409, 425, 429} or 500 <= code <= 599
+ name = type(exc).__name__.lower()
+ text = str(exc).lower()
+ markers = ("timeout", "timed out", "connection reset", "connection aborted", "temporarily unavailable")
+ return any(marker in name or marker in text for marker in markers)
+
+
+class ResilientProvider(BaseProvider):
+ """Retry transient failures and open a short circuit after repeated faults."""
+
+ def __init__(self, provider: BaseProvider, *, retries: int = 1, failure_threshold: int = 3, cooldown: float = 30.0) -> None:
+ super().__init__(provider.api_key, provider.base_url, provider.model)
+ self.provider = provider
+ self.protocol_family = "anthropic" if provider.__class__.__name__ in {"AnthropicProvider", "MinimaxProvider"} else "openai"
+ self.retries = max(0, int(retries))
+ self.failure_threshold = max(1, int(failure_threshold))
+ self.cooldown = max(0.0, float(cooldown))
+ self._failures = 0
+ self._opened_at = 0.0
+ self._lock = threading.Lock()
+
+ def health(self) -> dict[str, Any]:
+ with self._lock:
+ remaining = max(0.0, self.cooldown - (time.monotonic() - self._opened_at)) if self._opened_at else 0.0
+ return {"state": "open" if remaining > 0 else "closed", "consecutive_failures": self._failures, "retry_after_seconds": round(remaining, 2)}
+
+ def _before_call(self) -> None:
+ with self._lock:
+ if self._opened_at and time.monotonic() - self._opened_at < self.cooldown:
+ remaining = self.cooldown - (time.monotonic() - self._opened_at)
+ raise RuntimeError(f"Provider circuit is cooling down; retry in {remaining:.1f}s")
+ if self._opened_at:
+ self._opened_at = 0.0
+ self._failures = 0
+
+ def _success(self) -> None:
+ with self._lock:
+ self._failures = 0
+ self._opened_at = 0.0
+
+ def _failure(self) -> None:
+ with self._lock:
+ self._failures += 1
+ if self._failures >= self.failure_threshold:
+ self._opened_at = time.monotonic()
+
+ def _call(self, method: str, *args: Any, **kwargs: Any) -> Any:
+ self._before_call()
+ for attempt in range(self.retries + 1):
+ try:
+ result = getattr(self.provider, method)(*args, **kwargs)
+ self._success()
+ return result
+ except Exception as exc:
+ self._failure()
+ if attempt >= self.retries or not is_transient_provider_error(exc):
+ raise
+ time.sleep(min(0.2 * (2**attempt), 1.0))
+ raise RuntimeError("unreachable")
+
+ def chat(self, messages: list[MessageInput], tools: Optional[list[dict[str, Any]]] = None, **kwargs: Any) -> ChatResponse:
+ return self._call("chat", messages, tools=tools, **kwargs)
+
+ def chat_stream(self, messages: list[MessageInput], tools: Optional[list[dict[str, Any]]] = None, **kwargs: Any) -> Generator[str, None, None]:
+ self._before_call()
+ emitted = False
+ for attempt in range(self.retries + 1):
+ try:
+ for chunk in self.provider.chat_stream(messages, tools=tools, **kwargs):
+ emitted = True
+ yield chunk
+ self._success()
+ return
+ except Exception as exc:
+ self._failure()
+ if emitted or attempt >= self.retries or not is_transient_provider_error(exc):
+ raise
+ time.sleep(min(0.2 * (2**attempt), 1.0))
+
+ def chat_stream_response(self, messages: list[MessageInput], tools: Optional[list[dict[str, Any]]] = None, on_text_chunk: TextChunkCallback | None = None, **kwargs: Any) -> ChatResponse:
+ emitted = False
+ def observed(chunk: str) -> None:
+ nonlocal emitted
+ emitted = emitted or bool(chunk)
+ if on_text_chunk is not None:
+ on_text_chunk(chunk)
+ self._before_call()
+ for attempt in range(self.retries + 1):
+ try:
+ response = self.provider.chat_stream_response(messages, tools=tools, on_text_chunk=observed, **kwargs)
+ self._success()
+ return response
+ except Exception as exc:
+ self._failure()
+ if emitted or attempt >= self.retries or not is_transient_provider_error(exc):
+ raise
+ time.sleep(min(0.2 * (2**attempt), 1.0))
+ raise RuntimeError("unreachable")
+
+ def get_available_models(self) -> list[str]:
+ return self.provider.get_available_models()
+
+
+def resilient(provider: BaseProvider) -> ResilientProvider:
+ return provider if isinstance(provider, ResilientProvider) else ResilientProvider(provider)
diff --git a/src/repl/core.py b/src/repl/core.py
index 16893be..79f1896 100644
--- a/src/repl/core.py
+++ b/src/repl/core.py
@@ -13,6 +13,7 @@
except Exception: # pragma: no cover
FuzzyCompleter = None # type: ignore
from prompt_toolkit.key_binding import KeyBindings
+ from prompt_toolkit.output import DummyOutput
except ModuleNotFoundError: # pragma: no cover
class FileHistory: # type: ignore
def __init__(self, *args, **kwargs):
@@ -43,6 +44,9 @@ def __init__(self, *args, **kwargs):
def prompt(self, *args, **kwargs):
raise EOFError()
+ class DummyOutput: # type: ignore
+ pass
+
try:
from rich.console import Console, Group
from rich.align import Align
@@ -175,7 +179,7 @@ def _show_slash_completions(event): # type: ignore[no-untyped-def]
buf.insert_text("/")
buf.start_completion(select_first=False)
- self.prompt_session = PromptSession(
+ prompt_kwargs = dict(
history=FileHistory(str(history_file)),
auto_suggest=AutoSuggestFromHistory(),
completer=self.completer,
@@ -185,6 +189,9 @@ def _show_slash_completions(event): # type: ignore[no-untyped-def]
key_bindings=self.bindings,
complete_while_typing=True,
)
+ if sys.platform.startswith("win") and not getattr(sys.stdout, "isatty", lambda: False)():
+ prompt_kwargs["output"] = DummyOutput()
+ self.prompt_session = PromptSession(**prompt_kwargs)
def _ask_user_questions(self, questions: list[dict]) -> dict[str, str]:
# Stop the Rich status spinner if running, so we can get clean input
diff --git a/src/skills/bundled/1password/SKILL.md b/src/skills/bundled/1password/SKILL.md
new file mode 100644
index 0000000..bcb222e
--- /dev/null
+++ b/src/skills/bundled/1password/SKILL.md
@@ -0,0 +1,163 @@
+---
+name: 1password
+description: Set up op CLI, sign in, and read or inject secrets.
+version: 1.0.0
+author: arceus77-7, enhanced by Hermes Agent
+license: MIT
+platforms: [linux, macos, windows]
+metadata:
+ hermes:
+ tags: [security, secrets, 1password, op, cli]
+ category: security
+setup:
+ help: "Create a service account at https://my.1password.com → Settings → Service Accounts"
+ collect_secrets:
+ - env_var: OP_SERVICE_ACCOUNT_TOKEN
+ prompt: "1Password Service Account Token"
+ provider_url: "https://developer.1password.com/docs/service-accounts/"
+ secret: true
+---
+
+# 1Password CLI
+
+Use this skill when the user wants secrets managed through 1Password instead of plaintext env vars or files.
+
+## Requirements
+
+- 1Password account
+- 1Password CLI (`op`) installed
+- One of: desktop app integration, service account token (`OP_SERVICE_ACCOUNT_TOKEN`), or Connect server
+- `tmux` available for stable authenticated sessions during Hermes terminal calls (desktop app flow only)
+
+## When to Use
+
+- Install or configure 1Password CLI
+- Sign in with `op signin`
+- Read secret references like `op://Vault/Item/field`
+- Inject secrets into config/templates using `op inject`
+- Run commands with secret env vars via `op run`
+
+## Authentication Methods
+
+### Service Account (recommended for Hermes)
+
+Set `OP_SERVICE_ACCOUNT_TOKEN` in `${HERMES_HOME:-~/.hermes}/.env` (the skill will prompt for this on first load).
+No desktop app needed. Supports `op read`, `op inject`, `op run`.
+
+```bash
+export OP_SERVICE_ACCOUNT_TOKEN="your-token-here"
+op whoami # verify — should show Type: SERVICE_ACCOUNT
+```
+
+### Desktop App Integration (interactive)
+
+1. Enable in 1Password desktop app: Settings → Developer → Integrate with 1Password CLI
+2. Ensure app is unlocked
+3. Run `op signin` and approve the biometric prompt
+
+### Connect Server (self-hosted)
+
+```bash
+export OP_CONNECT_HOST="http://localhost:8080"
+export OP_CONNECT_TOKEN="your-connect-token"
+```
+
+## Setup
+
+1. Install CLI:
+
+```bash
+# macOS
+brew install 1password-cli
+
+# Linux (official package/install docs)
+# See references/get-started.md for distro-specific links.
+
+# Windows (winget)
+winget install AgileBits.1Password.CLI
+```
+
+2. Verify:
+
+```bash
+op --version
+```
+
+3. Choose an auth method above and configure it.
+
+## Hermes Execution Pattern (desktop app flow)
+
+Hermes terminal commands are non-interactive by default and can lose auth context between calls.
+For reliable `op` use with desktop app integration, run sign-in and secret operations inside a dedicated tmux session.
+
+Note: This is NOT needed when using `OP_SERVICE_ACCOUNT_TOKEN` — the token persists across terminal calls automatically.
+
+```bash
+SOCKET_DIR="${TMPDIR:-/tmp}/hermes-tmux-sockets"
+mkdir -p "$SOCKET_DIR"
+SOCKET="$SOCKET_DIR/hermes-op.sock"
+SESSION="op-auth-$(date +%Y%m%d-%H%M%S)"
+
+tmux -S "$SOCKET" new -d -s "$SESSION" -n shell
+
+# Sign in (approve in desktop app when prompted)
+tmux -S "$SOCKET" send-keys -t "$SESSION":0.0 -- "eval \"\$(op signin --account my.1password.com)\"" Enter
+
+# Verify auth
+tmux -S "$SOCKET" send-keys -t "$SESSION":0.0 -- "op whoami" Enter
+
+# Example read
+tmux -S "$SOCKET" send-keys -t "$SESSION":0.0 -- "op read 'op://Private/Npmjs/one-time password?attribute=otp'" Enter
+
+# Capture output when needed
+tmux -S "$SOCKET" capture-pane -p -J -t "$SESSION":0.0 -S -200
+
+# Cleanup
+tmux -S "$SOCKET" kill-session -t "$SESSION"
+```
+
+## Common Operations
+
+### Read a secret
+
+```bash
+op read "op://app-prod/db/password"
+```
+
+### Get OTP
+
+```bash
+op read "op://app-prod/npm/one-time password?attribute=otp"
+```
+
+### Inject into template
+
+```bash
+echo "db_password: {{ op://app-prod/db/password }}" | op inject
+```
+
+### Run a command with secret env var
+
+```bash
+export DB_PASSWORD="op://app-prod/db/password" # example op:// reference, resolved by `op run`
+op run -- sh -c '[ -n "$DB_PASSWORD" ] && echo "DB_PASSWORD is set" || echo "DB_PASSWORD missing"'
+```
+
+## Guardrails
+
+- Never print raw secrets back to user unless they explicitly request the value.
+- Prefer `op run` / `op inject` instead of writing secrets into files.
+- If command fails with "account is not signed in", run `op signin` again in the same tmux session.
+- If desktop app integration is unavailable (headless/CI), use service account token flow.
+
+## CI / Headless note
+
+For non-interactive use, authenticate with `OP_SERVICE_ACCOUNT_TOKEN` and avoid interactive `op signin`.
+Service accounts require CLI v2.18.0+.
+
+## References
+
+- `references/get-started.md`
+- `references/cli-examples.md`
+- https://developer.1password.com/docs/cli/
+- https://developer.1password.com/docs/service-accounts/
diff --git a/src/skills/bundled/1password/references/cli-examples.md b/src/skills/bundled/1password/references/cli-examples.md
new file mode 100644
index 0000000..1e9fd72
--- /dev/null
+++ b/src/skills/bundled/1password/references/cli-examples.md
@@ -0,0 +1,31 @@
+# op CLI examples
+
+## Sign-in and identity
+
+```bash
+op signin
+op signin --account my.1password.com
+op whoami
+op account list
+```
+
+## Read secrets
+
+```bash
+op read "op://app-prod/db/password"
+op read "op://app-prod/npm/one-time password?attribute=otp"
+```
+
+## Inject secrets
+
+```bash
+echo "api_key: {{ op://app-prod/openai/api key }}" | op inject
+op inject -i config.tpl.yml -o config.yml
+```
+
+## Run command with secrets
+
+```bash
+export DB_PASSWORD="op://app-prod/db/password" # example op:// reference, resolved by `op run`
+op run -- sh -c '[ -n "$DB_PASSWORD" ] && echo "DB_PASSWORD is set"'
+```
diff --git a/src/skills/bundled/1password/references/get-started.md b/src/skills/bundled/1password/references/get-started.md
new file mode 100644
index 0000000..5284d39
--- /dev/null
+++ b/src/skills/bundled/1password/references/get-started.md
@@ -0,0 +1,21 @@
+# 1Password CLI get-started (summary)
+
+Official docs: https://developer.1password.com/docs/cli/get-started/
+
+## Core flow
+
+1. Install `op` CLI.
+2. Enable desktop app integration in 1Password app.
+3. Unlock app.
+4. Run `op signin` and approve prompt.
+5. Verify with `op whoami`.
+
+## Multiple accounts
+
+- Use `op signin --account `
+- Or set `OP_ACCOUNT`
+
+## Non-interactive / automation
+
+- Use service accounts and `OP_SERVICE_ACCOUNT_TOKEN`
+- Prefer `op run` and `op inject` for runtime secret handling
diff --git a/src/skills/bundled/3-statement-model/SKILL.md b/src/skills/bundled/3-statement-model/SKILL.md
new file mode 100644
index 0000000..525049b
--- /dev/null
+++ b/src/skills/bundled/3-statement-model/SKILL.md
@@ -0,0 +1,433 @@
+---
+name: 3-statement-model
+description: Build integrated IS/BS/CF financial workbooks in Excel.
+version: 1.0.0
+author: Anthropic (adapted by Nous Research)
+license: Apache-2.0
+platforms: [linux, macos, windows]
+metadata:
+ hermes:
+ tags: [finance, three-statement, income-statement, balance-sheet, cash-flow, excel, openpyxl, modeling]
+ related_skills: [excel-author, pptx-author, dcf-model, lbo-model]
+---
+
+## Environment
+
+This skill assumes **headless openpyxl** — you are producing an .xlsx file on disk.
+Follow the `excel-author` skill's conventions for cell coloring, formulas, named ranges, and sensitivity tables.
+Recalculate before delivery: `python /path/to/excel-author/scripts/recalc.py ./out/model.xlsx`.
+
+# 3-Statement Financial Model Template Completion
+
+Complete and populate integrated financial model templates with proper linkages between Income Statement, Balance Sheet, and Cash Flow Statement.
+
+## ⚠️ CRITICAL PRINCIPLES — Read Before Populating Any Template
+
+**Formulas over hardcodes (non-negotiable):**
+- Every projection cell, roll-forward, linkage, and subtotal MUST be an Excel formula — never a pre-computed value
+- When using Python/openpyxl: write formula strings (`ws["D15"] = "=D14*(1+Assumptions!$B$5)"`), NOT computed results (`ws["D15"] = 12500`)
+- The ONLY cells that should contain hardcoded numbers are: (1) historical actuals, (2) assumption drivers in the Assumptions tab
+- If you find yourself computing a value in Python and writing the result to a cell — STOP. Write the formula instead.
+- Why: the model must flex when scenarios toggle or assumptions change. Hardcodes break every downstream integrity check silently.
+
+**Verify step-by-step with the user:**
+1. **After mapping the template** → show the user which tabs/sections you've identified and confirm before touching any cells
+2. **After populating historicals** → show the user the historical block and confirm values/periods match source data
+3. **After building IS projections** → run the subtotal checks, show the user the projected IS, confirm before moving to BS
+4. **After building BS** → show the user the balance check (Assets = L+E) for every period, confirm before moving to CF
+5. **After building CF** → show the user the cash tie-out (CF ending cash = BS cash), confirm before finalizing
+6. **Do NOT populate the entire model end-to-end and present it complete** — break at each statement, show the work, catch errors early
+
+## Formatting — Professional Blue/Grey Palette (Default unless template/user specifies otherwise)
+
+**Keep colors minimal.** Use only blues and greys for cell fills. Do NOT introduce greens, yellows, oranges, or multiple accent colors — a clean model uses restraint.
+
+| Element | Fill | Font |
+|---|---|---|
+| Section headers (IS / BS / CF titles) | Dark blue `#1F4E79` | White bold |
+| Column headers (FY2024A, FY2025E, etc.) | Light blue `#D9E1F2` | Black bold |
+| Input cells (historicals, assumption drivers) | Light grey `#F2F2F2` or white | Blue `#0000FF` |
+| Formula cells | White | Black |
+| Cross-tab links | White | Green `#008000` |
+| Check rows / key totals | Medium blue `#BDD7EE` | Black bold |
+
+**That's 3 blues + 1 grey + white.** If the template has its own color scheme, follow the template instead.
+
+Font color signals *what* a cell is (input/formula/link). Fill color signals *where* you are (header/data/check).
+
+## Model Structure
+
+### Identifying Template Tab Organization
+
+Templates vary in their tab naming conventions and organization. Before populating, review all tabs to understand the template's structure. Below are common tab names and their typical contents:
+
+| Common Tab Names | Contents to Look For |
+|------------------|----------------------|
+| IS, P&L, Income Statement | Income Statement |
+| BS, Balance Sheet | Balance Sheet |
+| CF, CFS, Cash Flow | Cash Flow Statement |
+| WC, Working Capital | Working Capital Schedule |
+| DA, D&A, Depreciation, PP&E | Depreciation & Amortization Schedule |
+| Debt, Debt Schedule | Debt Schedule |
+| NOL, Tax, DTA | Net Operating Loss Schedule |
+| Assumptions, Inputs, Drivers | Driver assumptions and inputs |
+| Checks, Audit, Validation | Error-checking dashboard |
+
+**Template Review Checklist**
+- Identify which tabs exist in the template (not all templates include every schedule)
+- Note any template-specific tabs not listed above
+- Understand tab dependencies (e.g., which schedules feed into the main statements)
+- Locate input cells vs. formula cells on each tab
+
+### Understanding Template Structure
+
+Before populating a template, familiarize yourself with its existing layout to ensure data is entered in the correct locations and formulas remain intact.
+
+**Identifying Row Structure**
+- Locate the model title at top of each tab
+- Identify section headers and their visual separation
+- Find the units row indicating $ millions, %, x, etc.
+- Note column headers distinguishing Actuals vs. Estimates periods
+- Confirm period labels (e.g., FY2024A, FY2025E)
+- Identify input cells vs. formula cells (typically distinguished by font color)
+
+**Identifying Column Structure**
+- Confirm line item labels in leftmost column
+- Verify historical years precede projection years
+- Note the visual border separating historical from projected periods
+- Check for consistent column order across all tabs
+
+**Working with Named Ranges**
+Templates often use named ranges for key inputs and outputs. Before entering data:
+- Review existing named ranges in the template (Formulas → Name Manager in Excel)
+- Common named ranges include: Revenue growth rates, cost percentages, key outputs (Net Income, EBITDA, Total Debt, Cash), scenario selector cell
+- Ensure inputs are entered in cells that feed into these named ranges
+
+### Projection Period
+- Templates typically project 5 years forward from last historical year
+- Verify historical (A) vs. projected (E) columns are clearly separated
+- Confirm columns use fiscal year notation (e.g., FY2024A, FY2025E)
+
+## Margin Analysis
+
+**Note: The following margin analysis should only be performed if prompted by the user or if the template explicitly requires it. If no prompt is given, skip this section.**
+
+Calculate and display profitability margins on the Income Statement (IS) tab to track operational efficiency and enable peer comparison.
+
+### Core Margins to Include
+
+| Margin | Formula | What It Measures |
+|--------|---------|------------------|
+| Gross Margin | Gross Profit / Revenue | Pricing power, production efficiency |
+| EBITDA Margin | EBITDA / Revenue | Core operating profitability |
+| EBIT Margin | EBIT / Revenue | Operating profitability after D&A |
+| Net Income Margin | Net Income / Revenue | Bottom-line profitability |
+
+### Income Statement Layout with Margins
+
+Display margin percentages directly below each profit line item:
+- Gross Margin % below Gross Profit
+- EBIT Margin % below EBIT
+- EBITDA Margin % below EBITDA
+- Net Income Margin % below Net Income
+
+## Credit Metrics
+
+**Note: The following Credit analysis should only be performed if prompted by the user or if the template explicitly requires it. If no prompt is given, skip this section.**
+
+Calculate and display credit/leverage metrics on the Balance Sheet (BS) tab to assess financial health, debt capacity, and covenant compliance.
+
+### Core Credit Metrics to Include
+
+| Metric | Formula | What It Measures |
+|--------|---------|------------------|
+| Total Debt / EBITDA | Total Debt / LTM EBITDA | Leverage multiple |
+| Net Debt / EBITDA | (Total Debt - Cash) / LTM EBITDA | Leverage net of cash |
+| Interest Coverage | EBITDA / Interest Expense | Ability to service debt |
+| Debt / Total Cap | Total Debt / (Total Debt + Equity) | Capital structure |
+| Debt / Equity | Total Debt / Total Equity | Financial leverage |
+| Current Ratio | Current Assets / Current Liabilities | Short-term liquidity |
+| Quick Ratio | (Current Assets - Inventory) / Current Liabilities | Immediate liquidity |
+
+### Credit Metric Hierarchy Checks
+
+Validate that Upside shows strongest credit profile:
+- Leverage: Upside < Base < Downside (lower is better)
+- Coverage: Upside > Base > Downside (higher is better)
+- Liquidity: Upside > Base > Downside (higher is better)
+
+### Covenant Compliance Tracking
+
+If debt covenants are known, add explicit compliance checks comparing actual metrics to covenant thresholds.
+
+## Scenario Analysis (Base / Upside / Downside)
+
+Use a scenario toggle (dropdown) in the Assumptions tab with CHOOSE or INDEX/MATCH formulas.
+
+| Scenario | Description |
+|----------|-------------|
+| Base Case | Management guidance or consensus estimates |
+| Upside Case | Above-guidance growth, margin expansion |
+| Downside Case | Below-trend growth, margin compression |
+
+**Key Drivers to Sensitize**: Revenue growth, Gross margin, SG&A %, DSO/DIO/DPO, CapEx %, Interest rate, Tax rate.
+
+**Scenario Audit Checks**: Toggle switches all statements, BS balances in all scenarios, Cash ties out, Hierarchy holds (Upside > Base > Downside for NI, EBITDA, FCF, margins).
+
+## SEC Filings Data Extraction
+
+If the template specifically requires pulling data from SEC filings (10-K, 10-Q), see [references/sec-filings.md](references/sec-filings.md) for detailed extraction guidance. This reference is only needed when populating templates with public company data from regulatory filings.
+
+## Completing Model Templates
+
+This section provides general guidance for completing any 3-statement financial model template while preserving existing formulas and ensuring data integrity.
+
+### Step 1: Analyze the Template Structure
+
+Before entering any data, thoroughly review the template to understand its architecture:
+
+**Identify Input vs. Formula Cells**
+- Look for visual cues (font color, cell shading) that distinguish input cells from formula cells
+- Common conventions: Blue font = inputs, Black font = formulas, Green font = links to other sheets
+- Use Excel's Trace Precedents/Dependents (Formulas → Trace Precedents) to understand cell relationships
+- Check for named ranges that may control key inputs (Formulas → Name Manager)
+
+**Map the Template's Flow**
+- Identify which tabs feed into others (e.g., Assumptions → IS → BS → CF)
+- Note any supporting schedules and their linkages to main statements
+- Document the template's specific line items and structure before populating
+
+### Step 2: Filling in Data Without Breaking Formulas
+
+**Golden Rules for Data Entry**
+
+| Rule | Description |
+|------|-------------|
+| Only edit input cells | Never overwrite cells containing formulas unless intentionally replacing the formula |
+| Preserve cell references | When copying data, use Paste Values (Ctrl+Shift+V) to avoid overwriting formulas with source formatting |
+| Match the template's units | Verify if template uses thousands, millions, or actual values before entering data |
+| Respect sign conventions | Follow the template's existing sign convention (e.g., expenses as positive or negative) |
+| Check for circular references | If the template uses iterative calculations, ensure Enable Iterative Calculation is turned on |
+
+**Safe Data Entry Process**
+1. Identify the exact cells designated for input (usually highlighted or labeled)
+2. Enter historical data first, then verify formulas are calculating correctly for those periods
+3. Enter assumption drivers that feed forecast calculations
+4. Review calculated outputs to confirm formulas are working as intended
+5. If a formula cell must be modified, document the original formula before making changes
+
+**Handling Pre-Built Formulas**
+- If formulas reference cells you haven't populated yet, expect temporary errors (#REF!, #DIV/0!) until all inputs are complete
+- When formulas produce unexpected results, trace precedents to identify missing or incorrect inputs
+- Never delete rows/columns without checking for formula dependencies across all tabs
+
+### Step 3: Validating Formulas
+
+**Formula Integrity Checks**
+
+Before relying on template outputs, validate that formulas are functioning correctly:
+
+| Check Type | Method |
+|------------|--------|
+| Trace precedents | Select a formula cell → Formulas → Trace Precedents to verify it references correct inputs |
+| Trace dependents | Verify key inputs flow to expected output cells |
+| Evaluate formula | Use Formulas → Evaluate Formula to step through complex calculations |
+| Check for hardcodes | Projection formulas should reference assumptions, not contain hardcoded values |
+| Test with known values | Input simple test values to verify formulas produce expected results |
+| Cross-tab consistency | Ensure the same formula logic applies across all projection periods |
+
+**Common Formula Issues to Watch For**
+- Mixed absolute/relative references causing incorrect results when copied across periods
+- Broken links to external files or deleted ranges (#REF! errors)
+- Division by zero in early periods before revenue ramps (#DIV/0! errors)
+- Circular reference warnings (may be intentional for interest calculations)
+- Inconsistent formulas across projection columns (use Ctrl+\ to find differences)
+
+**Validating Cross-Tab Linkages**
+- Confirm values that appear on multiple tabs are linked (not duplicated)
+- Verify schedule totals tie to corresponding line items on main statements
+- Check that period labels align across all tabs
+
+### Step 4: Quality Checks by Sheet
+
+Perform these validation checks on each sheet after populating the template:
+
+**Income Statement (IS) Quality Checks**
+- Revenue figures match source data for historical periods
+- All expense line items sum to reported totals
+- Subtotals (Gross Profit, EBIT, EBT, Net Income) calculate correctly
+- Tax calculation logic is appropriate (handles losses correctly)
+- Forecast drivers reference assumptions tab (no hardcodes)
+- Period-over-period changes are directionally reasonable
+
+**Balance Sheet (BS) Quality Checks**
+- Assets = Liabilities + Equity for every period (primary check)
+- Cash balance matches Cash Flow Statement ending cash
+- Working capital accounts tie to supporting schedules (if applicable)
+- Retained Earnings rolls forward correctly: Prior RE + Net Income - Dividends +/- Adjustments = Ending RE
+- Debt balances tie to debt schedule (if applicable)
+- All balance sheet items have appropriate signs (assets positive, most liabilities positive)
+
+**Cash Flow Statement (CF) Quality Checks**
+- Net Income at top of CFO matches Income Statement Net Income
+- Non-cash add-backs (D&A, SBC, etc.) tie to their source schedules/statements
+- Working capital changes have correct signs (increase in asset = use of cash = negative)
+- CapEx ties to PP&E schedule or fixed asset roll-forward
+- Financing activities tie to changes in debt and equity accounts on BS
+- Ending Cash matches Balance Sheet Cash
+- Beginning Cash equals prior period Ending Cash
+
+**Supporting Schedule Quality Checks**
+- Opening balances equal prior period closing balances
+- Roll-forward logic is complete (Beginning + Additions - Deductions = Ending)
+- Schedule totals tie to main statement line items
+- Assumptions used in calculations match Assumptions tab
+
+### Step 5: Cross-Statement Integrity Checks
+
+After validating individual sheets, confirm the three statements are properly integrated:
+
+| Check | Formula | Expected Result |
+|-------|---------|-----------------|
+| Balance Sheet Balance | Assets - Liabilities - Equity | = 0 |
+| Cash Tie-Out | CF Ending Cash - BS Cash | = 0 |
+| Net Income Link | IS Net Income - CF Starting Net Income | = 0 |
+| Retained Earnings | Prior RE + NI - Dividends - BS Ending RE | = 0 (adjust for SBC/other items as needed) |
+
+### Step 6: Final Review
+
+Before considering the model complete:
+- Toggle through all scenarios (if applicable) to verify checks pass in each case
+- Review all #REF!, #DIV/0!, #VALUE!, and #NAME? errors and resolve or document
+- Confirm all input cells have been populated (search for placeholder values)
+- Verify units are consistent across all tabs
+- Save a clean version before making any additional modifications
+
+## Model Validation and Audit
+
+This section consolidates all validation checks and audit procedures for completed templates.
+
+### Core Linkages (Must Always Hold)
+
+See [references/formulas.md](references/formulas.md) for all formula details.
+
+| Check | Formula | Expected Result |
+|-------|---------|-----------------|
+| Balance Sheet Balance | Assets - Liabilities - Equity | = 0 |
+| Cash Tie-Out | CF Ending Cash - BS Cash | = 0 |
+| Cash Monthly vs Annual | Closing Cash (Monthly) - Closing Cash (Annual) | = 0 |
+| Net Income Link | IS Net Income - CF Starting Net Income | = 0 |
+| Retained Earnings | Prior RE + NI + SBC - Dividends - BS Ending RE | = 0 |
+| Equity Financing | ΔCommon Stock/APIC (BS) - Equity Issuance (CFF) | = 0 |
+| Year 0 Equity | Equity Raised (Year 0) - Beginning Equity Capital (Year 1) | = 0 |
+
+### Sign Convention Reference
+
+| Statement | Item | Sign Convention |
+|-----------|------|-----------------|
+| CFO | D&A, SBC | Positive (add-back) |
+| CFO | ΔAR (increase) | Negative (use of cash) |
+| CFO | ΔAP (increase) | Positive (source of cash) |
+| CFI | CapEx | Negative |
+| CFF | Debt issuance | Positive |
+| CFF | Debt repayments | Negative |
+| CFF | Dividends | Negative |
+
+### Circular Reference Handling
+
+Interest expense creates circularity: Interest → Net Income → Cash → Debt Balance → Interest
+
+Enable iterative calculation in Excel: File → Options → Formulas → Enable iterative calculation. Set maximum iterations to 100, maximum change to 0.001. Add a circuit breaker toggle in Assumptions tab.
+
+### Check Categories
+
+**Section 1: Currency Consistency**
+- Currency identified and documented in Assumptions
+- All tabs use consistent currency symbol and scale
+- Units row matches model currency
+
+**Section 2: Balance Sheet Integrity**
+- Assets = Liabilities + Equity (for each period)
+- Formula: Assets - Liabilities - Equity (must = 0)
+
+**Section 3: Cash Flow Integrity**
+- Cash ties to BS (CF Ending Cash = BS Cash)
+- Cash Monthly vs Annual: Closing Cash (Monthly) = Closing Cash (Annual)
+- NI ties to IS (CF Net Income = IS Net Income)
+- D&A ties to schedule
+- SBC ties to IS
+- ΔAR, ΔInventory, ΔAP tie to WC schedule
+- CapEx ties to DA schedule
+
+**Section 4: Retained Earnings**
+- RE roll-forward check: Prior RE + NI + SBC - Dividends = Ending RE
+- Show component breakdown for debugging
+
+**Section 5: Working Capital**
+- AR, Inventory, AP tie to BS
+- DSO, DIO, DPO reasonability checks (flag if outside normal ranges)
+
+**Section 6: Debt Schedule**
+- Total Debt ties to BS (Current + LT Debt)
+- Interest calculation ties to IS
+
+**Section 6b: Equity Financing**
+- Equity issuance proceeds tie to BS Common Stock/APIC increase
+- Cash increase from equity = Equity account increase (must balance)
+- Equity Raise Tie-Out: ΔCommon Stock/APIC (BS) = Equity Issuance (CFF) (must = 0)
+- Year 0 Equity Tie-Out: Equity Raised (Year 0) = Beginning Equity Capital (Year 1)
+
+**Section 6c: NOL Schedule**
+- Beginning NOL (Year 1 / Formation) = 0 (new business starts with zero NOL)
+- NOL increases only when EBT < 0 (losses must be realized to generate NOL)
+- DTA ties to BS (NOL Schedule DTA = BS Deferred Tax Asset)
+- NOL utilization ≤ 80% of EBT (post-2017 federal limitation)
+- NOL balance is non-negative (cannot utilize more than available)
+- NOL generated only when EBT < 0
+- Tax expense = 0 when taxable income ≤ 0
+
+**Section 7: Scenario Hierarchy**
+- Absolute metrics: Upside > Base > Downside (NI, EBITDA, FCF)
+- Margins: Upside > Base > Downside (GM%, EBITDA%, NI%)
+- Credit metrics: Upside < Base < Downside for leverage (inverted)
+
+**Section 8: Formula Integrity**
+- COGS, S&M, G&A, R&D, SBC driven by % of Revenue (no hardcodes)
+- Consistent formulas across projection years
+- No #REF!, #DIV/0!, #VALUE! errors
+
+**Section 9: Credit Metric Thresholds**
+- Flag metrics as Green/Yellow/Red based on covenant thresholds
+- Summary of any red flags
+
+### Master Check Formula
+
+Aggregate all section statuses into a single master check:
+- If all sections pass → "✓ ALL CHECKS PASS"
+- If any section fails → "✗ ERRORS DETECTED - REVIEW BELOW"
+
+### Quick Debug Workflow
+
+When Master Status shows errors:
+1. Scroll to find red-highlighted sections
+2. Identify which check category has failures
+3. Navigate to source tab to investigate
+4. Fix the underlying issue
+5. Return to Checks tab to verify resolution
+
+
+## Data sources — MCP first, web fallback
+
+Many passages below say "use the S&P Kensho MCP / Daloopa MCP / FactSet MCP". Those are commercial financial-data MCPs from the original Cowork plugin context. In Hermes:
+
+- **If you have any structured financial-data MCP configured** (Hermes supports MCP — see `native-mcp` skill), prefer it for point-in-time comps, precedent transactions, and filings.
+- **Otherwise**, fall back to:
+ - `web_search` / `web_extract` against SEC EDGAR (`https://www.sec.gov/cgi-bin/browse-edgar`) for US filings
+ - Company IR pages for press releases, earnings decks
+ - `browser_navigate` for interactive data portals
+ - User-provided data (explicitly ask when the context doesn't have it)
+- **Never fabricate**. If a multiple, precedent, or filing number can't be sourced, flag the cell as `[UNSOURCED]` and surface it to the user.
+
+## Attribution
+
+This skill is adapted from Anthropic's Claude for Financial Services plugin suite (Apache-2.0). The Office-JS / Cowork live-Excel paths have been removed; this version targets headless openpyxl via the `excel-author` skill's conventions. Original: https://github.com/anthropics/financial-services
diff --git a/src/skills/bundled/3-statement-model/references/formatting.md b/src/skills/bundled/3-statement-model/references/formatting.md
new file mode 100644
index 0000000..1fbe938
--- /dev/null
+++ b/src/skills/bundled/3-statement-model/references/formatting.md
@@ -0,0 +1,118 @@
+# Formatting Standards Reference
+
+| Element | Format |
+|---------|--------|
+| Hard-coded inputs | Blue font |
+| Formulas | Black font |
+| Links to other sheets | Green font |
+| Check cells | Red if error, green if balanced |
+| Negative values | Parentheses, not minus signs |
+| Currency | No decimals for large figures, 2 decimals for per-share |
+| Percentages | 1 decimal place |
+| Headers | Bold, bottom border |
+| Units row | Include units row below headers ($ millions, %, etc.) |
+
+## Visual Separation Guidelines
+
+- Thin vertical border between historical and projected columns
+- Thick bottom border after section totals (e.g., Total Assets)
+- Single bottom border for subtotals
+- Double bottom border for grand totals
+
+## Total and Subtotal Row Formatting
+
+All total and subtotal rows must use **bold font formatting** for their numerical values to clearly distinguish aggregated figures from individual line items.
+
+### Income Statement (P&L) Tab
+| Row | Formatting |
+|-----|------------|
+| Gross Revenue | Bold |
+| Total Cost of Revenue | Bold |
+| Gross Profit | Bold |
+| Total SG&A | Bold |
+| EBITDA | Bold |
+| EBIT | Bold |
+| EBT | Bold |
+| Net Profit After Tax | Bold |
+
+### Balance Sheet Tab
+| Row | Formatting |
+|-----|------------|
+| Total Current Assets | Bold |
+| Total Non-Current Assets | Bold |
+| Total Other Assets | Bold |
+| Total Assets | Bold |
+| Total Current Liabilities | Bold |
+| Total Non-Current Liabilities | Bold |
+| Total Equity | Bold |
+| Total Liabilities and Equity | Bold |
+
+### Cash Flow Statement Tab
+| Row | Formatting |
+|-----|------------|
+| Cash Generated from Operations Before Working Capital Changes | Bold |
+| Total Working Capital Changes | Bold |
+| Net Cash Generated from Operations | Bold |
+| Net Cash Flow from Investing Activities | Bold |
+| Net Cash Flow from Financing Activities | Bold |
+| Closing Cash Balance | Bold |
+
+**Note:** This list is non-exhaustive. Apply bold formatting to any row that represents a total, subtotal, or summary calculation across the model.
+
+## Balance Sheet Check Row Formatting
+
+The Balance Sheet check row (below Total Liabilities and Equity) uses conditional number formatting that displays non-zero values in red. When the balance sheet balances correctly (check = 0), the values display in black or standard formatting.
+
+| Check Value | Font Color |
+|-------------|------------|
+| = 0 (balanced) | Black (standard) |
+| ≠ 0 (error) | Red |
+
+**Implementation:** Apply custom number format `[Red][<>0]0.00;[Red][<>0](0.00);0.00` or use Excel conditional formatting with the rule "Cell Value ≠ 0" → Red font.
+
+## Margin Row Formatting
+
+| Element | Format |
+|---------|--------|
+| Margin % rows | Indent, italics, 1 decimal place |
+| Positive trend | No special formatting (or subtle green) |
+| Negative trend | Flag for review (subtle yellow) |
+| Below peer average | Consider highlighting for discussion |
+
+## Credit Metric Formatting
+
+| Element | Format |
+|---------|--------|
+| Leverage multiples | 1 decimal with "x" suffix (e.g., 2.5x) |
+| Percentages | 1 decimal with "%" suffix |
+| Net Debt negative | Parentheses, indicates net cash position |
+| Section header | Bold, "CREDIT METRICS" |
+| Separator line | Thin border above credit metrics section |
+
+## Credit Metric Threshold Colors
+
+| Metric | Green | Yellow | Red |
+|--------|-------|--------|-----|
+| Total Debt / EBITDA | < 2.5x | 2.5x-4.0x | > 4.0x |
+| Net Debt / EBITDA | < 2.0x | 2.0x-3.5x | > 3.5x |
+| Interest Coverage | > 4.0x | 2.5x-4.0x | < 2.5x |
+| Debt / Total Cap | < 40% | 40%-60% | > 60% |
+| Current Ratio | > 1.5x | 1.0x-1.5x | < 1.0x |
+| Quick Ratio | > 1.0x | 0.75x-1.0x | < 0.75x |
+
+## Conditional Formatting for Checks Tab
+
+- Cell contains pass indicator → Green fill
+- Cell contains fail indicator → Red fill
+- Cell contains warning → Yellow fill
+- Difference cells = 0 → Light green fill
+- Difference cells ≠ 0 → Light red fill
+
+## Margin Reasonability Flags
+
+- Gross Margin < 0% → ERROR: Review COGS
+- Gross Margin > 80% → WARNING: Verify revenue/COGS
+- EBITDA Margin < 0% → FLAG: Operating losses
+- EBITDA Margin > 50% → WARNING: Unusually high
+- Net Margin < 0% → FLAG: Net losses (may be acceptable in growth phase)
+- Net Margin > Gross Margin → ERROR: Formula issue
diff --git a/src/skills/bundled/3-statement-model/references/formulas.md b/src/skills/bundled/3-statement-model/references/formulas.md
new file mode 100644
index 0000000..db26457
--- /dev/null
+++ b/src/skills/bundled/3-statement-model/references/formulas.md
@@ -0,0 +1,292 @@
+# Formula Reference
+
+**IMPORTANT:** Use the formulas outlined in this reference document unless otherwise specified by the user.
+
+---
+
+## Core Linkages
+
+```
+Balance Sheet: Assets = Liabilities + Equity
+Net Income: IS Net Income → CF Operations (starting point)
+Cash Flow: ΔCash = CFO + CFI + CFF
+Cash Tie-Out: Ending Cash (CF) = Cash (BS Asset)
+Cash Monthly/Annual: Closing Cash (Monthly) = Closing Cash (Annual)
+Retained Earnings: Prior RE + Net Income - Dividends = Ending RE
+Equity Raise: ΔCommon Stock/APIC (BS) = Equity Issuance (CFF)
+Year 0 Equity: Equity Raised (Year 0) = Beginning Equity (Year 1)
+```
+
+## Gross Profit Calculation
+
+**IMPORTANT:** Gross Profit must be calculated from Net Revenue, not Gross Revenue.
+
+```
+Net Revenue - Cost of Revenue = Gross Profit
+```
+
+| Term | Definition |
+|------|------------|
+| Gross Revenue | Total revenue before any deductions |
+| Net Revenue | Gross Revenue - Returns - Allowances - Discounts |
+| Cost of Revenue | Direct costs attributable to production of goods/services sold |
+| Gross Profit | Net Revenue - Cost of Revenue |
+
+**Note:** Always use Net Revenue (also called "Net Sales" or simply "Revenue" on most financial statements) as the starting point for profitability calculations. Gross Revenue overstates the true top-line performance.
+
+## Margin Formulas
+
+```
+Gross Margin % = Gross Profit / Net Revenue
+EBITDA = EBIT + D&A (or = Gross Profit - OpEx)
+EBITDA Margin % = EBITDA / Net Revenue
+EBIT Margin % = EBIT / Net Revenue
+Net Income Margin % = Net Income / Net Revenue
+```
+
+## Credit Metric Formulas
+
+```
+Total Debt = Current Portion of Debt + Long-Term Debt
+Net Debt = Total Debt - Cash
+Total Debt / EBITDA = Total Debt / EBITDA (from IS)
+Net Debt / EBITDA = Net Debt / EBITDA (from IS)
+Interest Coverage = EBITDA / Interest Expense (from IS)
+Net Int Exp % Debt = Net Interest Expense / Long-Term Debt
+Debt / Total Cap = Total Debt / (Total Debt + Total Equity)
+Debt / Equity = Total Debt / Total Equity
+Current Ratio = Total Current Assets / Total Current Liabilities
+Quick Ratio = (Total Current Assets - Inventory) / Total Current Liabilities
+```
+
+## Forecast Formulas (% of Net Revenue Method)
+
+```
+Cost of Revenue (Forecast) = Net Revenue × Cost of Revenue % Assumption
+S&M (Forecast) = Net Revenue × S&M % Assumption
+G&A (Forecast) = Net Revenue × G&A % Assumption
+R&D (Forecast) = Net Revenue × R&D % Assumption
+SBC (Forecast) = Net Revenue × SBC % Assumption
+```
+
+## Working Capital Formulas
+
+```
+Accounts Receivable
+ Prior AR
+ + Revenue (from IS)
+ - Cash Collections (plug)
+ = Ending AR
+ DSO = (AR / Revenue) × 365
+
+Inventory
+ Prior Inventory
+ + Purchases (plug)
+ - COGS (from IS)
+ = Ending Inventory
+ DIO = (Inventory / COGS) × 365
+
+Accounts Payable
+ Prior AP
+ + Purchases (from Inventory calc)
+ - Cash Payments (plug)
+ = Ending AP
+ DPO = (AP / COGS) × 365
+
+Net Working Capital = AR + Inventory - AP
+ΔWC = Current NWC - Prior NWC
+```
+
+## D&A Schedule Formulas
+
+```
+Beginning PP&E (Gross)
++ CapEx
+= Ending PP&E (Gross)
+
+Beginning Accumulated Depreciation
++ Depreciation Expense
+= Ending Accumulated Depreciation
+
+PP&E (Net) = Gross PP&E - Accumulated Depreciation
+```
+
+## Debt Schedule Formulas
+
+```
+Beginning Debt Balance
++ New Borrowings
+- Repayments
+= Ending Debt Balance
+
+Interest Expense = Avg Debt Balance × Interest Rate
+ (Use beginning balance to avoid circularity, or iterate if circular refs enabled)
+```
+
+## Retained Earnings Formula
+
+```
+Beginning Retained Earnings
++ Net Income (from IS)
++ Stock-Based Compensation (SBC) (from IS)
+- Dividends
+= Ending Retained Earnings
+```
+
+## NOL (Net Operating Loss) Schedule Formulas
+
+```
+NOL CARRYFORWARD SCHEDULE
+
+Beginning NOL Balance (Year 1 / Formation = 0)
++ NOL Generated (if EBT < 0, then ABS(EBT), else 0)
+- NOL Utilized (limited by taxable income and utilization cap)
+= Ending NOL Balance
+
+STARTING BALANCE RULE
+
+For a new business or first modeled period:
+ Beginning NOL Balance = 0
+ NOL can only increase through realized losses (EBT < 0)
+ NOL cannot be created from thin air or assumed
+
+NOL UTILIZATION CALCULATION
+
+Pre-Tax Income (EBT)
+ If EBT > 0:
+ NOL Available = Beginning NOL Balance
+ Utilization Limit = EBT × 80% (post-2017 federal limit)
+ NOL Utilized = MIN(NOL Available, Utilization Limit)
+ Taxable Income = EBT - NOL Utilized
+ If EBT ≤ 0:
+ NOL Utilized = 0
+ Taxable Income = 0
+ NOL Generated = ABS(EBT)
+
+TAX CALCULATION WITH NOL
+
+Taxes Payable = MAX(0, Taxable Income × Tax Rate)
+ (Taxes cannot be negative; losses create NOL asset instead)
+
+DEFERRED TAX ASSET (DTA) FOR NOL
+
+DTA - NOL Carryforward = Ending NOL Balance × Tax Rate
+ΔDTA = Current DTA - Prior DTA
+ (Increase in DTA = non-cash benefit on CF)
+ (Decrease in DTA = non-cash expense on CF)
+```
+
+## Balance Sheet Structure
+
+```
+ASSETS
+ Cash (from CF ending cash)
+ Accounts Receivable (from WC)
+ Inventory (from WC)
+ Total Current Assets
+
+ PP&E, Net (from DA)
+ Deferred Tax Asset - NOL (from NOL schedule)
+ Total Non-Current Assets
+ Total Assets
+
+LIABILITIES
+ Accounts Payable (from WC)
+ Current Portion of Debt (from Debt)
+ Total Current Liabilities
+
+ Long-Term Debt (from Debt)
+ Total Liabilities
+
+EQUITY
+ Common Stock
+ Retained Earnings (from RE schedule)
+ Total Equity
+
+CHECK: Assets - Liabilities - Equity = 0
+```
+
+## Cash Flow Statement Structure
+
+```
+CASH FROM OPERATIONS (CFO)
+ Net Income (LINK: IS)
+ + D&A (LINK: DA schedule)
+ + Stock-Based Compensation (SBC) (LINK: IS or Assumptions)
+ - ΔDTA (Deferred Tax Asset) (LINK: NOL schedule; increase in DTA = use of cash)
+ - ΔAR (LINK: WC)
+ - ΔInventory (LINK: WC)
+ + ΔAP (LINK: WC)
+ = CFO
+
+CASH FROM INVESTING (CFI)
+ - CapEx (LINK: DA schedule)
+ = CFI
+
+CASH FROM FINANCING (CFF)
+ + Debt Issuance (LINK: Debt)
+ - Debt Repayment (LINK: Debt)
+ + Equity Issuance (LINK: BS Common Stock/APIC)
+ - Dividends (LINK: RE schedule)
+ = CFF
+
+Net Change in Cash = CFO + CFI + CFF
+Beginning Cash
++ Net Change in Cash
+= Ending Cash (LINK TO: BS Cash)
+```
+
+## Income Statement Structure
+
+```
+Net Revenue
+ Growth %
+(-) Cost of Revenue
+ % of Net Revenue
+────────────────
+Gross Profit (= Net Revenue - Cost of Revenue)
+ Gross Margin %
+
+(-) S&M
+ % of Net Revenue
+(-) G&A
+ % of Net Revenue
+(-) R&D
+ % of Net Revenue
+(-) D&A
+(-) SBC
+ % of Net Revenue
+────────────────
+EBIT
+ EBIT Margin %
+
+EBITDA
+ EBITDA Margin %
+
+(-) Interest Expense
+────────────────
+EBT (Pre-Tax Income)
+(-) NOL Utilization (from NOL schedule, reduces taxable income)
+────────────────
+Taxable Income
+(-) Taxes (Taxable Income × Tax Rate)
+────────────────
+Net Income
+ Net Income Margin %
+```
+
+## Check Formulas
+
+```
+BS Balance Check: = Assets - Liabilities - Equity (must = 0)
+Cash Tie-Out: = BS Cash - CF Ending Cash (must = 0)
+RE Roll-Forward: = Prior RE + NI + SBC - Div - BS RE (must = 0)
+DTA Tie-Out: = NOL Schedule DTA - BS DTA (must = 0)
+Equity Raise Tie-Out: = ΔCommon Stock/APIC (BS) - Equity Issuance (CFF) (must = 0)
+Year 0 Equity Tie-Out: = Equity Raised (Year 0) - Beginning Equity (Year 1) (must = 0)
+Cash Monthly vs Annual: = Closing Cash (Monthly) - Closing Cash (Annual) (must = 0)
+NOL Utilization Cap: = NOL Utilized ≤ EBT × 80% (must be TRUE for post-2017)
+NOL Non-Negative: = Ending NOL Balance ≥ 0 (must be TRUE)
+NOL Starting Balance: = Beginning NOL (Year 1) = 0 (must be TRUE for new business)
+NOL Accumulation: = NOL increases only when EBT < 0 (losses generate NOL)
+```
diff --git a/src/skills/bundled/3-statement-model/references/sec-filings.md b/src/skills/bundled/3-statement-model/references/sec-filings.md
new file mode 100644
index 0000000..e0fa484
--- /dev/null
+++ b/src/skills/bundled/3-statement-model/references/sec-filings.md
@@ -0,0 +1,125 @@
+# SEC Filings Data Extraction Reference
+
+**When to Use:** Only reference this file when a model template specifically requires pulling data from SEC filings (10-K, 10-Q). For templates that provide data directly or use other data sources, this reference is not needed.
+
+---
+
+## Extracting Data from SEC Filings (10-K / 10-Q)
+
+When populating a model template with public company data, extract financials directly from SEC filings.
+
+### Step 1: Locate the Filing
+
+1. Use SEC EDGAR: `https://www.sec.gov/cgi-bin/browse-edgar?action=getcompany&CIK=[TICKER]&type=10-K`
+2. For quarterly data, use `type=10-Q`
+
+### Step 2: Identify Filing Currency
+
+Before extracting data, identify the reporting currency:
+- Check the cover page or header for reporting currency
+- Look at statement headers (e.g., "in thousands of U.S. dollars")
+- Review Note 1 (Summary of Significant Accounting Policies)
+
+**Common Currency Indicators**
+
+| Indicator | Currency |
+|-----------|----------|
+| $, USD | US Dollar |
+| €, EUR | Euro |
+| £, GBP | British Pound |
+| ¥, JPY | Japanese Yen |
+| ¥, CNY, RMB | Chinese Yuan |
+| CHF | Swiss Franc |
+| CAD, C$ | Canadian Dollar |
+
+Set model currency to match filing; document in Assumptions tab.
+
+### Step 3: Navigate to Financial Statements
+
+Within the 10-K or 10-Q, locate:
+- **Item 8** (10-K) or **Item 1** (10-Q): Financial Statements
+- Key sections to extract:
+ - Consolidated Statements of Operations (Income Statement)
+ - Consolidated Balance Sheets
+ - Consolidated Statements of Cash Flows
+ - Notes to Financial Statements (for schedule details)
+
+### Step 4: Data Extraction Mapping
+
+**Income Statement (from Consolidated Statements of Operations)**
+
+| Filing Line Item | Model Line Item |
+|------------------|-----------------|
+| Net revenues / Net sales | Revenue |
+| Cost of goods sold | COGS |
+| Selling, general and administrative | SG&A |
+| Depreciation and amortization | D&A |
+| Interest expense, net | Interest Expense |
+| Income tax expense | Taxes |
+| Net income | Net Income |
+
+**Balance Sheet (from Consolidated Balance Sheets)**
+
+| Filing Line Item | Model Line Item |
+|------------------|-----------------|
+| Cash and cash equivalents | Cash |
+| Accounts receivable, net | AR |
+| Inventories | Inventory |
+| Property, plant and equipment, net | PP&E (Net) |
+| Total assets | Total Assets |
+| Accounts payable | AP |
+| Short-term debt / Current portion of LT debt | Current Debt |
+| Long-term debt | LT Debt |
+| Retained earnings | Retained Earnings |
+| Total stockholders' equity | Total Equity |
+
+**Cash Flow Statement (from Consolidated Statements of Cash Flows)**
+
+| Filing Line Item | Model Line Item |
+|------------------|-----------------|
+| Net income | Net Income |
+| Depreciation and amortization | D&A |
+| Changes in accounts receivable | ΔAR |
+| Changes in inventories | ΔInventory |
+| Changes in accounts payable | ΔAP |
+| Capital expenditures | CapEx |
+| Proceeds from issuance of common stock | Equity Issuance |
+| Proceeds from / Repayments of debt | Debt activity |
+| Dividends paid | Dividends |
+
+### Step 5: Extract Supporting Detail from Notes
+
+For schedules, pull from Notes to Financial Statements:
+- **Note: Debt** → Maturity schedule, interest rates, covenants
+- **Note: Property, Plant & Equipment** → Gross PP&E, accumulated depreciation, useful lives
+- **Note: Revenue** → Segment breakdowns, geographic splits
+- **Note: Leases** → Operating vs. finance lease obligations
+
+### Step 6: Historical Data Requirements
+
+Extract 3 years of historical data minimum:
+- 10-K provides 3 years of IS/CF, 2 years of BS
+- For 3rd year BS, pull from prior year's 10-K
+- Use 10-Qs to fill in quarterly granularity if needed
+
+### Data Extraction Checklist
+
+- Identify reporting currency and scale (thousands, millions)
+- 3 years historical Income Statement
+- 3 years historical Cash Flow Statement
+- 3 years historical Balance Sheet
+- Verify IS Net Income = CF starting Net Income (each year)
+- Verify BS Cash = CF Ending Cash (each year)
+- Extract debt maturity schedule from notes
+- Extract D&A detail or useful life assumptions
+- Note any non-recurring / one-time items to normalize
+
+### Handling Common Filing Variations
+
+| Variation | How to Handle |
+|-----------|---------------|
+| D&A embedded in COGS/SG&A | Pull D&A from Cash Flow Statement |
+| "Other" line items are material | Check notes for breakdown |
+| Restatements | Use restated figures, note in assumptions |
+| Fiscal year ≠ calendar year | Label with fiscal year end (e.g., FYE Jan 2025) |
+| Non-USD reporting currency | Adapt model currency to match filing |
diff --git a/src/skills/bundled/accelerate/SKILL.md b/src/skills/bundled/accelerate/SKILL.md
new file mode 100644
index 0000000..128123f
--- /dev/null
+++ b/src/skills/bundled/accelerate/SKILL.md
@@ -0,0 +1,352 @@
+---
+name: accelerate
+description: Run PyTorch training across GPUs with minimal changes.
+version: 1.0.1
+author: Orchestra Research
+license: MIT
+dependencies: [accelerate, torch, transformers]
+platforms: [linux, macos, windows]
+metadata:
+ hermes:
+ tags: [Distributed Training, HuggingFace, Accelerate, DeepSpeed, FSDP, Mixed Precision, PyTorch, DDP, Unified API, Simple]
+
+---
+
+# HuggingFace Accelerate - Unified Distributed Training
+
+## Quick start
+
+Accelerate simplifies distributed training to 4 lines of code.
+
+**Installation**:
+```bash
+pip install accelerate
+```
+
+**Convert PyTorch script** (4 lines):
+```python
+import torch
++ from accelerate import Accelerator
+
++ accelerator = Accelerator()
+
+ model = torch.nn.Transformer()
+ optimizer = torch.optim.Adam(model.parameters())
+ dataloader = torch.utils.data.DataLoader(dataset)
+
++ model, optimizer, dataloader = accelerator.prepare(model, optimizer, dataloader)
+
+ for batch in dataloader:
+ optimizer.zero_grad()
+ loss = model(batch)
+- loss.backward()
++ accelerator.backward(loss)
+ optimizer.step()
+```
+
+**Run** (single command):
+```bash
+accelerate launch train.py
+```
+
+## Common workflows
+
+### Workflow 1: From single GPU to multi-GPU
+
+**Original script**:
+```python
+# train.py
+import torch
+
+model = torch.nn.Linear(10, 2).to('cuda')
+optimizer = torch.optim.Adam(model.parameters())
+dataloader = torch.utils.data.DataLoader(dataset, batch_size=32)
+
+for epoch in range(10):
+ for batch in dataloader:
+ batch = batch.to('cuda')
+ optimizer.zero_grad()
+ loss = model(batch).mean()
+ loss.backward()
+ optimizer.step()
+```
+
+**With Accelerate** (4 lines added):
+```python
+# train.py
+import torch
+from accelerate import Accelerator # +1
+
+accelerator = Accelerator() # +2
+
+model = torch.nn.Linear(10, 2)
+optimizer = torch.optim.Adam(model.parameters())
+dataloader = torch.utils.data.DataLoader(dataset, batch_size=32)
+
+model, optimizer, dataloader = accelerator.prepare(model, optimizer, dataloader) # +3
+
+for epoch in range(10):
+ for batch in dataloader:
+ # No .to('cuda') needed - automatic!
+ optimizer.zero_grad()
+ loss = model(batch).mean()
+ accelerator.backward(loss) # +4
+ optimizer.step()
+```
+
+**Configure** (interactive):
+```bash
+accelerate config
+```
+
+**Questions**:
+- Which machine? (single/multi GPU/TPU/CPU)
+- How many machines? (1)
+- Mixed precision? (no/fp16/bf16/fp8)
+- DeepSpeed? (no/yes)
+
+**Launch** (works on any setup):
+```bash
+# Single GPU
+accelerate launch train.py
+
+# Multi-GPU (8 GPUs)
+accelerate launch --multi_gpu --num_processes 8 train.py
+
+# Multi-node
+accelerate launch --multi_gpu --num_processes 16 \
+ --num_machines 2 --machine_rank 0 \
+ --main_process_ip $MASTER_ADDR \
+ train.py
+```
+
+### Workflow 2: Mixed precision training
+
+**Enable FP16/BF16**:
+```python
+from accelerate import Accelerator
+
+# FP16 (with gradient scaling)
+accelerator = Accelerator(mixed_precision='fp16')
+
+# BF16 (no scaling, more stable)
+accelerator = Accelerator(mixed_precision='bf16')
+
+# FP8 (H100+)
+accelerator = Accelerator(mixed_precision='fp8')
+
+model, optimizer, dataloader = accelerator.prepare(model, optimizer, dataloader)
+
+# Everything else is automatic!
+for batch in dataloader:
+ with accelerator.autocast(): # Optional, done automatically
+ loss = model(batch)
+ accelerator.backward(loss)
+```
+
+### Workflow 3: DeepSpeed ZeRO integration
+
+**Enable DeepSpeed ZeRO-2** (pass a `DeepSpeedPlugin`, not a raw dict):
+```python
+from accelerate import Accelerator, DeepSpeedPlugin
+
+deepspeed_plugin = DeepSpeedPlugin(
+ zero_stage=2, # ZeRO-2
+ offload_optimizer_device="none", # or "cpu" to offload
+ gradient_accumulation_steps=4,
+)
+
+accelerator = Accelerator(
+ mixed_precision='bf16',
+ deepspeed_plugin=deepspeed_plugin, # DeepSpeedPlugin instance (or dict[str, DeepSpeedPlugin])
+)
+
+# Same code as before!
+model, optimizer, dataloader = accelerator.prepare(model, optimizer, dataloader)
+```
+
+**Or point at a full DeepSpeed JSON config via the plugin**:
+```python
+from accelerate import Accelerator, DeepSpeedPlugin
+
+# hf_ds_config accepts a path to a DeepSpeed config JSON (or a dict)
+deepspeed_plugin = DeepSpeedPlugin(hf_ds_config="ds_config.json")
+accelerator = Accelerator(mixed_precision='bf16', deepspeed_plugin=deepspeed_plugin)
+```
+
+**ds_config.json** (a raw DeepSpeed config — passed via the plugin, NOT via `--config_file`):
+```json
+{
+ "fp16": {"enabled": false},
+ "bf16": {"enabled": true},
+ "zero_optimization": {
+ "stage": 2,
+ "offload_optimizer": {"device": "cpu"},
+ "allgather_bucket_size": 5e8,
+ "reduce_bucket_size": 5e8
+ }
+}
+```
+
+**Or via interactive config**:
+```bash
+accelerate config
+# Select: DeepSpeed → ZeRO-2
+# This writes an accelerate YAML config (default: ~/.cache/huggingface/accelerate/default_config.yaml)
+```
+
+**Launch** (`--config_file` expects an accelerate YAML, not a raw DeepSpeed JSON):
+```bash
+# Uses the default accelerate config written by `accelerate config`
+accelerate launch train.py
+
+# Or point at a specific accelerate YAML
+accelerate launch --config_file accelerate_deepspeed.yaml train.py
+```
+
+### Workflow 4: FSDP (Fully Sharded Data Parallel)
+
+**Enable FSDP**:
+```python
+from accelerate import Accelerator, FullyShardedDataParallelPlugin
+
+fsdp_plugin = FullyShardedDataParallelPlugin(
+ sharding_strategy="FULL_SHARD", # ZeRO-3 equivalent
+ auto_wrap_policy="transformer_based_wrap", # valid: transformer_based_wrap | size_based_wrap | no_wrap
+ cpu_offload=False
+)
+
+accelerator = Accelerator(
+ mixed_precision='bf16',
+ fsdp_plugin=fsdp_plugin
+)
+
+model, optimizer, dataloader = accelerator.prepare(model, optimizer, dataloader)
+```
+
+**Or via config**:
+```bash
+accelerate config
+# Select: FSDP → Full Shard → No CPU Offload
+```
+
+### Workflow 5: Gradient accumulation
+
+**Accumulate gradients**:
+```python
+from accelerate import Accelerator
+
+accelerator = Accelerator(gradient_accumulation_steps=4)
+
+model, optimizer, dataloader = accelerator.prepare(model, optimizer, dataloader)
+
+for batch in dataloader:
+ with accelerator.accumulate(model): # Handles accumulation
+ optimizer.zero_grad()
+ loss = model(batch)
+ accelerator.backward(loss)
+ optimizer.step()
+```
+
+**Effective batch size**: `batch_size * num_gpus * gradient_accumulation_steps`
+
+## When to use vs alternatives
+
+**Use Accelerate when**:
+- Want simplest distributed training
+- Need single script for any hardware
+- Use HuggingFace ecosystem
+- Want flexibility (DDP/DeepSpeed/FSDP/Megatron)
+- Need quick prototyping
+
+**Key advantages**:
+- **4 lines**: Minimal code changes
+- **Unified API**: Same code for DDP, DeepSpeed, FSDP, Megatron
+- **Automatic**: Device placement, mixed precision, sharding
+- **Interactive config**: No manual launcher setup
+- **Single launch**: Works everywhere
+
+**Use alternatives instead**:
+- **PyTorch Lightning**: Need callbacks, high-level abstractions
+- **Ray Train**: Multi-node orchestration, hyperparameter tuning
+- **DeepSpeed**: Direct API control, advanced features
+- **Raw DDP**: Maximum control, minimal abstraction
+
+## Common issues
+
+**Issue: Wrong device placement**
+
+Don't manually move to device:
+```python
+# WRONG
+batch = batch.to('cuda')
+
+# CORRECT
+# Accelerate handles it automatically after prepare()
+```
+
+**Issue: Gradient accumulation not working**
+
+Use context manager:
+```python
+# CORRECT
+with accelerator.accumulate(model):
+ optimizer.zero_grad()
+ accelerator.backward(loss)
+ optimizer.step()
+```
+
+**Issue: Checkpointing in distributed**
+
+Use accelerator methods:
+```python
+# Save only on main process
+if accelerator.is_main_process:
+ accelerator.save_state('checkpoint/')
+
+# Load on all processes
+accelerator.load_state('checkpoint/')
+```
+
+**Issue: Different results with FSDP**
+
+Ensure same random seed:
+```python
+from accelerate.utils import set_seed
+set_seed(42)
+```
+
+## Advanced topics
+
+**Megatron integration**: See [references/megatron-integration.md](references/megatron-integration.md) for tensor parallelism, pipeline parallelism, and sequence parallelism setup.
+
+**Custom plugins**: See [references/custom-plugins.md](references/custom-plugins.md) for creating custom distributed plugins and advanced configuration.
+
+**Performance tuning**: See [references/performance.md](references/performance.md) for profiling, memory optimization, and best practices.
+
+## Hardware requirements
+
+- **CPU**: Works (slow)
+- **Single GPU**: Works
+- **Multi-GPU**: DDP (default), DeepSpeed, or FSDP
+- **Multi-node**: DDP, DeepSpeed, FSDP, Megatron
+- **TPU**: Supported
+- **Apple MPS**: Supported
+
+**Launcher requirements**:
+- **DDP**: `torch.distributed.run` (built-in)
+- **DeepSpeed**: `deepspeed` (pip install deepspeed)
+- **FSDP**: PyTorch 1.12+ (built-in)
+- **Megatron**: Custom setup
+
+## Resources
+
+- Docs: https://huggingface.co/docs/accelerate
+- GitHub: https://github.com/huggingface/accelerate
+- Version: 1.11.0+
+- Tutorial: "Accelerate your scripts"
+- Examples: https://github.com/huggingface/accelerate/tree/main/examples
+- Used by: HuggingFace Transformers, TRL, PEFT, all HF libraries
+
+
+
diff --git a/src/skills/bundled/accelerate/references/custom-plugins.md b/src/skills/bundled/accelerate/references/custom-plugins.md
new file mode 100644
index 0000000..d8207ee
--- /dev/null
+++ b/src/skills/bundled/accelerate/references/custom-plugins.md
@@ -0,0 +1,453 @@
+# Custom Plugins for Accelerate
+
+## Overview
+
+Accelerate allows creating **custom plugins** to extend distributed training strategies beyond built-in options (DDP, FSDP, DeepSpeed).
+
+## Plugin Architecture
+
+### Base Plugin Structure
+
+```python
+from accelerate.utils import DistributedDataParallelKwargs
+from dataclasses import dataclass
+
+@dataclass
+class CustomPlugin:
+ """Custom training plugin."""
+
+ # Plugin configuration
+ param1: int = 1
+ param2: str = "default"
+
+ def __post_init__(self):
+ # Validation logic
+ if self.param1 < 1:
+ raise ValueError("param1 must be >= 1")
+```
+
+### Using Custom Plugin
+
+```python
+from accelerate import Accelerator
+
+# Create plugin
+custom_plugin = CustomPlugin(param1=4, param2="value")
+
+# Pass to Accelerator
+accelerator = Accelerator(
+ custom_plugin=custom_plugin # Not a real parameter, example only
+)
+```
+
+## Built-In Plugin Examples
+
+### 1. GradScalerKwargs (FP16 Configuration)
+
+```python
+from accelerate.utils import GradScalerKwargs
+
+# Configure gradient scaler for FP16
+scaler_kwargs = GradScalerKwargs(
+ init_scale=2.**16, # Initial loss scale
+ growth_factor=2.0, # Scale growth rate
+ backoff_factor=0.5, # Scale backoff rate
+ growth_interval=2000, # Steps between scale increases
+ enabled=True # Enable scaler
+)
+
+accelerator = Accelerator(
+ mixed_precision='fp16',
+ kwargs_handlers=[scaler_kwargs] # Pass as kwargs handler
+)
+```
+
+**Use case**: Fine-tune FP16 gradient scaling behavior
+
+### 2. DistributedDataParallelKwargs
+
+```python
+from accelerate.utils import DistributedDataParallelKwargs
+
+# Configure DDP behavior
+ddp_kwargs = DistributedDataParallelKwargs(
+ bucket_cap_mb=25, # Gradient bucketing size
+ find_unused_parameters=False, # Find unused params (slower)
+ check_reduction=False, # Check gradient reduction
+ gradient_as_bucket_view=True, # Memory optimization
+ static_graph=False # Static computation graph
+)
+
+accelerator = Accelerator(
+ kwargs_handlers=[ddp_kwargs]
+)
+```
+
+**Use case**: Optimize DDP performance for specific models
+
+### 3. FP8RecipeKwargs (H100 FP8)
+
+```python
+from accelerate.utils import FP8RecipeKwargs
+
+# Configure FP8 training (H100)
+fp8_recipe = FP8RecipeKwargs(
+ backend="te", # TransformerEngine backend
+ margin=0, # Scaling margin
+ interval=1, # Scaling interval
+ fp8_format="HYBRID", # E4M3 + E5M2 hybrid
+ amax_history_len=1024, # AMAX history length
+ amax_compute_algo="max" # AMAX computation algorithm
+)
+
+accelerator = Accelerator(
+ mixed_precision='fp8',
+ kwargs_handlers=[fp8_recipe]
+)
+```
+
+**Use case**: Ultra-fast training on H100 GPUs
+
+## Custom DeepSpeed Configuration
+
+### ZeRO-3 with CPU Offload
+
+```python
+from accelerate import Accelerator
+from accelerate.utils import DeepSpeedPlugin
+
+# Custom DeepSpeed config
+ds_plugin = DeepSpeedPlugin(
+ zero_stage=3, # ZeRO-3
+ offload_optimizer_device="cpu", # CPU offload optimizer
+ offload_param_device="cpu", # CPU offload parameters
+ zero3_init_flag=True, # ZeRO-3 initialization
+ zero3_save_16bit_model=True, # Save FP16 weights
+)
+
+accelerator = Accelerator(
+ deepspeed_plugin=ds_plugin,
+ mixed_precision='bf16'
+)
+```
+
+### ZeRO-2 with NVMe Offload
+
+```python
+ds_plugin = DeepSpeedPlugin(
+ zero_stage=2,
+ offload_optimizer_device="nvme", # NVMe offload
+ offload_param_device="nvme",
+ nvme_path="/local_nvme", # NVMe mount path
+)
+```
+
+### Custom JSON Config
+
+```python
+import json
+
+# Load custom DeepSpeed config
+with open('deepspeed_config.json', 'r') as f:
+ ds_config = json.load(f)
+
+ds_plugin = DeepSpeedPlugin(hf_ds_config=ds_config)
+
+accelerator = Accelerator(deepspeed_plugin=ds_plugin)
+```
+
+**Example config** (`deepspeed_config.json`):
+```json
+{
+ "train_batch_size": "auto",
+ "train_micro_batch_size_per_gpu": "auto",
+ "gradient_accumulation_steps": "auto",
+ "gradient_clipping": 1.0,
+ "zero_optimization": {
+ "stage": 3,
+ "offload_optimizer": {
+ "device": "cpu",
+ "pin_memory": true
+ },
+ "offload_param": {
+ "device": "cpu",
+ "pin_memory": true
+ },
+ "overlap_comm": true,
+ "contiguous_gradients": true,
+ "sub_group_size": 1e9,
+ "reduce_bucket_size": 5e8,
+ "stage3_prefetch_bucket_size": 5e8,
+ "stage3_param_persistence_threshold": 1e6,
+ "stage3_max_live_parameters": 1e9,
+ "stage3_max_reuse_distance": 1e9,
+ "stage3_gather_16bit_weights_on_model_save": true
+ },
+ "bf16": {
+ "enabled": true
+ },
+ "steps_per_print": 100,
+ "wall_clock_breakdown": false
+}
+```
+
+## Custom FSDP Configuration
+
+### FSDP with Custom Auto-Wrap Policy
+
+```python
+from accelerate.utils import FullyShardedDataParallelPlugin
+from torch.distributed.fsdp import BackwardPrefetch, ShardingStrategy
+from torch.distributed.fsdp.wrap import size_based_auto_wrap_policy
+import functools
+
+# Custom wrap policy (size-based)
+wrap_policy = functools.partial(
+ size_based_auto_wrap_policy,
+ min_num_params=1e6 # Wrap layers with 1M+ params
+)
+
+fsdp_plugin = FullyShardedDataParallelPlugin(
+ sharding_strategy=ShardingStrategy.FULL_SHARD, # ZeRO-3 equivalent
+ backward_prefetch=BackwardPrefetch.BACKWARD_PRE, # Prefetch strategy
+ mixed_precision_policy=None, # Use Accelerator's mixed precision
+ auto_wrap_policy=wrap_policy, # Custom wrapping
+ cpu_offload=False,
+ ignored_modules=None, # Modules to not wrap
+ state_dict_type="FULL_STATE_DICT", # Save format
+ optim_state_dict_config=None,
+ limit_all_gathers=False,
+ use_orig_params=True, # Use original param shapes
+)
+
+accelerator = Accelerator(
+ fsdp_plugin=fsdp_plugin,
+ mixed_precision='bf16'
+)
+```
+
+### FSDP with Transformer Auto-Wrap
+
+```python
+from torch.distributed.fsdp.wrap import transformer_auto_wrap_policy
+from transformers.models.gpt2.modeling_gpt2 import GPT2Block
+
+# Wrap at transformer block level
+wrap_policy = functools.partial(
+ transformer_auto_wrap_policy,
+ transformer_layer_cls={GPT2Block} # Wrap GPT2Block layers
+)
+
+fsdp_plugin = FullyShardedDataParallelPlugin(
+ auto_wrap_policy=wrap_policy
+)
+```
+
+## Creating Custom Training Strategy
+
+### Example: Custom Gradient Accumulation
+
+```python
+from accelerate import Accelerator
+
+class CustomGradientAccumulation:
+ def __init__(self, steps=4, adaptive=False):
+ self.steps = steps
+ self.adaptive = adaptive
+ self.current_step = 0
+
+ def should_sync(self, loss):
+ """Decide whether to sync gradients."""
+ self.current_step += 1
+
+ # Adaptive: sync on high loss
+ if self.adaptive and loss > threshold:
+ self.current_step = 0
+ return True
+
+ # Regular: sync every N steps
+ if self.current_step >= self.steps:
+ self.current_step = 0
+ return True
+
+ return False
+
+# Usage
+custom_accum = CustomGradientAccumulation(steps=8, adaptive=True)
+accelerator = Accelerator()
+
+for batch in dataloader:
+ outputs = model(**batch)
+ loss = outputs.loss
+
+ # Scale loss
+ loss = loss / custom_accum.steps
+ accelerator.backward(loss)
+
+ # Conditional sync
+ if custom_accum.should_sync(loss.item()):
+ optimizer.step()
+ optimizer.zero_grad()
+```
+
+### Example: Custom Mixed Precision
+
+```python
+import torch
+
+class CustomMixedPrecision:
+ """Custom mixed precision with dynamic loss scaling."""
+
+ def __init__(self, init_scale=2**16, scale_window=2000):
+ self.scaler = torch.cuda.amp.GradScaler(
+ init_scale=init_scale,
+ growth_interval=scale_window
+ )
+ self.scale_history = []
+
+ def scale_loss(self, loss):
+ """Scale loss for backward."""
+ return self.scaler.scale(loss)
+
+ def unscale_and_clip(self, optimizer, max_norm=1.0):
+ """Unscale gradients and clip."""
+ self.scaler.unscale_(optimizer)
+ torch.nn.utils.clip_grad_norm_(
+ optimizer.param_groups[0]['params'],
+ max_norm
+ )
+
+ def step(self, optimizer):
+ """Optimizer step with scaler update."""
+ scale_before = self.scaler.get_scale()
+ self.scaler.step(optimizer)
+ self.scaler.update()
+ scale_after = self.scaler.get_scale()
+
+ # Track scale changes
+ if scale_before != scale_after:
+ self.scale_history.append(scale_after)
+
+# Usage
+custom_mp = CustomMixedPrecision()
+
+for batch in dataloader:
+ with torch.cuda.amp.autocast(dtype=torch.float16):
+ loss = model(**batch).loss
+
+ scaled_loss = custom_mp.scale_loss(loss)
+ scaled_loss.backward()
+
+ custom_mp.unscale_and_clip(optimizer, max_norm=1.0)
+ custom_mp.step(optimizer)
+ optimizer.zero_grad()
+```
+
+## Advanced: Custom Distributed Backend
+
+### Custom AllReduce Strategy
+
+```python
+import torch.distributed as dist
+
+class CustomAllReduce:
+ """Custom all-reduce with compression."""
+
+ def __init__(self, compression_ratio=0.1):
+ self.compression_ratio = compression_ratio
+
+ def compress_gradients(self, tensor):
+ """Top-k gradient compression."""
+ k = int(tensor.numel() * self.compression_ratio)
+ values, indices = torch.topk(tensor.abs().view(-1), k)
+ return values, indices
+
+ def all_reduce_compressed(self, tensor):
+ """All-reduce with gradient compression."""
+ # Compress
+ values, indices = self.compress_gradients(tensor)
+
+ # All-reduce compressed gradients
+ dist.all_reduce(values, op=dist.ReduceOp.SUM)
+
+ # Decompress
+ tensor_compressed = torch.zeros_like(tensor).view(-1)
+ tensor_compressed[indices] = values / dist.get_world_size()
+
+ return tensor_compressed.view_as(tensor)
+
+# Usage in training loop
+custom_ar = CustomAllReduce(compression_ratio=0.1)
+
+for batch in dataloader:
+ loss = model(**batch).loss
+ loss.backward()
+
+ # Custom all-reduce
+ for param in model.parameters():
+ if param.grad is not None:
+ param.grad.data = custom_ar.all_reduce_compressed(param.grad.data)
+
+ optimizer.step()
+ optimizer.zero_grad()
+```
+
+## Plugin Best Practices
+
+### 1. Validation in `__post_init__`
+
+```python
+@dataclass
+class CustomPlugin:
+ learning_rate: float = 1e-3
+ warmup_steps: int = 1000
+
+ def __post_init__(self):
+ # Validate parameters
+ if self.learning_rate <= 0:
+ raise ValueError("learning_rate must be positive")
+ if self.warmup_steps < 0:
+ raise ValueError("warmup_steps must be non-negative")
+
+ # Compute derived values
+ self.min_lr = self.learning_rate * 0.1
+```
+
+### 2. Compatibility Checks
+
+```python
+@dataclass
+class CustomPlugin:
+ feature_enabled: bool = True
+
+ def is_compatible(self, accelerator):
+ """Check if plugin is compatible with accelerator config."""
+ if self.feature_enabled and accelerator.mixed_precision == 'fp8':
+ raise ValueError("Custom plugin not compatible with FP8")
+ return True
+```
+
+### 3. State Management
+
+```python
+@dataclass
+class CustomPlugin:
+ counter: int = 0
+ history: list = None
+
+ def __post_init__(self):
+ if self.history is None:
+ self.history = []
+
+ def update_state(self, value):
+ """Update plugin state during training."""
+ self.counter += 1
+ self.history.append(value)
+```
+
+## Resources
+
+- Accelerate Plugins: https://huggingface.co/docs/accelerate/package_reference/kwargs
+- DeepSpeed Config: https://www.deepspeed.ai/docs/config-json/
+- FSDP Guide: https://pytorch.org/docs/stable/fsdp.html
+- Custom Training Loops: https://huggingface.co/docs/accelerate/usage_guides/training_tpu
diff --git a/src/skills/bundled/accelerate/references/megatron-integration.md b/src/skills/bundled/accelerate/references/megatron-integration.md
new file mode 100644
index 0000000..61b025b
--- /dev/null
+++ b/src/skills/bundled/accelerate/references/megatron-integration.md
@@ -0,0 +1,489 @@
+# Megatron Integration with Accelerate
+
+## Overview
+
+Accelerate supports Megatron-LM for massive model training with tensor parallelism and pipeline parallelism.
+
+**Megatron capabilities**:
+- **Tensor Parallelism (TP)**: Split layers across GPUs
+- **Pipeline Parallelism (PP)**: Split model depth across GPUs
+- **Data Parallelism (DP)**: Replicate model across GPU groups
+- **Sequence Parallelism**: Split sequences for long contexts
+
+## Setup
+
+### Install Megatron-LM
+
+```bash
+# Clone Megatron-LM repository
+git clone https://github.com/NVIDIA/Megatron-LM.git
+cd Megatron-LM
+pip install -e .
+
+# Install Apex (NVIDIA optimizations)
+git clone https://github.com/NVIDIA/apex
+cd apex
+pip install -v --disable-pip-version-check --no-cache-dir --no-build-isolation \
+ --config-settings "--build-option=--cpp_ext" --config-settings "--build-option=--cuda_ext" ./
+```
+
+### Accelerate Configuration
+
+```bash
+accelerate config
+```
+
+**Questions**:
+```
+In which compute environment are you running?
+> This machine
+
+Which type of machine are you using?
+> Multi-GPU
+
+How many different machines will you use?
+> 1
+
+Do you want to use DeepSpeed/FSDP?
+> No
+
+Do you want to use Megatron-LM?
+> Yes
+
+What is the Tensor Parallelism degree? [1-8]
+> 2
+
+Do you want to enable Sequence Parallelism?
+> No
+
+What is the Pipeline Parallelism degree? [1-8]
+> 2
+
+What is the Data Parallelism degree? [1-8]
+> 2
+
+Where to perform activation checkpointing? ['SELECTIVE', 'FULL', 'NONE']
+> SELECTIVE
+
+Where to perform activation partitioning? ['SEQUENTIAL', 'UNIFORM']
+> SEQUENTIAL
+```
+
+**Generated config** (`~/.cache/huggingface/accelerate/default_config.yaml`):
+```yaml
+compute_environment: LOCAL_MACHINE
+distributed_type: MEGATRON_LM
+downcast_bf16: 'no'
+machine_rank: 0
+main_training_function: main
+megatron_lm_config:
+ megatron_lm_gradient_clipping: 1.0
+ megatron_lm_learning_rate_decay_iters: 320000
+ megatron_lm_num_micro_batches: 1
+ megatron_lm_pp_degree: 2
+ megatron_lm_recompute_activations: true
+ megatron_lm_sequence_parallelism: false
+ megatron_lm_tp_degree: 2
+mixed_precision: bf16
+num_machines: 1
+num_processes: 8
+rdzv_backend: static
+same_network: true
+tpu_env: []
+tpu_use_cluster: false
+tpu_use_sudo: false
+use_cpu: false
+```
+
+## Parallelism Strategies
+
+### Tensor Parallelism (TP)
+
+**Splits each transformer layer across GPUs**:
+
+```python
+# Layer split across 2 GPUs
+# GPU 0: First half of attention heads
+# GPU 1: Second half of attention heads
+
+# Each GPU computes partial outputs
+# All-reduce combines results
+```
+
+**TP degree recommendations**:
+- **TP=1**: No tensor parallelism (single GPU per layer)
+- **TP=2**: 2 GPUs per layer (good for 7-13B models)
+- **TP=4**: 4 GPUs per layer (good for 20-40B models)
+- **TP=8**: 8 GPUs per layer (good for 70B+ models)
+
+**Benefits**:
+- Reduces memory per GPU
+- All-reduce communication (fast)
+
+**Drawbacks**:
+- Requires fast inter-GPU bandwidth (NVLink)
+- Communication overhead per layer
+
+### Pipeline Parallelism (PP)
+
+**Splits model depth across GPUs**:
+
+```python
+# 12-layer model, PP=4
+# GPU 0: Layers 0-2
+# GPU 1: Layers 3-5
+# GPU 2: Layers 6-8
+# GPU 3: Layers 9-11
+```
+
+**PP degree recommendations**:
+- **PP=1**: No pipeline parallelism
+- **PP=2**: 2 pipeline stages (good for 20-40B models)
+- **PP=4**: 4 pipeline stages (good for 70B+ models)
+- **PP=8**: 8 pipeline stages (good for 175B+ models)
+
+**Benefits**:
+- Linear memory reduction (4× PP = 4× less memory)
+- Works across nodes (slower interconnect OK)
+
+**Drawbacks**:
+- Pipeline bubbles (idle time)
+- Requires micro-batching
+
+### Data Parallelism (DP)
+
+**Replicates model across GPU groups**:
+
+```python
+# 8 GPUs, TP=2, PP=2, DP=2
+# Group 0 (GPUs 0-3): Full model replica
+# Group 1 (GPUs 4-7): Full model replica
+```
+
+**DP degree**:
+- `DP = total_gpus / (TP × PP)`
+- Example: 8 GPUs, TP=2, PP=2 → DP=2
+
+**Benefits**:
+- Increases throughput
+- Scales batch size
+
+### Sequence Parallelism
+
+**Splits long sequences across GPUs** (extends TP):
+
+```python
+# 8K sequence, TP=2, Sequence Parallel=True
+# GPU 0: Tokens 0-4095
+# GPU 1: Tokens 4096-8191
+```
+
+**Benefits**:
+- Enables very long sequences (100K+ tokens)
+- Reduces activation memory
+
+**Requirements**:
+- Must use with TP > 1
+- RoPE/ALiBi position encodings work best
+
+## Accelerate Code Example
+
+### Basic Setup
+
+```python
+from accelerate import Accelerator
+from accelerate.utils import MegatronLMPlugin
+
+# Configure Megatron
+megatron_plugin = MegatronLMPlugin(
+ tp_degree=2, # Tensor parallelism degree
+ pp_degree=2, # Pipeline parallelism degree
+ num_micro_batches=4, # Micro-batches for pipeline
+ gradient_clipping=1.0, # Gradient clipping value
+ sequence_parallelism=False, # Enable sequence parallelism
+ recompute_activations=True, # Activation checkpointing
+ use_distributed_optimizer=True, # Distributed optimizer
+ custom_prepare_model_function=None, # Custom model prep
+)
+
+# Initialize accelerator
+accelerator = Accelerator(
+ mixed_precision='bf16',
+ megatron_lm_plugin=megatron_plugin
+)
+
+# Prepare model and optimizer
+model, optimizer, train_dataloader = accelerator.prepare(
+ model, optimizer, train_dataloader
+)
+
+# Training loop (same as DDP!)
+for batch in train_dataloader:
+ optimizer.zero_grad()
+ outputs = model(**batch)
+ loss = outputs.loss
+ accelerator.backward(loss)
+ optimizer.step()
+```
+
+### Full Training Script
+
+```python
+import torch
+from accelerate import Accelerator
+from accelerate.utils import MegatronLMPlugin
+from transformers import GPT2Config, GPT2LMHeadModel
+
+def main():
+ # Megatron configuration
+ megatron_plugin = MegatronLMPlugin(
+ tp_degree=2,
+ pp_degree=2,
+ num_micro_batches=4,
+ gradient_clipping=1.0,
+ )
+
+ accelerator = Accelerator(
+ mixed_precision='bf16',
+ gradient_accumulation_steps=8,
+ megatron_lm_plugin=megatron_plugin
+ )
+
+ # Model
+ config = GPT2Config(
+ n_layer=24,
+ n_head=16,
+ n_embd=1024,
+ )
+ model = GPT2LMHeadModel(config)
+
+ # Optimizer
+ optimizer = torch.optim.AdamW(model.parameters(), lr=6e-4)
+
+ # Prepare
+ model, optimizer, train_loader = accelerator.prepare(
+ model, optimizer, train_loader
+ )
+
+ # Training loop
+ for epoch in range(num_epochs):
+ for batch in train_loader:
+ with accelerator.accumulate(model):
+ outputs = model(**batch)
+ loss = outputs.loss
+ accelerator.backward(loss)
+ optimizer.step()
+ optimizer.zero_grad()
+
+ # Save checkpoint
+ accelerator.wait_for_everyone()
+ accelerator.save_state(f'checkpoint-epoch-{epoch}')
+
+if __name__ == '__main__':
+ main()
+```
+
+### Launch Command
+
+```bash
+# 8 GPUs, TP=2, PP=2, DP=2
+accelerate launch --multi_gpu --num_processes 8 train.py
+
+# Multi-node (2 nodes, 8 GPUs each)
+# Node 0
+accelerate launch --multi_gpu --num_processes 16 \
+ --num_machines 2 --machine_rank 0 \
+ --main_process_ip $MASTER_ADDR \
+ --main_process_port 29500 \
+ train.py
+
+# Node 1
+accelerate launch --multi_gpu --num_processes 16 \
+ --num_machines 2 --machine_rank 1 \
+ --main_process_ip $MASTER_ADDR \
+ --main_process_port 29500 \
+ train.py
+```
+
+## Activation Checkpointing
+
+**Reduces memory by recomputing activations**:
+
+```python
+megatron_plugin = MegatronLMPlugin(
+ recompute_activations=True, # Enable checkpointing
+ checkpoint_num_layers=1, # Checkpoint every N layers
+ distribute_checkpointed_activations=True, # Distribute across TP
+ partition_activations=True, # Partition in PP
+ check_for_nan_in_loss_and_grad=True, # Stability check
+)
+```
+
+**Strategies**:
+- `SELECTIVE`: Checkpoint transformer blocks only
+- `FULL`: Checkpoint all layers
+- `NONE`: No checkpointing
+
+**Memory savings**: 30-50% with 10-15% slowdown
+
+## Distributed Optimizer
+
+**Shards optimizer state across DP ranks**:
+
+```python
+megatron_plugin = MegatronLMPlugin(
+ use_distributed_optimizer=True, # Enable sharded optimizer
+)
+```
+
+**Benefits**:
+- Reduces optimizer memory by DP degree
+- Example: DP=4 → 4× less optimizer memory per GPU
+
+**Compatible with**:
+- AdamW, Adam, SGD
+- Mixed precision training
+
+## Performance Tuning
+
+### Micro-Batch Size
+
+```python
+# Pipeline parallelism requires micro-batching
+megatron_plugin = MegatronLMPlugin(
+ pp_degree=4,
+ num_micro_batches=16, # 16 micro-batches per pipeline
+)
+
+# Effective batch = num_micro_batches × micro_batch_size × DP
+# Example: 16 × 2 × 4 = 128
+```
+
+**Recommendations**:
+- More micro-batches → less pipeline bubble
+- Typical: 4-16 micro-batches
+
+### Sequence Length
+
+```python
+# For long sequences, enable sequence parallelism
+megatron_plugin = MegatronLMPlugin(
+ tp_degree=4,
+ sequence_parallelism=True, # Required: TP > 1
+)
+
+# Enables sequences up to TP × normal limit
+# Example: TP=4, 8K normal → 32K with sequence parallel
+```
+
+### GPU Topology
+
+**NVLink required for TP**:
+```bash
+# Check NVLink topology
+nvidia-smi topo -m
+
+# Good topology (NVLink between all GPUs)
+# GPU0 - GPU1: NV12 (fast)
+# GPU0 - GPU2: NV12 (fast)
+
+# Bad topology (PCIe only)
+# GPU0 - GPU4: PHB (slow, avoid TP across these)
+```
+
+**Recommendations**:
+- **TP**: Within same node (NVLink)
+- **PP**: Across nodes (slower interconnect OK)
+- **DP**: Any topology
+
+## Model Size Guidelines
+
+| Model Size | GPUs | TP | PP | DP | Micro-Batches |
+|------------|------|----|----|----|--------------|
+| 7B | 8 | 1 | 1 | 8 | 1 |
+| 13B | 8 | 2 | 1 | 4 | 1 |
+| 20B | 16 | 4 | 1 | 4 | 1 |
+| 40B | 32 | 4 | 2 | 4 | 4 |
+| 70B | 64 | 8 | 2 | 4 | 8 |
+| 175B | 128 | 8 | 4 | 4 | 16 |
+
+**Assumptions**: BF16, 2K sequence length, A100 80GB
+
+## Checkpointing
+
+### Save Checkpoint
+
+```python
+# Save full model state
+accelerator.save_state('checkpoint-1000')
+
+# Megatron saves separate files per rank
+# checkpoint-1000/
+# pytorch_model_tp_0_pp_0.bin
+# pytorch_model_tp_0_pp_1.bin
+# pytorch_model_tp_1_pp_0.bin
+# pytorch_model_tp_1_pp_1.bin
+# optimizer_tp_0_pp_0.bin
+# ...
+```
+
+### Load Checkpoint
+
+```python
+# Resume training
+accelerator.load_state('checkpoint-1000')
+
+# Automatically loads correct shard per rank
+```
+
+### Convert to Standard PyTorch
+
+```bash
+# Merge Megatron checkpoint to single file
+python merge_megatron_checkpoint.py \
+ --checkpoint-dir checkpoint-1000 \
+ --output pytorch_model.bin
+```
+
+## Common Issues
+
+### Issue: OOM with Pipeline Parallelism
+
+**Solution**: Increase micro-batches
+```python
+megatron_plugin = MegatronLMPlugin(
+ pp_degree=4,
+ num_micro_batches=16, # Increase from 4
+)
+```
+
+### Issue: Slow Training
+
+**Check 1**: Pipeline bubbles (PP too high)
+```python
+# Reduce PP, increase TP
+tp_degree=4 # Increase
+pp_degree=2 # Decrease
+```
+
+**Check 2**: Micro-batch size too small
+```python
+num_micro_batches=8 # Increase
+```
+
+### Issue: NVLink Not Detected
+
+```bash
+# Verify NVLink
+nvidia-smi nvlink -s
+
+# If no NVLink, avoid TP > 1
+# Use PP or DP instead
+```
+
+## Resources
+
+- Megatron-LM: https://github.com/NVIDIA/Megatron-LM
+- Accelerate Megatron docs: https://huggingface.co/docs/accelerate/usage_guides/megatron_lm
+- Paper: "Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism"
+- NVIDIA Apex: https://github.com/NVIDIA/apex
diff --git a/src/skills/bundled/accelerate/references/performance.md b/src/skills/bundled/accelerate/references/performance.md
new file mode 100644
index 0000000..62560d2
--- /dev/null
+++ b/src/skills/bundled/accelerate/references/performance.md
@@ -0,0 +1,525 @@
+# Accelerate Performance Tuning
+
+## Profiling
+
+### Basic Profiling
+
+```python
+from accelerate import Accelerator
+import time
+
+accelerator = Accelerator()
+
+# Warmup
+for _ in range(10):
+ batch = next(iter(dataloader))
+ outputs = model(**batch)
+ loss = outputs.loss
+ accelerator.backward(loss)
+ optimizer.step()
+ optimizer.zero_grad()
+
+# Profile training loop
+start = time.time()
+total_batches = 100
+
+for i, batch in enumerate(dataloader):
+ if i >= total_batches:
+ break
+
+ outputs = model(**batch)
+ loss = outputs.loss
+ accelerator.backward(loss)
+ optimizer.step()
+ optimizer.zero_grad()
+
+accelerator.wait_for_everyone() # Sync all processes
+elapsed = time.time() - start
+
+# Metrics
+batches_per_sec = total_batches / elapsed
+samples_per_sec = (total_batches * batch_size * accelerator.num_processes) / elapsed
+
+print(f"Throughput: {samples_per_sec:.2f} samples/sec")
+print(f"Batches/sec: {batches_per_sec:.2f}")
+```
+
+### PyTorch Profiler Integration
+
+```python
+from torch.profiler import profile, ProfilerActivity
+
+with profile(
+ activities=[ProfilerActivity.CPU, ProfilerActivity.CUDA],
+ record_shapes=True,
+ profile_memory=True,
+ with_stack=True
+) as prof:
+ for i, batch in enumerate(dataloader):
+ if i >= 10: # Profile first 10 batches
+ break
+
+ outputs = model(**batch)
+ loss = outputs.loss
+ accelerator.backward(loss)
+ optimizer.step()
+ optimizer.zero_grad()
+
+# Print profiling results
+print(prof.key_averages().table(
+ sort_by="cuda_time_total", row_limit=20
+))
+
+# Export to Chrome tracing
+prof.export_chrome_trace("trace.json")
+# View at chrome://tracing
+```
+
+## Memory Optimization
+
+### 1. Gradient Accumulation
+
+**Problem**: Large batch size causes OOM
+
+**Solution**: Accumulate gradients across micro-batches
+
+```python
+accelerator = Accelerator(gradient_accumulation_steps=8)
+
+# Effective batch = batch_size × accumulation_steps × num_gpus
+# Example: 4 × 8 × 8 = 256
+
+for batch in dataloader:
+ with accelerator.accumulate(model): # Handles accumulation logic
+ outputs = model(**batch)
+ loss = outputs.loss
+ accelerator.backward(loss)
+ optimizer.step()
+ optimizer.zero_grad()
+```
+
+**Memory savings**: 8× less activation memory (with 8 accumulation steps)
+
+### 2. Gradient Checkpointing
+
+**Enable in model**:
+
+```python
+from transformers import AutoModelForCausalLM
+
+model = AutoModelForCausalLM.from_pretrained(
+ "gpt2",
+ use_cache=False # Required for gradient checkpointing
+)
+
+# Enable checkpointing
+model.gradient_checkpointing_enable()
+
+# Prepare with Accelerate
+model = accelerator.prepare(model)
+```
+
+**Memory savings**: 30-50% with 10-15% slowdown
+
+### 3. Mixed Precision
+
+**BF16 (A100/H100)**:
+```python
+accelerator = Accelerator(mixed_precision='bf16')
+
+# Automatic mixed precision
+for batch in dataloader:
+ outputs = model(**batch) # Forward in BF16
+ loss = outputs.loss
+ accelerator.backward(loss) # Backward in FP32
+ optimizer.step()
+```
+
+**FP16 (V100, older GPUs)**:
+```python
+from accelerate.utils import GradScalerKwargs
+
+scaler_kwargs = GradScalerKwargs(
+ init_scale=2.**16,
+ growth_interval=2000
+)
+
+accelerator = Accelerator(
+ mixed_precision='fp16',
+ kwargs_handlers=[scaler_kwargs]
+)
+```
+
+**Memory savings**: 50% compared to FP32
+
+### 4. CPU Offloading (DeepSpeed)
+
+```python
+from accelerate.utils import DeepSpeedPlugin
+
+ds_plugin = DeepSpeedPlugin(
+ zero_stage=3,
+ offload_optimizer_device="cpu", # Offload optimizer to CPU
+ offload_param_device="cpu", # Offload parameters to CPU
+)
+
+accelerator = Accelerator(
+ deepspeed_plugin=ds_plugin,
+ mixed_precision='bf16'
+)
+```
+
+**Memory savings**: 10-20× for optimizer state, 5-10× for parameters
+
+**Trade-off**: 20-30% slower due to CPU-GPU transfers
+
+### 5. Flash Attention
+
+```python
+# Install flash-attn
+# pip install flash-attn
+
+from transformers import AutoModelForCausalLM
+
+model = AutoModelForCausalLM.from_pretrained(
+ "gpt2",
+ attn_implementation="flash_attention_2" # Enable Flash Attention 2
+)
+
+model = accelerator.prepare(model)
+```
+
+**Memory savings**: 50% for attention, 2× faster
+
+**Requirements**: A100/H100, sequence length must be multiple of 128
+
+## Communication Optimization
+
+### 1. Gradient Bucketing (DDP)
+
+```python
+from accelerate.utils import DistributedDataParallelKwargs
+
+ddp_kwargs = DistributedDataParallelKwargs(
+ bucket_cap_mb=25, # Bucket size for gradient reduction
+ gradient_as_bucket_view=True, # Reduce memory copies
+ static_graph=False # Set True if model doesn't change
+)
+
+accelerator = Accelerator(kwargs_handlers=[ddp_kwargs])
+```
+
+**Recommended bucket sizes**:
+- Small models (<1B): 25 MB
+- Medium models (1-10B): 50-100 MB
+- Large models (>10B): 100-200 MB
+
+### 2. Find Unused Parameters
+
+```python
+# Only enable if model has unused parameters (slower!)
+ddp_kwargs = DistributedDataParallelKwargs(
+ find_unused_parameters=True
+)
+```
+
+**Use case**: Models with conditional branches (e.g., mixture of experts)
+
+**Cost**: 10-20% slower
+
+### 3. NCCL Tuning
+
+```bash
+# Set environment variables before launch
+export NCCL_DEBUG=INFO # Debug info
+export NCCL_IB_DISABLE=0 # Enable InfiniBand
+export NCCL_SOCKET_IFNAME=eth0 # Network interface
+export NCCL_P2P_LEVEL=NVL # Use NVLink
+
+accelerate launch train.py
+```
+
+**NCCL_P2P_LEVEL options**:
+- `NVL`: NVLink (fastest, within node)
+- `PIX`: PCIe (fast, within node)
+- `PHB`: PCIe host bridge (slow, cross-node)
+
+## Data Loading Optimization
+
+### 1. DataLoader Workers
+
+```python
+from torch.utils.data import DataLoader
+
+train_loader = DataLoader(
+ dataset,
+ batch_size=32,
+ num_workers=4, # Parallel data loading
+ pin_memory=True, # Pin memory for faster GPU transfer
+ prefetch_factor=2, # Prefetch batches per worker
+ persistent_workers=True # Keep workers alive between epochs
+)
+
+train_loader = accelerator.prepare(train_loader)
+```
+
+**Recommendations**:
+- `num_workers`: 2-4 per GPU (8 GPUs → 16-32 workers)
+- `pin_memory`: Always True for GPU training
+- `prefetch_factor`: 2-4 (higher for slow data loading)
+
+### 2. Data Preprocessing
+
+```python
+from datasets import load_dataset
+
+# Bad: Preprocess during training (slow)
+dataset = load_dataset("openwebtext")
+
+for batch in dataset:
+ tokens = tokenizer(batch['text']) # Slow!
+ ...
+
+# Good: Preprocess once, save
+dataset = load_dataset("openwebtext")
+tokenized = dataset.map(
+ lambda x: tokenizer(x['text']),
+ batched=True,
+ num_proc=8, # Parallel preprocessing
+ remove_columns=['text']
+)
+tokenized.save_to_disk("preprocessed_data")
+
+# Load preprocessed
+dataset = load_from_disk("preprocessed_data")
+```
+
+### 3. Faster Tokenization
+
+```python
+import os
+
+# Enable Rust-based tokenizers (10× faster)
+os.environ["TOKENIZERS_PARALLELISM"] = "true"
+
+from transformers import AutoTokenizer
+
+tokenizer = AutoTokenizer.from_pretrained(
+ "gpt2",
+ use_fast=True # Use fast Rust tokenizer
+)
+```
+
+## Compilation (PyTorch 2.0+)
+
+### Compile Model
+
+```python
+import torch
+
+# Compile model for faster execution
+model = torch.compile(
+ model,
+ mode="reduce-overhead", # Options: default, reduce-overhead, max-autotune
+ fullgraph=False, # Compile entire graph (stricter)
+ dynamic=True # Support dynamic shapes
+)
+
+model = accelerator.prepare(model)
+```
+
+**Speedup**: 10-50% depending on model
+
+**Compilation modes**:
+- `default`: Balanced (best for most cases)
+- `reduce-overhead`: Min overhead (best for small batches)
+- `max-autotune`: Max performance (slow compile, best for production)
+
+### Compilation Best Practices
+
+```python
+# Bad: Compile after prepare (won't work)
+model = accelerator.prepare(model)
+model = torch.compile(model) # Error!
+
+# Good: Compile before prepare
+model = torch.compile(model)
+model = accelerator.prepare(model)
+
+# Training loop
+for batch in dataloader:
+ # First iteration: slow (compilation)
+ # Subsequent iterations: fast (compiled)
+ outputs = model(**batch)
+ ...
+```
+
+## Benchmarking Different Strategies
+
+### Script Template
+
+```python
+import time
+import torch
+from accelerate import Accelerator
+
+def benchmark_strategy(strategy_name, accelerator_kwargs):
+ """Benchmark a specific training strategy."""
+ accelerator = Accelerator(**accelerator_kwargs)
+
+ # Setup
+ model = create_model()
+ optimizer = torch.optim.AdamW(model.parameters(), lr=1e-4)
+ dataloader = create_dataloader()
+
+ model, optimizer, dataloader = accelerator.prepare(
+ model, optimizer, dataloader
+ )
+
+ # Warmup
+ for i, batch in enumerate(dataloader):
+ if i >= 10:
+ break
+ outputs = model(**batch)
+ loss = outputs.loss
+ accelerator.backward(loss)
+ optimizer.step()
+ optimizer.zero_grad()
+
+ # Benchmark
+ accelerator.wait_for_everyone()
+ torch.cuda.synchronize()
+ start = time.time()
+
+ num_batches = 100
+ for i, batch in enumerate(dataloader):
+ if i >= num_batches:
+ break
+
+ outputs = model(**batch)
+ loss = outputs.loss
+ accelerator.backward(loss)
+ optimizer.step()
+ optimizer.zero_grad()
+
+ accelerator.wait_for_everyone()
+ torch.cuda.synchronize()
+ elapsed = time.time() - start
+
+ # Metrics
+ throughput = (num_batches * batch_size * accelerator.num_processes) / elapsed
+ memory_used = torch.cuda.max_memory_allocated() / 1e9 # GB
+
+ if accelerator.is_main_process:
+ print(f"\n{strategy_name}:")
+ print(f" Throughput: {throughput:.2f} samples/sec")
+ print(f" Memory: {memory_used:.2f} GB")
+ print(f" Time: {elapsed:.2f} sec")
+
+ torch.cuda.reset_peak_memory_stats()
+
+# Benchmark different strategies
+strategies = [
+ ("DDP + FP32", {}),
+ ("DDP + BF16", {"mixed_precision": "bf16"}),
+ ("DDP + BF16 + GradAccum", {"mixed_precision": "bf16", "gradient_accumulation_steps": 4}),
+ ("FSDP", {"fsdp_plugin": fsdp_plugin}),
+ ("DeepSpeed ZeRO-2", {"deepspeed_plugin": ds_plugin_stage2}),
+ ("DeepSpeed ZeRO-3", {"deepspeed_plugin": ds_plugin_stage3}),
+]
+
+for name, kwargs in strategies:
+ benchmark_strategy(name, kwargs)
+```
+
+## Performance Checklist
+
+**Before training**:
+- [ ] Use BF16/FP16 mixed precision
+- [ ] Enable gradient checkpointing (if OOM)
+- [ ] Set appropriate `num_workers` (2-4 per GPU)
+- [ ] Enable `pin_memory=True`
+- [ ] Preprocess data once, not during training
+- [ ] Compile model with `torch.compile` (PyTorch 2.0+)
+
+**For large models**:
+- [ ] Use FSDP or DeepSpeed ZeRO-3
+- [ ] Enable CPU offloading (if still OOM)
+- [ ] Use Flash Attention
+- [ ] Increase gradient accumulation
+
+**For multi-node**:
+- [ ] Check network topology (InfiniBand > Ethernet)
+- [ ] Tune NCCL settings
+- [ ] Use larger bucket sizes for DDP
+- [ ] Verify NVLink for tensor parallelism
+
+**Profiling**:
+- [ ] Profile first 10-100 batches
+- [ ] Check GPU utilization (`nvidia-smi dmon`)
+- [ ] Check data loading time (should be <5% of iteration)
+- [ ] Identify communication bottlenecks
+
+## Common Performance Issues
+
+### Issue: Low GPU Utilization (<80%)
+
+**Cause 1**: Data loading bottleneck
+```python
+# Solution: Increase workers and prefetch
+num_workers=8
+prefetch_factor=4
+```
+
+**Cause 2**: Small batch size
+```python
+# Solution: Increase batch size or use gradient accumulation
+batch_size=32 # Increase
+gradient_accumulation_steps=4 # Or accumulate
+```
+
+### Issue: High Memory Usage
+
+**Solution 1**: Gradient checkpointing
+```python
+model.gradient_checkpointing_enable()
+```
+
+**Solution 2**: Reduce batch size, increase accumulation
+```python
+batch_size=8 # Reduce from 32
+gradient_accumulation_steps=16 # Maintain effective batch
+```
+
+**Solution 3**: Use FSDP or DeepSpeed ZeRO-3
+```python
+accelerator = Accelerator(fsdp_plugin=fsdp_plugin)
+```
+
+### Issue: Slow Multi-GPU Training
+
+**Cause**: Communication bottleneck
+
+**Check 1**: Gradient bucket size
+```python
+ddp_kwargs = DistributedDataParallelKwargs(bucket_cap_mb=100)
+```
+
+**Check 2**: NCCL settings
+```bash
+export NCCL_DEBUG=INFO
+# Check for "Using NVLS" (good) vs "Using PHB" (bad)
+```
+
+**Check 3**: Network bandwidth
+```bash
+# Test inter-GPU bandwidth
+nvidia-smi nvlink -s
+```
+
+## Resources
+
+- Accelerate Performance: https://huggingface.co/docs/accelerate/usage_guides/performance
+- PyTorch Profiler: https://pytorch.org/tutorials/recipes/recipes/profiler_recipe.html
+- NCCL Tuning: https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/env.html
+- Flash Attention: https://github.com/Dao-AILab/flash-attention
diff --git a/src/skills/bundled/actual-setup/SKILL.md b/src/skills/bundled/actual-setup/SKILL.md
new file mode 100644
index 0000000..42521ca
--- /dev/null
+++ b/src/skills/bundled/actual-setup/SKILL.md
@@ -0,0 +1,147 @@
+---
+name: actual-setup
+description: Set up Actual Computer (actual.inc) inference in Hermes.
+version: 2.0.0
+author: shl0ms + Hermes Agent
+license: MIT
+platforms: [linux, macos, windows]
+metadata:
+ hermes:
+ tags: [actual, actual-inc, provider, local-inference, relay, gguf, setup]
+ category: devops
+---
+
+# Actual Computer Setup Skill
+
+Sets up [actual.inc](https://actual.inc) (Actual Computer) as a Hermes inference
+provider. Actual turns the user's own hardware into a private inference cluster
+and exposes an OpenAI-compatible API two ways: a hosted end-to-end-encrypted
+relay at `https://api.actual.inc` (authenticated with an `ac_` key), and a local
+on-device daemon at `http://127.0.0.1:8080` (no auth on loopback). This skill
+does not install the Actual daemon for the user — device authorization requires
+a human in a browser.
+
+## When to Use
+
+- User wants to add actual.inc as an inference provider (cloud relay or local).
+- User has an `ac_` key and wants Hermes routed through their Actual cluster.
+- User wants fully-local, on-device inference via the Actual daemon.
+- Troubleshooting: Actual requests failing with cryptic 400s or empty streams.
+
+## Prerequisites
+
+- Hermes has **first-class `actual` provider support** (provider id `actual`,
+ aliases `actual-computer`, `actualcomputer`, `aci`). Do NOT configure Actual
+ as a `custom_providers` / `providers.actual.*` entry on current Hermes — the
+ built-in provider owns the name and handles base-url normalization, the
+ Responses transport, and local no-auth automatically.
+- Relay mode: an Actual account and an `ac_` inference key from
+ https://actual.inc/user/keys.
+- Local mode: the user has installed the daemon
+ (`curl -fsSL "https://actual.inc/install" | bash`) and completed device
+ authorization by running `actual` once and opening the printed
+ `https://actual.inc/device?code=...` URL in a browser. Relay that URL to the
+ user and WAIT — never invent an email or authorize on their behalf. Codes
+ expire in 5 minutes; re-run `actual` for a fresh one.
+
+## How to Run
+
+### Relay / API mode
+
+1. Put the key in `.env` (secrets only — never config.yaml):
+ append `ACTUAL_API_KEY=ac_...` to `~/.hermes/.env`.
+2. Verify the key and discover models with `terminal`:
+ ```bash
+ curl -s https://api.actual.inc/v1/models -H "Authorization: Bearer $ACTUAL_API_KEY"
+ ```
+3. Select provider + model:
+ ```bash
+ hermes config set model.provider actual
+ hermes config set model.default "MODEL_ID_FROM_DISCOVERY"
+ ```
+4. Verify end-to-end:
+ ```bash
+ hermes chat -Q -q "Reply with exactly: ACTUAL_OK" --provider actual -m MODEL_ID
+ ```
+
+### Local mode
+
+1. Human has installed + authorized the daemon (see Prerequisites).
+2. Download and load a model (scriptable once authorized):
+ ```bash
+ actual models search "qwen2.5 0.5b instruct gguf" --limit 8 --no-prompt
+ # Downloads REQUIRE an explicit quantization (409 ambiguous_model_download otherwise):
+ actual models download "Qwen/Qwen2.5-0.5B-Instruct-GGUF/Q4_K_M"
+ actual models list # note the INSTALLED name (differs from download id)
+ actual models load "qwen2.5-0.5b-instruct-q4_k_m" # load by installed name
+ ```
+3. Point Hermes at the daemon. `ACTUAL_BASE_URL` with a loopback host flips the
+ built-in provider into local no-auth mode automatically — no key needed:
+ append `ACTUAL_BASE_URL=http://127.0.0.1:8080` to `~/.hermes/.env`, then:
+ ```bash
+ hermes config set model.provider actual
+ hermes config set model.default "INSTALLED_MODEL_NAME"
+ ```
+4. Verify (reduced toolset — see context-window pitfall below):
+ ```bash
+ hermes chat -Q -q "Reply with exactly: LOCAL_OK" --provider actual -m INSTALLED_NAME -t file,web
+ ```
+
+## Quick Reference
+
+| Thing | Value |
+|---|---|
+| Hosted relay | `https://api.actual.inc/v1` (normalized from bare host automatically) |
+| Local daemon | `http://127.0.0.1:8080/v1` (no auth on loopback) |
+| Key env var | `ACTUAL_API_KEY` (`ac_...`) |
+| Base URL env var | `ACTUAL_BASE_URL` (loopback host ⇒ local no-auth mode) |
+| Provider id / aliases | `actual` / `actual-computer`, `actualcomputer`, `aci` |
+| Transport | Responses API (`codex_responses`) — built-in, do not override |
+| Cluster pinning | `X-Cluster-ID` header via `providers.actual.extra_headers` in config.yaml |
+| Model size guide | 0.5B Q4_K_M ~470MB (toy), 7-8B Q4_K_M ~4.5GB (daily driver), 32B ~20GB |
+
+## Pitfalls
+
+1. **reasoning_effort trap (handled by Hermes since the first-class provider).**
+ Actual's SGLang/vLLM backends accept only `none/low/medium/high/max`;
+ `xhigh`/`ultra` used to fail with a cryptic
+ `Expecting value: line 1 column 1 (char 0)` (a wrapped HTTP 400). The
+ built-in provider clamps `xhigh→high` and `ultra→max` on the wire. If a
+ request still 400s this way on an old Hermes, set a per-model cap:
+ `agent.reasoning_overrides.: high` in config.yaml.
+2. **Context-window overflow on small local models.** Hermes' default toolset
+ is ~26k tokens of schemas plus a ~9k-token system prompt. A model loaded
+ with a 32k context overflows before the first turn, and llama.cpp-family
+ servers emit a bare `data: [DONE]` — Hermes reports
+ `Provider returned an empty stream with no finish_reason`. This is NOT an
+ SSE bug. Fixes: restrict tools (`-t file,web`), load the model with a
+ larger `n_ctx`, or pick a >=64k-context model for the full toolset.
+ Upstream tracking: #51448 (do not file new issues; add evidence there).
+ Related but distinct: #65631 (HTTP-200 SSE carrying a 400), #56516
+ (reasoning-only streams).
+3. **Download ids vs installed names.** `actual models download` takes
+ `repo/QUANT` and 409s without an explicit quantization;
+ `actual models load` takes the INSTALLED name from `actual models list`.
+4. **Reasoning models returning empty content.** GLM/Qwen reasoning variants
+ emit thinking in a separate `reasoning` field and can burn a small
+ `max_tokens` entirely on reasoning. Give generous max_tokens before
+ assuming failure.
+5. **Do not create a custom provider named `actual`.** Older setup guides
+ (pre first-class support) wrote `providers.actual.*` config blocks. On
+ current Hermes the built-in provider wins the name; stale custom blocks
+ are ignored or conflict. Remove them and use the env vars + model.provider
+ flow above.
+
+## Verification
+
+```bash
+# Relay:
+hermes chat -Q -q "Reply with exactly: ACTUAL_OK" --provider actual -m MODEL
+# Local (small model — reduced toolset):
+hermes chat -Q -q "Reply with exactly: LOCAL_OK" --provider actual -m MODEL -t file,web
+# Provider status (local no-auth shows key_source=local-offline):
+hermes status
+```
+
+For other OpenAI-compatible clients (e.g. OpenCode), see
+`references/opencode.md`.
diff --git a/src/skills/bundled/actual-setup/references/opencode.md b/src/skills/bundled/actual-setup/references/opencode.md
new file mode 100644
index 0000000..268c076
--- /dev/null
+++ b/src/skills/bundled/actual-setup/references/opencode.md
@@ -0,0 +1,91 @@
+# actual.inc as an OpenCode provider
+
+Verified end-to-end 2026-07 (OpenCode 1.18.3, macOS). Adds Actual's relay/GLM
+cluster to OpenCode as a custom OpenAI-compatible provider.
+
+## Design: secret in auth.json, config in opencode.json
+
+OpenCode auto-injects a credential when the provider **id** in `opencode.json`
+matches a credential **id** in `~/.local/share/opencode/auth.json`. So put the
+key in auth.json and NOTHING sensitive goes in opencode.json. This is more robust
+than `options.apiKey: "{env:...}"` with the var name, because `{env:...}` only resolves
+if the var is exported in the shell OpenCode launches from — and the Actual key
+is typically only in `~/.hermes/.env`, not a shell profile, so the env form
+breaks outside an inheriting terminal.
+
+### 1. Add the credential to auth.json
+
+File: `~/.local/share/opencode/auth.json`. Shape (preserve existing entries):
+```json
+{
+ "anthropic": { "type": "api", "key": "..." },
+ "actual": { "type": "api", "key": "ac_..." }
+}
+```
+Do this with a read-modify-write (json load, add the `actual` key, dump) so the
+other credentials stay intact — don't overwrite the file.
+
+### 2. Add the provider to opencode.json
+
+File: `~/.config/opencode/opencode.json` (or `~/.opencode.json`). Add under
+`provider` alongside anything already there. NO `apiKey` field — it comes from
+auth.json by id match.
+```json
+{
+ "$schema": "https://opencode.ai/config.json",
+ "provider": {
+ "actual": {
+ "npm": "@ai-sdk/openai-compatible",
+ "name": "Actual (GLM cluster)",
+ "options": {
+ "baseURL": "https://api.actual.inc/v1",
+ "headers": {
+ "X-Cluster-ID": ""
+ }
+ },
+ "models": {
+ "glm-5.2-nvfp4": {
+ "name": "GLM-5.2 (b300x8)",
+ "limit": { "context": 1048576, "output": 65536 }
+ }
+ }
+ }
+ }
+}
+```
+- `npm`: `@ai-sdk/openai-compatible` for `/v1/chat/completions`. Use
+ `@ai-sdk/openai` only if the model needs `/v1/responses`.
+- `options.headers.X-Cluster-ID`: pin to a specific cluster (optional; omit to
+ let the relay route). Get the hash from the Actual console URL
+ (`console/computers?cluster=`).
+- `models.`: the id MUST match what `GET /v1/models` returns. Discover it
+ first: `curl -s https://api.actual.inc/v1/models -H "Authorization: Bearer ac_..." -H "X-Cluster-ID: "`.
+- `limit`: lets OpenCode track remaining context (custom providers don't get
+ this from models.dev). GLM-5.2 context = 1_048_576.
+
+### 3. Verify live (headless)
+
+```bash
+opencode run -m actual/glm-5.2-nvfp4 "Reply with exactly this text: OPENCODE_ACTUAL_OK"
+```
+OpenCode DOES use the `provider/model` slash form on the CLI (unlike Hermes,
+where the slash form 404s custom providers). Expect the exact reply. Run a second
+reasoning check (e.g. "What is 17 * 23?") since GLM-5.2 is a reasoning model.
+
+## Why no reasoning_effort trap here
+
+The Actual relay rejects `reasoning_effort: xhigh` with an HTTP 400 (see the
+`hermes-custom-providers` skill, pitfall 2). Hermes hits this because it forwards
+its global `agent.reasoning_effort`. OpenCode's ai-sdk does NOT send that param,
+so Actual + OpenCode works with zero reasoning config. No `reasoning_overrides`
+equivalent needed.
+
+## Gotchas
+
+- `auth.json` is the same store `/connect` writes; editing it directly is fine
+ and equivalent. `opencode auth list` should then show `actual` under
+ Credentials.
+- If discovery/models don't appear, confirm the provider id in opencode.json
+ EXACTLY matches the auth.json credential id (`actual` == `actual`).
+- No git-tracking risk on the default config dir (`~/.config/opencode` is not a
+ repo), but still keep the key in auth.json, not opencode.json, as the habit.
diff --git a/src/skills/bundled/adversarial-ux-test/SKILL.md b/src/skills/bundled/adversarial-ux-test/SKILL.md
new file mode 100644
index 0000000..2c5ba20
--- /dev/null
+++ b/src/skills/bundled/adversarial-ux-test/SKILL.md
@@ -0,0 +1,191 @@
+---
+name: adversarial-ux-test
+description: Roleplay a hostile user to find and triage UX pain points.
+version: 1.0.0
+author: Omni @ Comelse
+license: MIT
+platforms: [linux, macos, windows]
+metadata:
+ hermes:
+ tags: [qa, ux, testing, adversarial, dogfood, personas, user-testing]
+ related_skills: [dogfood]
+---
+
+# Adversarial UX Test
+
+Roleplay the worst-case user for your product — the person who hates technology, doesn't want your software, and will find every reason to complain. Then filter their feedback through a pragmatism layer to separate real UX problems from "I hate computers" noise.
+
+Think of it as an automated "mom test" — but angry.
+
+## Why This Works
+
+Most QA finds bugs. This finds **friction**. A technically correct app can still be unusable for real humans. The adversarial persona catches:
+- Confusing terminology that makes sense to developers but not users
+- Too many steps to accomplish basic tasks
+- Missing onboarding or "aha moments"
+- Accessibility issues (font size, contrast, click targets)
+- Cold-start problems (empty states, no demo content)
+- Paywall/signup friction that kills conversion
+
+The **pragmatism filter** (Phase 3) is what makes this useful instead of just entertaining. Without it, you'd add a "print this page" button to every screen because Grandpa can't figure out PDFs.
+
+## How to Use
+
+Tell the agent:
+```
+"Run an adversarial UX test on [URL]"
+"Be a grumpy [persona type] and test [app name]"
+"Do an asshole user test on my staging site"
+```
+
+You can provide a persona or let the agent generate one based on your product's target audience.
+
+## Step 1: Define the Persona
+
+If no persona is provided, generate one by answering:
+
+1. **Who is the HARDEST user for this product?** (age 50+, non-technical role, decades of experience doing it "the old way")
+2. **What is their tech comfort level?** (the lower the better — WhatsApp-only, paper notebooks, wife set up their email)
+3. **What is the ONE thing they need to accomplish?** (their core job, not your feature list)
+4. **What would make them give up?** (too many clicks, jargon, slow, confusing)
+5. **How do they talk when frustrated?** (blunt, sweary, dismissive, sighing)
+
+### Good Persona Example
+> **"Big Mick" McAllister** — 58-year-old S&C coach. Uses WhatsApp and that's it. His "spreadsheet" is a paper notebook. "If I can't figure it out in 10 seconds I'm going back to my notebook." Needs to log session results for 25 players. Hates small text, jargon, and passwords.
+
+### Bad Persona Example
+> "A user who doesn't like the app" — too vague, no constraints, no voice.
+
+The persona must be **specific enough to stay in character** for 20 minutes of testing.
+
+## Step 2: Become the Asshole (Browse as the Persona)
+
+1. Read any available project docs for app context and URLs
+2. **Fully inhabit the persona** — their frustrations, limitations, goals
+3. Navigate to the app using browser tools
+4. **Attempt the persona's ACTUAL TASKS** (not a feature tour):
+ - Can they do what they came to do?
+ - How many clicks/screens to accomplish it?
+ - What confuses them?
+ - What makes them angry?
+ - Where do they get lost?
+ - What would make them give up and go back to their old way?
+
+5. Test these friction categories:
+ - **First impression** — would they even bother past the landing page?
+ - **Core workflow** — the ONE thing they need to do most often
+ - **Error recovery** — what happens when they do something wrong?
+ - **Readability** — text size, contrast, information density
+ - **Speed** — does it feel faster than their current method?
+ - **Terminology** — any jargon they wouldn't understand?
+ - **Navigation** — can they find their way back? do they know where they are?
+
+6. Take screenshots of every pain point
+7. Check browser console for JS errors on every page
+
+## Step 3: The Rant (Write Feedback in Character)
+
+Write the feedback AS THE PERSONA — in their voice, with their frustrations. This is not a bug report. This is a real human venting.
+
+```
+[PERSONA NAME]'s Review of [PRODUCT]
+
+Overall: [Would they keep using it? Yes/No/Maybe with conditions]
+
+THE GOOD (grudging admission):
+- [things even they have to admit work]
+
+THE BAD (legitimate UX issues):
+- [real problems that would stop them from using the product]
+
+THE UGLY (showstoppers):
+- [things that would make them uninstall/cancel immediately]
+
+SPECIFIC COMPLAINTS:
+1. [Page/feature]: "[quote in persona voice]" — [what happened, expected]
+2. ...
+
+VERDICT: "[one-line persona quote summarizing their experience]"
+```
+
+## Step 4: The Pragmatism Filter (Critical — Do Not Skip)
+
+Step OUT of the persona. Evaluate each complaint as a product person:
+
+- **RED: REAL UX BUG** — Any user would have this problem, not just grumpy ones. Fix it.
+- **YELLOW: VALID BUT LOW PRIORITY** — Real issue but only for extreme users. Note it.
+- **WHITE: PERSONA NOISE** — "I hate computers" talking, not a product problem. Skip it.
+- **GREEN: FEATURE REQUEST** — Good idea hidden in the complaint. Consider it.
+
+### Filter Criteria
+1. Would a 35-year-old competent-but-busy user have the same complaint? → RED
+2. Is this a genuine accessibility issue (font size, contrast, click targets)? → RED
+3. Is this "I want it to work like paper" resistance to digital? → WHITE
+4. Is this a real workflow inefficiency the persona stumbled on? → YELLOW or RED
+5. Would fixing this add complexity for the 80% who are fine? → WHITE
+6. Does the complaint reveal a missing onboarding moment? → GREEN
+
+**This filter is MANDATORY.** Never ship raw persona complaints as tickets.
+
+## Step 5: Create Tickets
+
+For **RED** and **GREEN** items only:
+- Clear, actionable title
+- Include the persona's verbatim quote (entertaining + memorable)
+- The real UX issue underneath (objective)
+- A suggested fix (actionable)
+- Tag/label: "ux-review"
+
+For **YELLOW** items: one catch-all ticket with all notes.
+
+**WHITE** items appear in the report only. No tickets.
+
+**Max 10 tickets per session** — focus on the worst issues.
+
+## Step 6: Report
+
+Deliver:
+1. The persona rant (Step 3) — entertaining and visceral
+2. The filtered assessment (Step 4) — pragmatic and actionable
+3. Tickets created (Step 5) — with links
+4. Screenshots of key issues
+
+## Tips
+
+- **One persona per session.** Don't mix perspectives.
+- **Stay in character during Steps 2-3.** Break character only at Step 4.
+- **Test the CORE WORKFLOW first.** Don't get distracted by settings pages.
+- **Empty states are gold.** New user experience reveals the most friction.
+- **The best findings are RED items the persona found accidentally** while trying to do something else.
+- **If the persona has zero complaints, your persona is too tech-savvy.** Make them older, less patient, more set in their ways.
+- **Run this before demos, launches, or after shipping a batch of features.**
+- **Register as a NEW user when possible.** Don't use pre-seeded admin accounts — the cold start experience is where most friction lives.
+- **Zero WHITE items is a signal, not a failure.** If the pragmatism filter finds no noise, your product has real UX problems, not just a grumpy persona.
+- **Check known issues in project docs AFTER the test.** If the persona found a bug that's already in the known issues list, that's actually the most damning finding — it means the team knew about it but never felt the user's pain.
+- **Subscription/paywall testing is critical.** Test with expired accounts, not just active ones. The "what happens when you can't pay" experience reveals whether the product respects users or holds their data hostage.
+- **Count the clicks to accomplish the persona's ONE task.** If it's more than 5, that's almost always a RED finding regardless of persona tech level.
+
+## Example Personas by Industry
+
+These are starting points — customize for your specific product:
+
+| Product Type | Persona | Age | Key Trait |
+|-------------|---------|-----|-----------|
+| CRM | Retirement home director | 68 | Filing cabinet is the current CRM |
+| Photography SaaS | Rural wedding photographer | 62 | Books clients by phone, invoices on paper |
+| AI/ML Tool | Department store buyer | 55 | Burned by 3 failed tech startups |
+| Fitness App | Old-school gym coach | 58 | Paper notebook, thick fingers, bad eyes |
+| Accounting | Family bakery owner | 64 | Shoebox of receipts, hates subscriptions |
+| E-commerce | Market stall vendor | 60 | Cash only, smartphone is for calls |
+| Healthcare | Senior GP | 63 | Dictates notes, nurse handles the computer |
+| Education | Veteran teacher | 57 | Chalk and talk, worksheets in ring binders |
+
+## Rules
+
+- Stay in character during Steps 2-3
+- Be genuinely mean but fair — find real problems, not manufactured ones
+- The pragmatism filter (Step 4) is **MANDATORY**
+- Screenshots required for every complaint
+- Max 10 tickets per session
+- Test on staging/deployed app, not local dev
+- One persona, one session, one report
diff --git a/src/skills/bundled/agentmail/SKILL.md b/src/skills/bundled/agentmail/SKILL.md
new file mode 100644
index 0000000..770ca8d
--- /dev/null
+++ b/src/skills/bundled/agentmail/SKILL.md
@@ -0,0 +1,128 @@
+---
+name: agentmail
+description: "Give the agent its own inbox: send and receive email."
+version: 1.0.0
+author: teyrebaz33, Hermes Agent
+license: MIT
+platforms: [linux, macos, windows]
+metadata:
+ hermes:
+ tags: [email, communication, agentmail, mcp]
+ category: email
+---
+
+# AgentMail — Agent-Owned Email Inboxes
+
+## Requirements
+
+- **AgentMail API key** (required) — sign up at https://console.agentmail.to (free tier: 3 inboxes, 3,000 emails/month; paid plans from $20/mo)
+- Node.js 18+ (for the MCP server)
+
+## When to Use
+Use this skill when you need to:
+- Give the agent its own dedicated email address
+- Send emails autonomously on behalf of the agent
+- Receive and read incoming emails
+- Manage email threads and conversations
+- Sign up for services or authenticate via email
+- Communicate with other agents or humans via email
+
+This is NOT for reading the user's personal email (use himalaya or Gmail for that).
+AgentMail gives the agent its own identity and inbox.
+
+## Setup
+
+### 1. Get an API Key
+- Go to https://console.agentmail.to
+- Create an account and generate an API key (starts with `am_`)
+
+### 2. Configure MCP Server
+Add to `~/.hermes/config.yaml` (paste your actual key — MCP env vars are not expanded from .env):
+```yaml
+mcp_servers:
+ agentmail:
+ command: "npx"
+ args: ["-y", "agentmail-mcp"]
+ env:
+ AGENTMAIL_API_KEY: "am_your_key_here"
+```
+
+### 3. Restart Hermes
+```bash
+hermes
+```
+All 11 AgentMail tools are now available automatically.
+
+## Available Tools (via MCP)
+
+| Tool | Description |
+|------|-------------|
+| `list_inboxes` | List all agent inboxes |
+| `get_inbox` | Get details of a specific inbox |
+| `create_inbox` | Create a new inbox (gets a real email address) |
+| `delete_inbox` | Delete an inbox |
+| `list_threads` | List email threads in an inbox |
+| `get_thread` | Get a specific email thread |
+| `send_message` | Send a new email |
+| `reply_to_message` | Reply to an existing email |
+| `forward_message` | Forward an email |
+| `update_message` | Update message labels/status |
+| `get_attachment` | Download an email attachment |
+
+## Procedure
+
+### Create an inbox and send an email
+1. Create a dedicated inbox:
+ - Use `create_inbox` with a username (e.g. `hermes-agent`)
+ - The agent gets address: `hermes-agent@agentmail.to`
+2. Send an email:
+ - Use `send_message` with `inbox_id`, `to`, `subject`, `text`
+3. Check for replies:
+ - Use `list_threads` to see incoming conversations
+ - Use `get_thread` to read a specific thread
+
+### Check incoming email
+1. Use `list_inboxes` to find your inbox ID
+2. Use `list_threads` with the inbox ID to see conversations
+3. Use `get_thread` to read a thread and its messages
+
+### Reply to an email
+1. Get the thread with `get_thread`
+2. Use `reply_to_message` with the message ID and your reply text
+
+## Example Workflows
+
+**Sign up for a service:**
+```
+1. create_inbox (username: "signup-bot")
+2. Use the inbox address to register on the service
+3. list_threads to check for verification email
+4. get_thread to read the verification code
+```
+
+**Agent-to-human outreach:**
+```
+1. create_inbox (username: "hermes-outreach")
+2. send_message (to: user@example.com, subject: "Hello", text: "...")
+3. list_threads to check for replies
+```
+
+## Pitfalls
+- Free tier limited to 3 inboxes and 3,000 emails/month
+- Emails come from `@agentmail.to` domain on free tier (custom domains on paid plans)
+- Node.js (18+) is required for the MCP server (`npx -y agentmail-mcp`)
+- The `mcp` Python package must be installed: `pip install mcp`
+- Real-time inbound email (webhooks) requires a public server — use `list_threads` polling via cronjob instead for personal use
+
+## Verification
+After setup, test with:
+```
+hermes --toolsets mcp -q "Create an AgentMail inbox called test-agent and tell me its email address"
+```
+You should see the new inbox address returned.
+
+## References
+- AgentMail docs: https://docs.agentmail.to/
+- AgentMail console: https://console.agentmail.to
+- AgentMail MCP repo: https://github.com/agentmail-to/agentmail-mcp
+- Pricing: https://www.agentmail.to/pricing
diff --git a/src/skills/bundled/airtable/SKILL.md b/src/skills/bundled/airtable/SKILL.md
new file mode 100644
index 0000000..d628c23
--- /dev/null
+++ b/src/skills/bundled/airtable/SKILL.md
@@ -0,0 +1,229 @@
+---
+name: airtable
+description: Airtable REST API via curl. Records CRUD, filters, upserts.
+version: 1.1.0
+author: community
+license: MIT
+platforms: [linux, macos, windows]
+prerequisites:
+ env_vars: [AIRTABLE_API_KEY]
+ commands: [curl]
+metadata:
+ hermes:
+ tags: [Airtable, Productivity, Database, API]
+ homepage: https://airtable.com/developers/web/api/introduction
+---
+
+# Airtable — Bases, Tables & Records
+
+Work with Airtable's REST API directly via `curl` using the `terminal` tool. No MCP server, no OAuth flow, no Python SDK — just `curl` and a personal access token.
+
+## Prerequisites
+
+1. Create a **Personal Access Token (PAT)** at https://airtable.com/create/tokens (tokens start with `pat...`).
+2. Grant these scopes (minimum):
+ - `data.records:read` — read rows
+ - `data.records:write` — create / update / delete rows
+ - `schema.bases:read` — list bases and tables
+3. **Important:** in the same token UI, add each base you want to access to the token's **Access** list. PATs are scoped per-base — a valid token on the wrong base returns `403`.
+4. Store the token in `${HERMES_HOME:-~/.hermes}/.env` (or via `hermes setup`):
+ ```
+ AIRTABLE_API_KEY=pat_your_token_here
+ ```
+
+> Note: legacy `key...` API keys were deprecated Feb 2024. Only PATs and OAuth tokens work now.
+
+## API Basics
+
+- **Endpoint:** `https://api.airtable.com/v0`
+- **Auth header:** `Authorization: Bearer $AIRTABLE_API_KEY`
+- **All requests** use JSON (`Content-Type: application/json` for any POST/PATCH/PUT body).
+- **Object IDs:** bases `app...`, tables `tbl...`, records `rec...`, fields `fld...`. IDs never change; names can. Prefer IDs in automations.
+- **Rate limit:** 5 requests/sec/base. `429` → back off. Burst on a single base will be throttled.
+
+Base curl pattern:
+```bash
+curl -s "https://api.airtable.com/v0/$BASE_ID/$TABLE?maxRecords=5" \
+ -H "Authorization: Bearer $AIRTABLE_API_KEY" | python -m json.tool
+```
+
+`-s` suppresses curl's progress bar — keep it set for every call so the tool output stays clean for Hermes. Pipe through `python -m json.tool` (always present) or `jq` (if installed) for readable JSON.
+
+## Field Types (request body shapes)
+
+| Field type | Write shape |
+|---|---|
+| Single line text | `"Name": "hello"` |
+| Long text | `"Notes": "multi\nline"` |
+| Number | `"Score": 42` |
+| Checkbox | `"Done": true` |
+| Single select | `"Status": "Todo"` (name must already exist unless `typecast: true`) |
+| Multi-select | `"Tags": ["urgent", "bug"]` |
+| Date | `"Due": "2026-04-01"` |
+| DateTime (UTC) | `"At": "2026-04-01T14:30:00.000Z"` |
+| URL / Email / Phone | `"Link": "https://…"` |
+| Attachment | `"Files": [{"url": "https://…"}]` (Airtable fetches + rehosts) |
+| Linked record | `"Owner": ["recXXXXXXXXXXXXXX"]` (array of record IDs) |
+| User | `"AssignedTo": {"id": "usrXXXXXXXXXXXXXX"}` |
+
+Pass `"typecast": true` at the top level of a create/update body to let Airtable auto-coerce values (e.g. create a new select option on the fly, convert `"42"` → `42`).
+
+## Common Queries
+
+### List bases the token can see
+```bash
+curl -s "https://api.airtable.com/v0/meta/bases" \
+ -H "Authorization: Bearer $AIRTABLE_API_KEY" | python -m json.tool
+```
+
+### List tables + schema for a base
+```bash
+curl -s "https://api.airtable.com/v0/meta/bases/$BASE_ID/tables" \
+ -H "Authorization: Bearer $AIRTABLE_API_KEY" | python -m json.tool
+```
+Use this BEFORE mutating — confirms exact field names and IDs, surfaces `options.choices` for select fields, and shows primary-field names.
+
+### List records (first 10)
+```bash
+curl -s "https://api.airtable.com/v0/$BASE_ID/$TABLE?maxRecords=10" \
+ -H "Authorization: Bearer $AIRTABLE_API_KEY" | python -m json.tool
+```
+
+### Get a single record
+```bash
+curl -s "https://api.airtable.com/v0/$BASE_ID/$TABLE/$RECORD_ID" \
+ -H "Authorization: Bearer $AIRTABLE_API_KEY" | python -m json.tool
+```
+
+### Filter records (filterByFormula)
+Airtable formulas must be URL-encoded. Let Python stdlib do it — never hand-encode:
+```bash
+FORMULA="{Status}='Todo'"
+ENC=$(python -c 'import sys, urllib.parse; print(urllib.parse.quote(sys.argv[1], safe=""))' "$FORMULA")
+curl -s "https://api.airtable.com/v0/$BASE_ID/$TABLE?filterByFormula=$ENC&maxRecords=20" \
+ -H "Authorization: Bearer $AIRTABLE_API_KEY" | python -m json.tool
+```
+
+Useful formula patterns:
+- Exact match: `{Email}='user@example.com'`
+- Contains: `FIND('bug', LOWER({Title}))`
+- Multiple conditions: `AND({Status}='Todo', {Priority}='High')`
+- Or: `OR({Owner}='alice', {Owner}='bob')`
+- Not empty: `NOT({Assignee}='')`
+- Date comparison: `IS_AFTER({Due}, TODAY())`
+
+### Sort + select specific fields
+```bash
+curl -s "https://api.airtable.com/v0/$BASE_ID/$TABLE?sort%5B0%5D%5Bfield%5D=Priority&sort%5B0%5D%5Bdirection%5D=asc&fields%5B%5D=Name&fields%5B%5D=Status" \
+ -H "Authorization: Bearer $AIRTABLE_API_KEY" | python -m json.tool
+```
+Square brackets in query params MUST be URL-encoded (`%5B` / `%5D`).
+
+### Use a named view
+```bash
+curl -s "https://api.airtable.com/v0/$BASE_ID/$TABLE?view=Grid%20view&maxRecords=50" \
+ -H "Authorization: Bearer $AIRTABLE_API_KEY" | python -m json.tool
+```
+Views apply their saved filter + sort server-side.
+
+## Common Mutations
+
+### Create a record
+```bash
+curl -s -X POST "https://api.airtable.com/v0/$BASE_ID/$TABLE" \
+ -H "Authorization: Bearer $AIRTABLE_API_KEY" \
+ -H "Content-Type: application/json" \
+ -d '{"fields":{"Name":"New task","Status":"Todo","Priority":"High"}}' | python -m json.tool
+```
+
+### Create up to 10 records in one call
+```bash
+curl -s -X POST "https://api.airtable.com/v0/$BASE_ID/$TABLE" \
+ -H "Authorization: Bearer $AIRTABLE_API_KEY" \
+ -H "Content-Type: application/json" \
+ -d '{
+ "typecast": true,
+ "records": [
+ {"fields": {"Name": "Task A", "Status": "Todo"}},
+ {"fields": {"Name": "Task B", "Status": "In progress"}}
+ ]
+ }' | python -m json.tool
+```
+Batch endpoints are capped at **10 records per request**. For larger inserts, loop in batches of 10 with a short sleep to respect 5 req/sec/base.
+
+### Update a record (PATCH — merges, preserves unchanged fields)
+```bash
+curl -s -X PATCH "https://api.airtable.com/v0/$BASE_ID/$TABLE/$RECORD_ID" \
+ -H "Authorization: Bearer $AIRTABLE_API_KEY" \
+ -H "Content-Type: application/json" \
+ -d '{"fields":{"Status":"Done"}}' | python -m json.tool
+```
+
+### Upsert by a merge field (no ID needed)
+```bash
+curl -s -X PATCH "https://api.airtable.com/v0/$BASE_ID/$TABLE" \
+ -H "Authorization: Bearer $AIRTABLE_API_KEY" \
+ -H "Content-Type: application/json" \
+ -d '{
+ "performUpsert": {"fieldsToMergeOn": ["Email"]},
+ "records": [
+ {"fields": {"Email": "user@example.com", "Status": "Active"}}
+ ]
+ }' | python -m json.tool
+```
+`performUpsert` creates records whose merge-field values are new, patches records whose merge-field values already exist. Great for idempotent syncs.
+
+### Delete a record
+```bash
+curl -s -X DELETE "https://api.airtable.com/v0/$BASE_ID/$TABLE/$RECORD_ID" \
+ -H "Authorization: Bearer $AIRTABLE_API_KEY" | python -m json.tool
+```
+
+### Delete up to 10 records in one call
+```bash
+curl -s -X DELETE "https://api.airtable.com/v0/$BASE_ID/$TABLE?records%5B%5D=rec1&records%5B%5D=rec2" \
+ -H "Authorization: Bearer $AIRTABLE_API_KEY" | python -m json.tool
+```
+
+## Pagination
+
+List endpoints return at most **100 records per page**. If the response includes `"offset": "..."`, pass it back on the next call. Loop until the field is absent:
+
+```bash
+OFFSET=""
+while :; do
+ URL="https://api.airtable.com/v0/$BASE_ID/$TABLE?pageSize=100"
+ [ -n "$OFFSET" ] && URL="$URL&offset=$OFFSET"
+ RESP=$(curl -s "$URL" -H "Authorization: Bearer $AIRTABLE_API_KEY")
+ echo "$RESP" | python -c 'import json,sys; d=json.load(sys.stdin); [print(r["id"], r["fields"].get("Name","")) for r in d["records"]]'
+ OFFSET=$(echo "$RESP" | python -c 'import json,sys; d=json.load(sys.stdin); print(d.get("offset",""))')
+ [ -z "$OFFSET" ] && break
+done
+```
+
+## Typical Hermes Workflow
+
+1. **Confirm auth.** `curl -s -o /dev/null -w "%{http_code}\n" https://api.airtable.com/v0/meta/bases -H "Authorization: Bearer $AIRTABLE_API_KEY"` — expect `200`.
+2. **Find the base.** List bases (step above) OR ask the user for the `app...` ID directly if the token lacks `schema.bases:read`.
+3. **Inspect the schema.** `GET /v0/meta/bases/$BASE_ID/tables` — cache the exact field names and primary-field name locally in the session before mutating anything.
+4. **Read before you write.** For "update X where Y", `filterByFormula` first to resolve the `rec...` ID, then `PATCH /v0/$BASE_ID/$TABLE/$RECORD_ID`. Never guess record IDs.
+5. **Batch writes.** Combine related creates into one 10-record POST to stay under the 5 req/sec budget.
+6. **Destructive ops.** Deletions can't be undone via API. If the user says "delete all Xs", echo back the filter + record count and confirm before firing.
+
+## Pitfalls
+
+- **`filterByFormula` MUST be URL-encoded.** Field names with spaces or non-ASCII also need encoding (`{My Field}` → `%7BMy%20Field%7D`). Use Python stdlib (pattern above) — never hand-escape.
+- **Empty fields are omitted from responses.** A missing `"Assignee"` key doesn't mean the field doesn't exist — it means this record's value is empty. Check the schema (step 3) before concluding a field is missing.
+- **PATCH vs PUT.** `PATCH` merges supplied fields into the record. `PUT` replaces the record entirely and clears any field you didn't include. Default to `PATCH`.
+- **Single-select options must exist.** Writing `"Status": "Shipping"` when `Shipping` isn't in the field's option list errors with `INVALID_MULTIPLE_CHOICE_OPTIONS` unless you pass `"typecast": true` (which auto-creates the option).
+- **Per-base token scoping.** A `403` on one base while another works means the token's Access list doesn't include that base — not a scope or auth issue. Send the user to https://airtable.com/create/tokens to grant it.
+- **Rate limits are per base, not per token.** 5 req/sec on `baseA` and 5 req/sec on `baseB` is fine; 6 req/sec on `baseA` alone will throttle. Monitor the `Retry-After` header on `429`.
+
+## Important Notes for Hermes
+
+- **Always use the `terminal` tool with `curl`.** Do NOT use `web_extract` (it can't send auth headers) or `browser_navigate` (needs UI auth and is slow).
+- **`AIRTABLE_API_KEY` flows from `${HERMES_HOME:-~/.hermes}/.env` into the subprocess automatically** when this skill is loaded — no need to re-export it before each `curl` call.
+- **Escape curly braces in formulas carefully.** In a heredoc body, `{Status}` is literal. In a shell argument, `{Status}` is safe outside `{...}` brace-expansion context — but pass dynamic strings through `python urllib.parse.quote` before splicing into a URL.
+- **Pretty-print with `python -m json.tool`** (always present) rather than `jq` (optional). Only reach for `jq` when you need filtering/projection.
+- **Pagination is per-page, not global.** Airtable's 100-record cap is a hard limit; there is no way to bump it. Loop with `offset` until the field is absent.
+- **Read the `errors` array** on non-2xx responses — Airtable returns structured error codes like `AUTHENTICATION_REQUIRED`, `INVALID_PERMISSIONS`, `MODEL_ID_NOT_FOUND`, `INVALID_MULTIPLE_CHOICE_OPTIONS` that tell you exactly what's wrong.
diff --git a/src/skills/bundled/antigravity-cli/SKILL.md b/src/skills/bundled/antigravity-cli/SKILL.md
new file mode 100644
index 0000000..2286c8d
--- /dev/null
+++ b/src/skills/bundled/antigravity-cli/SKILL.md
@@ -0,0 +1,241 @@
+---
+name: antigravity-cli
+description: "Operate the Antigravity CLI (agy): plugins, auth, sandbox."
+version: 0.2.0
+author: Tony Simons (asimons81), Hermes Agent
+license: MIT
+platforms: [linux, macos, windows]
+metadata:
+ hermes:
+ tags: [Coding-Agent, Antigravity, CLI, Auth, Plugins, Sandbox]
+ related_skills: [grok, codex, claude-code, hermes-agent]
+---
+
+# Antigravity CLI (`agy`)
+
+Operator guide for the Antigravity CLI, invoked as `agy`. Run all `agy`
+commands through the Hermes `terminal` tool; inspect its config and logs with
+`read_file`. This skill is reference + procedure — it does not wrap a network
+API, so there is nothing to authenticate from Hermes itself.
+
+## When to Use
+
+- Installing, updating, or smoke-testing the `agy` binary
+- Driving non-interactive `agy --print` / `agy -p` one-shots
+- Debugging Antigravity auth, sandbox, permissions, or plugin state
+- Reading Antigravity settings, keybindings, conversations, or logs
+
+## Mental model
+
+Antigravity has two layers — keep them distinct or the guidance will be wrong:
+
+1. **Shell wrapper commands** — `agy help`, `agy install`, `agy plugin`,
+ `agy update`, `agy changelog`. Run these through the `terminal` tool.
+2. **Interactive in-session slash commands** — `/config`, `/permissions`,
+ `/skills`, `/agents`, etc. These only exist inside a running `agy` TUI
+ session, not on the shell wrapper.
+
+`agy help` shows the shell wrapper surface, NOT the in-session slash commands.
+
+## Prerequisites
+
+- The `agy` binary on PATH. Verify through the `terminal` tool:
+ `command -v agy && agy --version`.
+- No env vars or API keys required by this skill — Antigravity manages its own
+ auth via the OS keyring / browser sign-in (see Authentication below).
+
+## How to Run
+
+Invoke every `agy` command through the `terminal` tool. Examples:
+
+```
+terminal(command="agy --version")
+terminal(command="agy help")
+terminal(command="agy plugin list")
+terminal(command="agy --print 'Summarize the repo in 3 bullets'", workdir="/path/to/project")
+```
+
+For an interactive multi-turn TUI session, launch `agy` with `pty=true` (and
+tmux for capture/monitoring), the same pattern the `codex` / `claude-code`
+skills use. For one-shot smoke tests and scripted prompts, prefer
+`agy --print` (non-interactive).
+
+To inspect Antigravity's own files, use `read_file` on the paths under Core
+paths below — do not `cat` them through the terminal.
+
+## Delegation patterns
+
+`agy` is a coding-agent backend in the same family as `codex` / `claude-code`,
+so the same delegation shapes apply. Use these when handing real work (features,
+fixes, reviews, second opinions) to Antigravity rather than just smoke-testing.
+
+### One-shot (preferred for scripted prompts and second opinions)
+
+```
+terminal(command="agy -p 'Review this diff for bugs and security issues' --model 'Gemini 3.1 Pro (High)'", workdir="/path/to/repo", timeout=300)
+```
+
+`-p` is non-interactive: it runs the prompt and exits. Pick the engine with
+`--model` (run `agy models` for the exact display strings, e.g.
+`'Gemini 3.1 Pro (High)'`, `'Claude Opus 4.6 (Thinking)'`). Add extra context
+roots with repeatable `--add-dir`.
+
+### Long / bounded runs (tests, builds, multi-file changes)
+
+Background it and get notified on completion, the same as the `codex` skill:
+
+```
+terminal(command="agy -p 'Implement the change described in TASK.md and run the tests' --dangerously-skip-permissions", workdir="/path/to/repo", background=true, notify_on_complete=true)
+# then: process(action="poll"/"log"/"wait", session_id=)
+```
+
+### Interactive multi-turn (PTY + tmux)
+
+For a conversational session, launch `agy -i` (or bare `agy`) under `pty=true`
+with tmux for `capture-pane` / `send-keys`, exactly the pattern documented in
+the `codex` / `claude-code` skills. Resume later with `--continue` / `-c` or a
+specific `--conversation `.
+
+### Parallel instances (batch sub-issue / worktree fan-out)
+
+Create one git worktree per task and launch an independent `agy -p` in each
+(background), then collect results — same worktree fan-out the `codex` skill
+uses for batch issue fixing. Bound concurrency to what the machine and your
+review capacity can absorb.
+
+### Output + bounding caveat (differs from Claude Code)
+
+- `agy -p` returns **plain text** — there is **no `--output-format json`** and
+ no result envelope with `session_id` / cost / turn count. Parse stdout
+ directly; don't expect a JSON object.
+- There is **no `--max-turns`**. A print run is bounded by **`--print-timeout`**
+ (default `5m`). Raise it for long tasks: `--print-timeout 20m`. Pair with the
+ `terminal` `timeout=` so the outer call doesn't cut the run short.
+
+### Orchestration boundary
+
+Antigravity is a **worker execution backend or third-opinion reviewer** — an
+execution detail owned by the agent/profile running a task, NOT a first-class
+orchestration primitive. Do not put `agy` on a kanban board as its own card or
+treat it as a coordination layer; route work through the normal task graph and
+let the assigned worker choose `agy` (vs. codex/claude-code/direct tools) as its
+method. Reach for it explicitly only when the user asks, when a worker is
+configured to wrap it, or when you want a Gemini-family cross-check against
+another agent's plan or diff.
+
+## Core paths
+
+- Binary / entrypoint: `agy`
+- App data dir: `~/.gemini/antigravity-cli/`
+- Settings file: `~/.gemini/antigravity-cli/settings.json`
+- Keybindings file: `~/.gemini/antigravity-cli/keybindings.json`
+- Logs: `~/.gemini/antigravity-cli/log/cli-*.log`
+- Conversations: `~/.gemini/antigravity-cli/conversations/`
+- Brain artifacts: `~/.gemini/antigravity-cli/brain/`
+- History: `~/.gemini/antigravity-cli/history.jsonl`
+- Plugin staging: `~/.gemini/antigravity-cli/plugins//`
+
+## Quick Reference
+
+### Wrapper commands
+- `agy changelog`
+- `agy help`
+- `agy install`
+- `agy plugin` / `agy plugins`
+- `agy update`
+
+### Useful flags
+- `--add-dir`
+- `--continue` / `-c`
+- `--conversation`
+- `--dangerously-skip-permissions`
+- `--print` / `-p`
+- `--print-timeout`
+- `--prompt`
+- `--prompt-interactive` / `-i`
+- `--sandbox`
+- `--log-file`
+- `--version`
+
+### Plugin subcommands (`agy plugin --help`)
+- `list`, `import [source]`, `install `, `uninstall `,
+ `enable `, `disable `, `validate [path]`, `link `,
+ `help`
+
+### Install flags (`agy install --help`)
+- `--dir`, `--skip-aliases`, `--skip-path`
+
+### In-session slash commands
+- **Conversation control:** `/resume` (`/switch`), `/rewind` (`/undo`),
+ `/rename `, `/clear`, `/fork`, `/reset`, `/new`
+- **Settings & tools:** `/config`, `/settings`, `/permissions`, `/model`,
+ `/keybindings`, `/statusline`, `/tasks`, `/skills`, `/mcp`, `/open `,
+ `/usage`, `/logout`, `/agents`
+- **Prompt helpers:** `@` path autocomplete, `esc esc` clears the prompt (when
+ not streaming), `!` runs a terminal command directly, `?` opens help
+
+## Settings and permissions
+
+### Common settings keys (`settings.json`)
+- `allowNonWorkspaceAccess`
+- `colorScheme`
+- `permissions.allow`
+- `trustedWorkspaces`
+
+### Permission modes
+`request-review`, `always-proceed`, `strict`, `proceed-in-sandbox`.
+
+### Sandbox behavior
+- `enableTerminalSandbox` is a boolean in `settings.json`; default `false`.
+- Launch-time overrides (`--sandbox`, `--dangerously-skip-permissions`) can
+ supersede persistent settings for the current session.
+
+## Authentication behavior
+
+- The CLI tries the OS secure keyring first.
+- With no saved session, it falls back to browser-based Google sign-in.
+- Locally it opens the default browser; over SSH it prints an authorization URL
+ and expects the auth code pasted back.
+- `/logout` removes saved credentials.
+
+## Plugins
+
+- Plugins stage under `~/.gemini/antigravity-cli/plugins//`.
+- They can bundle skills, agents, rules, MCP servers, and hooks.
+- `agy plugin list` returning no imported plugins is a valid empty state.
+
+## Pitfalls
+
+- `agy help` shows wrapper commands, not interactive slash commands.
+- `agy --version` is the safe non-interactive version check; `agy version` is
+ interactive and can fail without a real TTY.
+- First place to look for failures: `~/.gemini/antigravity-cli/log/cli-*.log`
+ (read with `read_file`).
+- Don't confuse persistent JSON settings with launch-time overrides.
+- `~/.gemini/antigravity-cli/bin/agentapi` is a thin wrapper to `agy agentapi`.
+- On WSL, token storage is file-based, so auth issues are usually local-file /
+ session-state problems, not browser-only problems.
+- Workspace identity can depend on launch directory and the `.antigravitycli`
+ project marker.
+- `agy -p` prints plain text only — no `--output-format json`, no result
+ envelope. Don't try to parse a JSON object out of it (unlike `claude-code`).
+- Bound print runs with `--print-timeout` (default `5m`), not `--max-turns`
+ (which does not exist on `agy`).
+
+## Verification
+
+Confirm the install is real and usable, all through the `terminal` tool (read
+files with `read_file`):
+
+1. `terminal(command="command -v agy")`
+2. `terminal(command="agy --version")`
+3. `terminal(command="agy help")`
+4. `terminal(command="agy plugin list")`
+5. `read_file` on `~/.gemini/antigravity-cli/settings.json`
+6. `read_file` on the latest `~/.gemini/antigravity-cli/log/cli-*.log`
+7. If needed, `read_file` on `~/.gemini/antigravity-cli/keybindings.json`
+
+## Support files
+
+- `references/cli-docs.md` — condensed notes from the getting-started, usage,
+ and features docs.
diff --git a/src/skills/bundled/antigravity-cli/references/cli-docs.md b/src/skills/bundled/antigravity-cli/references/cli-docs.md
new file mode 100644
index 0000000..2a7158c
--- /dev/null
+++ b/src/skills/bundled/antigravity-cli/references/cli-docs.md
@@ -0,0 +1,64 @@
+# Antigravity CLI docs, condensed
+
+Source pages reviewed:
+- `/docs/cli-getting-started`
+- `/docs/cli-using`
+- `/docs/cli-features`
+
+## Install
+- macOS/Linux: `curl -fsSL https://antigravity.google/cli/install.sh | bash`
+- Windows PowerShell: `irm https://antigravity.google/cli/install.ps1 | iex`
+- Windows CMD: `curl -fsSL https://antigravity.google/cli/install.cmd -o install.cmd && install.cmd && del install.cmd`
+
+## Authentication
+- Tries secure keyring first.
+- If no saved session exists, falls back to browser-based Google sign-in.
+- Local machine: opens the default browser.
+- SSH/remote: prints a secure authorization URL, then expects the auth code to be pasted back.
+- `/logout` removes saved credentials.
+
+## Config and files
+- Settings: `~/.gemini/antigravity-cli/settings.json`
+- Keybindings: `~/.gemini/antigravity-cli/keybindings.json`
+- Plugins: `~/.gemini/antigravity-cli/plugins//`
+
+## Useful slash commands
+- `/config`, `/settings`
+- `/permissions`
+- `/resume` / `/switch`
+- `/rewind` / `/undo`
+- `/rename `
+- `/model`
+- `/keybindings`
+- `/statusline`
+- `/tasks`
+- `/skills`
+- `/mcp`
+- `/open `
+- `/usage`
+- `/logout`
+- `/agents`
+
+## Prompt helpers
+- `@` path autocomplete
+- `esc esc` clears prompt when not streaming
+- `!` runs a terminal command
+- `?` opens help / slash command list
+
+## Permissions and sandbox
+- Permission modes: `request-review`, `always-proceed`, `strict`, `proceed-in-sandbox`
+- Launch overrides: `--sandbox`, `--dangerously-skip-permissions`
+- Sandbox setting: `enableTerminalSandbox` in `settings.json` (default `false`)
+
+## Plugins
+- Plugins can bundle skills, agents, rules, MCP servers, and hooks.
+- They are staged locally and auto-discovered once installed.
+
+## Subagents
+- `/agents` opens the panel for active/completed subagents.
+- Subagents can run in parallel and request approvals.
+
+## Keybindings
+- `~/.gemini/antigravity-cli/keybindings.json`
+- Malformed JSON falls back to defaults for broken actions.
+- Docs list default bindings for clear, submit, cancel, exit, suspend, editor, approval yes/no, navigation, clipboard, undo/redo, and newline insertion.
diff --git a/src/skills/bundled/apple-notes/SKILL.md b/src/skills/bundled/apple-notes/SKILL.md
new file mode 100644
index 0000000..3fcf4c2
--- /dev/null
+++ b/src/skills/bundled/apple-notes/SKILL.md
@@ -0,0 +1,94 @@
+---
+name: apple-notes
+description: "Manage Apple Notes via memo CLI: create, search, edit."
+version: 1.0.1
+author: Hermes Agent
+license: MIT
+platforms: [macos]
+metadata:
+ hermes:
+ tags: [Notes, Apple, macOS, note-taking]
+ related_skills: [obsidian]
+prerequisites:
+ commands: [memo]
+---
+
+# Apple Notes
+
+Use `memo` to manage Apple Notes directly from the terminal. Notes sync across all Apple devices via iCloud.
+
+## Prerequisites
+
+- **macOS** with Notes.app
+- Install: `brew tap antoniorodr/memo && brew install antoniorodr/memo/memo`
+- Grant Automation access to Notes.app when prompted (System Settings → Privacy → Automation)
+
+## When to Use
+
+- User asks to create, view, or search Apple Notes
+- Saving information to Notes.app for cross-device access
+- Organizing notes into folders
+- Exporting notes to Markdown/HTML
+
+## When NOT to Use
+
+- Obsidian vault management → use the `obsidian` skill
+- Bear Notes → separate app (not supported here)
+- Quick agent-only notes → use the `memory` tool instead
+
+## Quick Reference
+
+### View Notes
+
+```bash
+memo notes # List all notes
+memo notes -f "Folder Name" # Filter by folder
+memo notes -s "query" # Search notes (fuzzy)
+```
+
+### Create Notes
+
+```bash
+memo notes -a # Add a note (opens your $EDITOR)
+memo notes -a -f "Folder Name" # Add a note into a specific folder
+```
+
+`-a`/`--add` is a bare flag — it opens your `$EDITOR` to compose the note; it does
+not take a title argument. Use `-f/--folder` to target a folder. Set `$EDITOR`
+first (e.g. `export EDITOR=vim`).
+
+### Edit Notes
+
+```bash
+memo notes -e # Interactive selection to edit
+```
+
+### Delete Notes
+
+```bash
+memo notes -d # Interactive selection to delete
+```
+
+### Move Notes
+
+```bash
+memo notes -m # Move note to folder (interactive)
+```
+
+### Export Notes
+
+```bash
+memo notes -ex # Export to HTML/Markdown
+```
+
+## Limitations
+
+- Cannot edit notes containing images or attachments
+- Interactive prompts require terminal access (use pty=true if needed)
+- macOS only — requires Apple Notes.app
+
+## Rules
+
+1. Prefer Apple Notes when user wants cross-device sync (iPhone/iPad/Mac)
+2. Use the `memory` tool for agent-internal notes that don't need to sync
+3. Use the `obsidian` skill for Markdown-native knowledge management
diff --git a/src/skills/bundled/apple-reminders/SKILL.md b/src/skills/bundled/apple-reminders/SKILL.md
new file mode 100644
index 0000000..4536644
--- /dev/null
+++ b/src/skills/bundled/apple-reminders/SKILL.md
@@ -0,0 +1,130 @@
+---
+name: apple-reminders
+description: "Apple Reminders via remindctl: add, list, complete."
+version: 1.0.0
+author: Hermes Agent
+license: MIT
+platforms: [macos]
+metadata:
+ hermes:
+ tags: [Reminders, tasks, todo, macOS, Apple]
+prerequisites:
+ commands: [remindctl]
+---
+
+# Apple Reminders
+
+Use `remindctl` to manage Apple Reminders directly from the terminal. Tasks sync across all Apple devices via iCloud.
+
+## Prerequisites
+
+- **macOS** with Reminders.app
+- Install: `brew install steipete/tap/remindctl`
+- Grant Reminders permission when prompted
+- Check: `remindctl status` / Request: `remindctl authorize`
+
+## When to Use
+
+- User mentions "reminder" or "Reminders app"
+- Creating personal to-dos with due dates that sync to iOS
+- Managing Apple Reminders lists
+- User wants tasks to appear on their iPhone/iPad
+
+## When NOT to Use
+
+- Scheduling agent alerts → use the cronjob tool instead
+- Calendar events → use Apple Calendar or Google Calendar
+- Project task management → use GitHub Issues, Notion, etc.
+- If user says "remind me" but means an agent alert → clarify first
+
+## Quick Reference
+
+### View Reminders
+
+```bash
+remindctl # Today's reminders
+remindctl today # Today
+remindctl tomorrow # Tomorrow
+remindctl week # This week
+remindctl overdue # Past due
+remindctl all # Everything
+remindctl 2026-01-04 # Specific date
+```
+
+### Manage Lists
+
+```bash
+remindctl list # List all lists
+remindctl list Work # Show specific list
+remindctl list Projects --create # Create list
+remindctl list Work --delete # Delete list
+```
+
+### Create Reminders
+
+```bash
+remindctl add "Buy milk"
+remindctl add --title "Call mom" --list Personal --due tomorrow
+remindctl add --title "Meeting prep" --due "2026-02-15 09:00"
+```
+
+### Due Time vs Alarm / Early Nudge
+
+`--due` and `--alarm` are different fields:
+
+- `--due` sets the reminder's due date/time.
+- `--alarm` sets the EventKit alarm/notification trigger. Timed due reminders may default to an alarm at the due time, but pass `--alarm` explicitly when the user asks for an earlier nudge.
+
+For a reminder due at 2:00 PM with a notification 30 minutes earlier:
+
+```bash
+remindctl add --title "Hairdresser" --due "2026-05-15 14:00" --alarm "2026-05-15 13:30"
+```
+
+To edit an existing reminder:
+
+```bash
+remindctl edit 87354 --due "2026-05-15 14:00" --alarm "2026-05-15 13:30"
+```
+
+The Reminders UI may show or group the item by the alarm time because that is when the notification fires. Verify with JSON instead of assuming the due time moved:
+
+```bash
+remindctl today --json
+```
+
+Expected shape:
+
+- `dueDate`: actual due time
+- `alarmDate`: notification / early nudge time
+
+Apple's public `EKReminder` docs list only reminder-specific properties. Alarm support comes from inherited `EKCalendarItem` behavior exposed by remindctl's `--alarm` flag.
+
+### Complete / Delete
+
+```bash
+remindctl complete 1 2 3 # Complete by ID
+remindctl delete 4A83 --force # Delete by ID
+```
+
+### Output Formats
+
+```bash
+remindctl today --json # JSON for scripting
+remindctl today --plain # TSV format
+remindctl today --quiet # Counts only
+```
+
+## Date Formats
+
+Accepted by `--due` and date filters:
+- `today`, `tomorrow`, `yesterday`
+- `YYYY-MM-DD`
+- `YYYY-MM-DD HH:mm`
+- ISO 8601 (`2026-01-04T12:34:56Z`)
+
+## Rules
+
+1. When user says "remind me", clarify: Apple Reminders (syncs to phone) vs agent cronjob alert
+2. Always confirm reminder content and due date before creating
+3. Use `--json` for programmatic parsing
diff --git a/src/skills/bundled/architecture-diagram/SKILL.md b/src/skills/bundled/architecture-diagram/SKILL.md
new file mode 100644
index 0000000..2c813c5
--- /dev/null
+++ b/src/skills/bundled/architecture-diagram/SKILL.md
@@ -0,0 +1,148 @@
+---
+name: architecture-diagram
+description: "Dark-themed SVG architecture/cloud/infra diagrams as HTML."
+version: 1.0.0
+author: Cocoon AI (hello@cocoon-ai.com), ported by Hermes Agent
+license: MIT
+dependencies: []
+platforms: [linux, macos, windows]
+metadata:
+ hermes:
+ tags: [architecture, diagrams, SVG, HTML, visualization, infrastructure, cloud]
+ related_skills: [concept-diagrams, excalidraw]
+---
+
+# Architecture Diagram Skill
+
+Generate professional, dark-themed technical architecture diagrams as standalone HTML files with inline SVG graphics. No external tools, no API keys, no rendering libraries — just write the HTML file and open it in a browser.
+
+## Scope
+
+**Best suited for:**
+- Software system architecture (frontend / backend / database layers)
+- Cloud infrastructure (VPC, regions, subnets, managed services)
+- Microservice / service-mesh topology
+- Database + API map, deployment diagrams
+- Anything with a tech-infra subject that fits a dark, grid-backed aesthetic
+
+**Look elsewhere first for:**
+- Physics, chemistry, math, biology, or other scientific subjects
+- Physical objects (vehicles, hardware, anatomy, cross-sections)
+- Floor plans, narrative journeys, educational / textbook-style visuals
+- Hand-drawn whiteboard sketches (consider `excalidraw`)
+- Animated explainers (consider an animation skill)
+
+If a more specialized skill is available for the subject, prefer that. If none fits, this skill can also serve as a general SVG diagram fallback — the output will just carry the dark tech aesthetic described below.
+
+Based on [Cocoon AI's architecture-diagram-generator](https://github.com/Cocoon-AI/architecture-diagram-generator) (MIT).
+
+## Workflow
+
+1. User describes their system architecture (components, connections, technologies)
+2. Generate the HTML file following the design system below
+3. Save with `write_file` to a `.html` file (e.g. `~/architecture-diagram.html`)
+4. User opens in any browser — works offline, no dependencies
+
+### Output Location
+
+Save diagrams to a user-specified path, or default to the current working directory:
+```
+./[project-name]-architecture.html
+```
+
+### Preview
+
+After saving, suggest the user open it:
+```bash
+# macOS
+open ./my-architecture.html
+# Linux
+xdg-open ./my-architecture.html
+```
+
+## Design System & Visual Language
+
+### Color Palette (Semantic Mapping)
+
+Use specific `rgba` fills and hex strokes to categorize components:
+
+| Component Type | Fill (rgba) | Stroke (Hex) |
+| :--- | :--- | :--- |
+| **Frontend** | `rgba(8, 51, 68, 0.4)` | `#22d3ee` (cyan-400) |
+| **Backend** | `rgba(6, 78, 59, 0.4)` | `#34d399` (emerald-400) |
+| **Database** | `rgba(76, 29, 149, 0.4)` | `#a78bfa` (violet-400) |
+| **AWS/Cloud** | `rgba(120, 53, 15, 0.3)` | `#fbbf24` (amber-400) |
+| **Security** | `rgba(136, 19, 55, 0.4)` | `#fb7185` (rose-400) |
+| **Message Bus** | `rgba(251, 146, 60, 0.3)` | `#fb923c` (orange-400) |
+| **External** | `rgba(30, 41, 59, 0.5)` | `#94a3b8` (slate-400) |
+
+### Typography & Background
+- **Font:** JetBrains Mono (Monospace), loaded from Google Fonts
+- **Sizes:** 12px (Names), 9px (Sublabels), 8px (Annotations), 7px (Tiny labels)
+- **Background:** Slate-950 (`#020617`) with a subtle 40px grid pattern
+
+```svg
+
+
+
+
+```
+
+## Technical Implementation Details
+
+### Component Rendering
+Components are rounded rectangles (`rx="6"`) with 1.5px strokes. To prevent arrows from showing through semi-transparent fills, use a **double-rect masking technique**:
+1. Draw an opaque background rect (`#0f172a`)
+2. Draw the semi-transparent styled rect on top
+
+### Connection Rules
+- **Z-Order:** Draw arrows *early* in the SVG (after the grid) so they render behind component boxes
+- **Arrowheads:** Defined via SVG markers
+- **Security Flows:** Use dashed lines in rose color (`#fb7185`)
+- **Boundaries:**
+ - *Security Groups:* Dashed (`4,4`), rose color
+ - *Regions:* Large dashed (`8,4`), amber color, `rx="12"`
+
+### Spacing & Layout Logic
+- **Standard Height:** 60px (Services); 80-120px (Large components)
+- **Vertical Gap:** Minimum 40px between components
+- **Message Buses:** Must be placed *in the gap* between services, not overlapping them
+- **Legend Placement:** **CRITICAL.** Must be placed outside all boundary boxes. Calculate the lowest Y-coordinate of all boundaries and place the legend at least 20px below it.
+
+## Document Structure
+
+The generated HTML file follows a four-part layout:
+1. **Header:** Title with a pulsing dot indicator and subtitle
+2. **Main SVG:** The diagram contained within a rounded border card
+3. **Summary Cards:** A grid of three cards below the diagram for high-level details
+4. **Footer:** Minimal metadata
+
+### Info Card Pattern
+```html
+
+
+
+
Title
+
+
+
• Item one
+
• Item two
+
+
+```
+
+## Output Requirements
+- **Single File:** One self-contained `.html` file
+- **No External Dependencies:** All CSS and SVG must be inline (except Google Fonts)
+- **No JavaScript:** Use pure CSS for any animations (like pulsing dots)
+- **Compatibility:** Must render correctly in any modern web browser
+
+## Template Reference
+
+Load the full HTML template for the exact structure, CSS, and SVG component examples:
+
+```
+skill_view(name="architecture-diagram", file_path="templates/template.html")
+```
+
+The template contains working examples of every component type (frontend, backend, database, cloud, security), arrow styles (standard, dashed, curved), security groups, region boundaries, and the legend — use it as your structural reference when generating diagrams.
diff --git a/src/skills/bundled/architecture-diagram/templates/template.html b/src/skills/bundled/architecture-diagram/templates/template.html
new file mode 100644
index 0000000..f5b32fb
--- /dev/null
+++ b/src/skills/bundled/architecture-diagram/templates/template.html
@@ -0,0 +1,319 @@
+
+
+
+
+
+ [PROJECT NAME] Architecture Diagram
+
+
+
+
+
+
+
+
+
+
[PROJECT NAME] Architecture
+
+
[Subtitle description]
+
+
+
+
+
+
+
+
+
+
+
+
+
Card Title 1
+
+
+
• Item one
+
• Item two
+
• Item three
+
• Item four
+
+
+
+
+
+
+
Card Title 2
+
+
+
• Item one
+
• Item two
+
• Item three
+
• Item four
+
+
+
+
+
+
+
Card Title 3
+
+
+
• Item one
+
• Item two
+
• Item three
+
• Item four
+
+
+
+
+
+
+ [Project Name] • [Additional metadata]
+
+
+
+
diff --git a/src/skills/bundled/arxiv/SKILL.md b/src/skills/bundled/arxiv/SKILL.md
new file mode 100644
index 0000000..e3e6ac7
--- /dev/null
+++ b/src/skills/bundled/arxiv/SKILL.md
@@ -0,0 +1,282 @@
+---
+name: arxiv
+description: "Search arXiv papers by keyword, author, category, or ID."
+version: 1.0.0
+author: Hermes Agent
+license: MIT
+platforms: [linux, macos, windows]
+metadata:
+ hermes:
+ tags: [Research, Arxiv, Papers, Academic, Science, API]
+ related_skills: [ocr-and-documents]
+---
+
+# arXiv Research
+
+Search and retrieve academic papers from arXiv via their free REST API. No API key, no dependencies — just curl.
+
+## Quick Reference
+
+| Action | Command |
+|--------|---------|
+| Search papers | `curl "https://export.arxiv.org/api/query?search_query=all:QUERY&max_results=5"` |
+| Get specific paper | `curl "https://export.arxiv.org/api/query?id_list=2402.03300"` |
+| Read abstract (web) | `web_extract(urls=["https://arxiv.org/abs/2402.03300"])` |
+| Read full paper (PDF) | `web_extract(urls=["https://arxiv.org/pdf/2402.03300"])` |
+
+## Searching Papers
+
+The API returns Atom XML. Parse with `grep`/`sed` or pipe through `python` for clean output.
+
+### Basic search
+
+```bash
+curl -s "https://export.arxiv.org/api/query?search_query=all:GRPO+reinforcement+learning&max_results=5"
+```
+
+### Clean output (parse XML to readable format)
+
+```bash
+curl -s "https://export.arxiv.org/api/query?search_query=all:GRPO+reinforcement+learning&max_results=5&sortBy=submittedDate&sortOrder=descending" | python -c "
+import sys, xml.etree.ElementTree as ET
+ns = {'a': 'http://www.w3.org/2005/Atom'}
+root = ET.parse(sys.stdin).getroot()
+for i, entry in enumerate(root.findall('a:entry', ns)):
+ title = entry.find('a:title', ns).text.strip().replace('\n', ' ')
+ arxiv_id = entry.find('a:id', ns).text.strip().split('/abs/')[-1]
+ published = entry.find('a:published', ns).text[:10]
+ authors = ', '.join(a.find('a:name', ns).text for a in entry.findall('a:author', ns))
+ summary = entry.find('a:summary', ns).text.strip()[:200]
+ cats = ', '.join(c.get('term') for c in entry.findall('a:category', ns))
+ print(f'{i+1}. [{arxiv_id}] {title}')
+ print(f' Authors: {authors}')
+ print(f' Published: {published} | Categories: {cats}')
+ print(f' Abstract: {summary}...')
+ print(f' PDF: https://arxiv.org/pdf/{arxiv_id}')
+ print()
+"
+```
+
+## Search Query Syntax
+
+| Prefix | Searches | Example |
+|--------|----------|---------|
+| `all:` | All fields | `all:transformer+attention` |
+| `ti:` | Title | `ti:large+language+models` |
+| `au:` | Author | `au:vaswani` |
+| `abs:` | Abstract | `abs:reinforcement+learning` |
+| `cat:` | Category | `cat:cs.AI` |
+| `co:` | Comment | `co:accepted+NeurIPS` |
+
+### Boolean operators
+
+```
+# AND (default when using +)
+search_query=all:transformer+attention
+
+# OR
+search_query=all:GPT+OR+all:BERT
+
+# AND NOT
+search_query=all:language+model+ANDNOT+all:vision
+
+# Exact phrase
+search_query=ti:"chain+of+thought"
+
+# Combined
+search_query=au:hinton+AND+cat:cs.LG
+```
+
+## Sort and Pagination
+
+| Parameter | Options |
+|-----------|---------|
+| `sortBy` | `relevance`, `lastUpdatedDate`, `submittedDate` |
+| `sortOrder` | `ascending`, `descending` |
+| `start` | Result offset (0-based) |
+| `max_results` | Number of results (default 10, max 30000) |
+
+```bash
+# Latest 10 papers in cs.AI
+curl -s "https://export.arxiv.org/api/query?search_query=cat:cs.AI&sortBy=submittedDate&sortOrder=descending&max_results=10"
+```
+
+## Fetching Specific Papers
+
+```bash
+# By arXiv ID
+curl -s "https://export.arxiv.org/api/query?id_list=2402.03300"
+
+# Multiple papers
+curl -s "https://export.arxiv.org/api/query?id_list=2402.03300,2401.12345,2403.00001"
+```
+
+## BibTeX Generation
+
+After fetching metadata for a paper, generate a BibTeX entry:
+
+{% raw %}
+```bash
+curl -s "https://export.arxiv.org/api/query?id_list=1706.03762" | python -c "
+import sys, xml.etree.ElementTree as ET
+ns = {'a': 'http://www.w3.org/2005/Atom', 'arxiv': 'http://arxiv.org/schemas/atom'}
+root = ET.parse(sys.stdin).getroot()
+entry = root.find('a:entry', ns)
+if entry is None: sys.exit('Paper not found')
+title = entry.find('a:title', ns).text.strip().replace('\n', ' ')
+authors = ' and '.join(a.find('a:name', ns).text for a in entry.findall('a:author', ns))
+year = entry.find('a:published', ns).text[:4]
+raw_id = entry.find('a:id', ns).text.strip().split('/abs/')[-1]
+cat = entry.find('arxiv:primary_category', ns)
+primary = cat.get('term') if cat is not None else 'cs.LG'
+last_name = entry.find('a:author', ns).find('a:name', ns).text.split()[-1]
+print(f'@article{{{last_name}{year}_{raw_id.replace(\".\", \"\")},')
+print(f' title = {{{title}}},')
+print(f' author = {{{authors}}},')
+print(f' year = {{{year}}},')
+print(f' eprint = {{{raw_id}}},')
+print(f' archivePrefix = {{arXiv}},')
+print(f' primaryClass = {{{primary}}},')
+print(f' url = {{https://arxiv.org/abs/{raw_id}}}')
+print('}')
+"
+```
+{% endraw %}
+
+## Reading Paper Content
+
+After finding a paper, read it:
+
+```
+# Abstract page (fast, metadata + abstract)
+web_extract(urls=["https://arxiv.org/abs/2402.03300"])
+
+# Full paper (PDF → markdown via Firecrawl)
+web_extract(urls=["https://arxiv.org/pdf/2402.03300"])
+```
+
+For local PDF processing, see the `ocr-and-documents` skill.
+
+## Common Categories
+
+| Category | Field |
+|----------|-------|
+| `cs.AI` | Artificial Intelligence |
+| `cs.CL` | Computation and Language (NLP) |
+| `cs.CV` | Computer Vision |
+| `cs.LG` | Machine Learning |
+| `cs.CR` | Cryptography and Security |
+| `stat.ML` | Machine Learning (Statistics) |
+| `math.OC` | Optimization and Control |
+| `physics.comp-ph` | Computational Physics |
+
+Full list: https://arxiv.org/category_taxonomy
+
+## Helper Script
+
+The `scripts/search_arxiv.py` script handles XML parsing and provides clean output:
+
+```bash
+python scripts/search_arxiv.py "GRPO reinforcement learning"
+python scripts/search_arxiv.py "transformer attention" --max 10 --sort date
+python scripts/search_arxiv.py --author "Yann LeCun" --max 5
+python scripts/search_arxiv.py --category cs.AI --sort date
+python scripts/search_arxiv.py --id 2402.03300
+python scripts/search_arxiv.py --id 2402.03300,2401.12345
+```
+
+No dependencies — uses only Python stdlib.
+
+---
+
+## Semantic Scholar (Citations, Related Papers, Author Profiles)
+
+arXiv doesn't provide citation data or recommendations. Use the **Semantic Scholar API** for that — free, no key needed for basic use (1 req/sec), returns JSON.
+
+### Get paper details + citations
+
+```bash
+# By arXiv ID
+curl -s "https://api.semanticscholar.org/graph/v1/paper/arXiv:2402.03300?fields=title,authors,citationCount,referenceCount,influentialCitationCount,year,abstract" | python -m json.tool
+
+# By Semantic Scholar paper ID or DOI
+curl -s "https://api.semanticscholar.org/graph/v1/paper/DOI:10.1234/example?fields=title,citationCount"
+```
+
+### Get citations OF a paper (who cited it)
+
+```bash
+curl -s "https://api.semanticscholar.org/graph/v1/paper/arXiv:2402.03300/citations?fields=title,authors,year,citationCount&limit=10" | python -m json.tool
+```
+
+### Get references FROM a paper (what it cites)
+
+```bash
+curl -s "https://api.semanticscholar.org/graph/v1/paper/arXiv:2402.03300/references?fields=title,authors,year,citationCount&limit=10" | python -m json.tool
+```
+
+### Search papers (alternative to arXiv search, returns JSON)
+
+```bash
+curl -s "https://api.semanticscholar.org/graph/v1/paper/search?query=GRPO+reinforcement+learning&limit=5&fields=title,authors,year,citationCount,externalIds" | python -m json.tool
+```
+
+### Get paper recommendations
+
+```bash
+curl -s -X POST "https://api.semanticscholar.org/recommendations/v1/papers/" \
+ -H "Content-Type: application/json" \
+ -d '{"positivePaperIds": ["arXiv:2402.03300"], "negativePaperIds": []}' | python -m json.tool
+```
+
+### Author profile
+
+```bash
+curl -s "https://api.semanticscholar.org/graph/v1/author/search?query=Yann+LeCun&fields=name,hIndex,citationCount,paperCount" | python -m json.tool
+```
+
+### Useful Semantic Scholar fields
+
+`title`, `authors`, `year`, `abstract`, `citationCount`, `referenceCount`, `influentialCitationCount`, `isOpenAccess`, `openAccessPdf`, `fieldsOfStudy`, `publicationVenue`, `externalIds` (contains arXiv ID, DOI, etc.)
+
+---
+
+## Complete Research Workflow
+
+1. **Discover**: `python scripts/search_arxiv.py "your topic" --sort date --max 10`
+2. **Assess impact**: `curl -s "https://api.semanticscholar.org/graph/v1/paper/arXiv:ID?fields=citationCount,influentialCitationCount"`
+3. **Read abstract**: `web_extract(urls=["https://arxiv.org/abs/ID"])`
+4. **Read full paper**: `web_extract(urls=["https://arxiv.org/pdf/ID"])`
+5. **Find related work**: `curl -s "https://api.semanticscholar.org/graph/v1/paper/arXiv:ID/references?fields=title,citationCount&limit=20"`
+6. **Get recommendations**: POST to Semantic Scholar recommendations endpoint
+7. **Track authors**: `curl -s "https://api.semanticscholar.org/graph/v1/author/search?query=NAME"`
+
+## Rate Limits
+
+| API | Rate | Auth |
+|-----|------|------|
+| arXiv | ~1 req / 3 seconds | None needed |
+| Semantic Scholar | 1 req / second | None (100/sec with API key) |
+
+## Notes
+
+- arXiv returns Atom XML — use the helper script or parsing snippet for clean output
+- Semantic Scholar returns JSON — pipe through `python -m json.tool` for readability
+- arXiv IDs: old format (`hep-th/0601001`) vs new (`2402.03300`)
+- PDF: `https://arxiv.org/pdf/{id}` — Abstract: `https://arxiv.org/abs/{id}`
+- HTML (when available): `https://arxiv.org/html/{id}`
+- For local PDF processing, see the `ocr-and-documents` skill
+
+## ID Versioning
+
+- `arxiv.org/abs/1706.03762` always resolves to the **latest** version
+- `arxiv.org/abs/1706.03762v1` points to a **specific** immutable version
+- When generating citations, preserve the version suffix you actually read to prevent citation drift (a later version may substantially change content)
+- The API `` field returns the versioned URL (e.g., `http://arxiv.org/abs/1706.03762v7`)
+
+## Withdrawn Papers
+
+Papers can be withdrawn after submission. When this happens:
+- The `` field contains a withdrawal notice (look for "withdrawn" or "retracted")
+- Metadata fields may be incomplete
+- Always check the summary before treating a result as a valid paper
diff --git a/src/skills/bundled/arxiv/scripts/search_arxiv.py b/src/skills/bundled/arxiv/scripts/search_arxiv.py
new file mode 100644
index 0000000..0bd6b23
--- /dev/null
+++ b/src/skills/bundled/arxiv/scripts/search_arxiv.py
@@ -0,0 +1,114 @@
+#!/usr/bin/env python3
+"""Search arXiv and display results in a clean format.
+
+Usage:
+ python search_arxiv.py "GRPO reinforcement learning"
+ python search_arxiv.py "GRPO reinforcement learning" --max 10
+ python search_arxiv.py "GRPO reinforcement learning" --sort date
+ python search_arxiv.py --author "Yann LeCun" --max 5
+ python search_arxiv.py --category cs.AI --sort date --max 10
+ python search_arxiv.py --id 2402.03300
+ python search_arxiv.py --id 2402.03300,2401.12345
+"""
+import sys
+import urllib.request
+import urllib.parse
+import xml.etree.ElementTree as ET
+
+NS = {'a': 'http://www.w3.org/2005/Atom'}
+
+def search(query=None, author=None, category=None, ids=None, max_results=5, sort="relevance"):
+ params = {}
+
+ if ids:
+ params['id_list'] = ids
+ else:
+ parts = []
+ if query:
+ parts.append(f'all:{urllib.parse.quote(query)}')
+ if author:
+ parts.append(f'au:{urllib.parse.quote(author)}')
+ if category:
+ parts.append(f'cat:{category}')
+ if not parts:
+ print("Error: provide a query, --author, --category, or --id")
+ sys.exit(1)
+ params['search_query'] = '+AND+'.join(parts)
+
+ params['max_results'] = str(max_results)
+
+ sort_map = {"relevance": "relevance", "date": "submittedDate", "updated": "lastUpdatedDate"}
+ params['sortBy'] = sort_map.get(sort, sort)
+ params['sortOrder'] = 'descending'
+
+ url = "https://export.arxiv.org/api/query?" + "&".join(f"{k}={v}" for k, v in params.items())
+
+ req = urllib.request.Request(url, headers={'User-Agent': 'HermesAgent/1.0'})
+ with urllib.request.urlopen(req, timeout=15) as resp:
+ data = resp.read()
+
+ root = ET.fromstring(data)
+ entries = root.findall('a:entry', NS)
+
+ if not entries:
+ print("No results found.")
+ return
+
+ total = root.find('{http://a9.com/-/spec/opensearch/1.1/}totalResults')
+ if total is not None:
+ print(f"Found {total.text} results (showing {len(entries)})\n")
+
+ for i, entry in enumerate(entries):
+ title = entry.find('a:title', NS).text.strip().replace('\n', ' ')
+ raw_id = entry.find('a:id', NS).text.strip()
+ full_id = raw_id.split('/abs/')[-1] if '/abs/' in raw_id else raw_id
+ arxiv_id = full_id.split('v')[0] # base ID for links
+ published = entry.find('a:published', NS).text[:10]
+ updated = entry.find('a:updated', NS).text[:10]
+ authors = ', '.join(a.find('a:name', NS).text for a in entry.findall('a:author', NS))
+ summary = entry.find('a:summary', NS).text.strip().replace('\n', ' ')
+ cats = ', '.join(c.get('term') for c in entry.findall('a:category', NS))
+
+ version = full_id[len(arxiv_id):] if full_id != arxiv_id else ""
+ print(f"{i+1}. {title}")
+ print(f" ID: {arxiv_id}{version} | Published: {published} | Updated: {updated}")
+ print(f" Authors: {authors}")
+ print(f" Categories: {cats}")
+ print(f" Abstract: {summary[:300]}{'...' if len(summary) > 300 else ''}")
+ print(f" Links: https://arxiv.org/abs/{arxiv_id} | https://arxiv.org/pdf/{arxiv_id}")
+ print()
+
+
+if __name__ == "__main__":
+ args = sys.argv[1:]
+ if not args or args[0] in {"-h", "--help"}:
+ print(__doc__)
+ sys.exit(0)
+
+ query = None
+ author = None
+ category = None
+ ids = None
+ max_results = 5
+ sort = "relevance"
+
+ i = 0
+ positional = []
+ while i < len(args):
+ if args[i] == "--max" and i + 1 < len(args):
+ max_results = int(args[i + 1]); i += 2
+ elif args[i] == "--sort" and i + 1 < len(args):
+ sort = args[i + 1]; i += 2
+ elif args[i] == "--author" and i + 1 < len(args):
+ author = args[i + 1]; i += 2
+ elif args[i] == "--category" and i + 1 < len(args):
+ category = args[i + 1]; i += 2
+ elif args[i] == "--id" and i + 1 < len(args):
+ ids = args[i + 1]; i += 2
+ else:
+ positional.append(args[i]); i += 1
+
+ if positional:
+ query = " ".join(positional)
+
+ search(query=query, author=author, category=category, ids=ids, max_results=max_results, sort=sort)
diff --git a/src/skills/bundled/ascii-art/SKILL.md b/src/skills/bundled/ascii-art/SKILL.md
new file mode 100644
index 0000000..dd9a88b
--- /dev/null
+++ b/src/skills/bundled/ascii-art/SKILL.md
@@ -0,0 +1,322 @@
+---
+name: ascii-art
+description: "ASCII art: pyfiglet, cowsay, boxes, image-to-ascii."
+version: 4.0.0
+author: 0xbyt4, Hermes Agent
+license: MIT
+dependencies: []
+platforms: [linux, macos, windows]
+metadata:
+ hermes:
+ tags: [ASCII, Art, Banners, Creative, Unicode, Text-Art, pyfiglet, figlet, cowsay, boxes]
+ related_skills: [excalidraw]
+
+---
+
+# ASCII Art Skill
+
+Multiple tools for different ASCII art needs. All tools are local CLI programs or free REST APIs — no API keys required.
+
+## Tool 1: Text Banners (pyfiglet — local)
+
+Render text as large ASCII art banners. 571 built-in fonts.
+
+### Setup
+
+```bash
+pip install pyfiglet --break-system-packages -q
+```
+
+### Usage
+
+```bash
+python -m pyfiglet "YOUR TEXT" -f slant
+python -m pyfiglet "TEXT" -f doom -w 80 # Set width
+python -m pyfiglet --list_fonts # List all 571 fonts
+```
+
+### Recommended fonts
+
+| Style | Font | Best for |
+|-------|------|----------|
+| Clean & modern | `slant` | Project names, headers |
+| Bold & blocky | `doom` | Titles, logos |
+| Big & readable | `big` | Banners |
+| Classic banner | `banner3` | Wide displays |
+| Compact | `small` | Subtitles |
+| Cyberpunk | `cyberlarge` | Tech themes |
+| 3D effect | `3-d` | Splash screens |
+| Gothic | `gothic` | Dramatic text |
+
+### Tips
+
+- Preview 2-3 fonts and let the user pick their favorite
+- Short text (1-8 chars) works best with detailed fonts like `doom` or `block`
+- Long text works better with compact fonts like `small` or `mini`
+
+## Tool 2: Text Banners (asciified API — remote, no install)
+
+Free REST API that converts text to ASCII art. 250+ FIGlet fonts. Returns plain text directly — no parsing needed. Use this when pyfiglet is not installed or as a quick alternative.
+
+### Usage (via terminal curl)
+
+```bash
+# Basic text banner (default font)
+curl -s "https://asciified.thelicato.io/api/v2/ascii?text=Hello+World"
+
+# With a specific font
+curl -s "https://asciified.thelicato.io/api/v2/ascii?text=Hello&font=Slant"
+curl -s "https://asciified.thelicato.io/api/v2/ascii?text=Hello&font=Doom"
+curl -s "https://asciified.thelicato.io/api/v2/ascii?text=Hello&font=Star+Wars"
+curl -s "https://asciified.thelicato.io/api/v2/ascii?text=Hello&font=3-D"
+curl -s "https://asciified.thelicato.io/api/v2/ascii?text=Hello&font=Banner3"
+
+# List all available fonts (returns JSON array)
+curl -s "https://asciified.thelicato.io/api/v2/fonts"
+```
+
+### Tips
+
+- URL-encode spaces as `+` in the text parameter
+- The response is plain text ASCII art — no JSON wrapping, ready to display
+- Font names are case-sensitive; use the fonts endpoint to get exact names
+- Works from any terminal with curl — no Python or pip needed
+
+## Tool 3: Cowsay (Message Art)
+
+Classic tool that wraps text in a speech bubble with an ASCII character.
+
+### Setup
+
+```bash
+sudo apt install cowsay -y # Debian/Ubuntu
+# brew install cowsay # macOS
+```
+
+### Usage
+
+```bash
+cowsay "Hello World"
+cowsay -f tux "Linux rules" # Tux the penguin
+cowsay -f dragon "Rawr!" # Dragon
+cowsay -f stegosaurus "Roar!" # Stegosaurus
+cowthink "Hmm..." # Thought bubble
+cowsay -l # List all characters
+```
+
+### Available characters (50+)
+
+`beavis.zen`, `bong`, `bunny`, `cheese`, `daemon`, `default`, `dragon`,
+`dragon-and-cow`, `elephant`, `eyes`, `flaming-skull`, `ghostbusters`,
+`hellokitty`, `kiss`, `kitty`, `koala`, `luke-koala`, `mech-and-cow`,
+`meow`, `moofasa`, `moose`, `ren`, `sheep`, `skeleton`, `small`,
+`stegosaurus`, `stimpy`, `supermilker`, `surgery`, `three-eyes`,
+`turkey`, `turtle`, `tux`, `udder`, `vader`, `vader-koala`, `www`
+
+### Eye/tongue modifiers
+
+```bash
+cowsay -b "Borg" # =_= eyes
+cowsay -d "Dead" # x_x eyes
+cowsay -g "Greedy" # $_$ eyes
+cowsay -p "Paranoid" # @_@ eyes
+cowsay -s "Stoned" # *_* eyes
+cowsay -w "Wired" # O_O eyes
+cowsay -e "OO" "Msg" # Custom eyes
+cowsay -T "U " "Msg" # Custom tongue
+```
+
+## Tool 4: Boxes (Decorative Borders)
+
+Draw decorative ASCII art borders/frames around any text. 70+ built-in designs.
+
+### Setup
+
+```bash
+sudo apt install boxes -y # Debian/Ubuntu
+# brew install boxes # macOS
+```
+
+### Usage
+
+```bash
+echo "Hello World" | boxes # Default box
+echo "Hello World" | boxes -d stone # Stone border
+echo "Hello World" | boxes -d parchment # Parchment scroll
+echo "Hello World" | boxes -d cat # Cat border
+echo "Hello World" | boxes -d dog # Dog border
+echo "Hello World" | boxes -d unicornsay # Unicorn
+echo "Hello World" | boxes -d diamonds # Diamond pattern
+echo "Hello World" | boxes -d c-cmt # C-style comment
+echo "Hello World" | boxes -d html-cmt # HTML comment
+echo "Hello World" | boxes -a c # Center text
+boxes -l # List all 70+ designs
+```
+
+### Combine with pyfiglet or asciified
+
+```bash
+python -m pyfiglet "HERMES" -f slant | boxes -d stone
+# Or without pyfiglet installed:
+curl -s "https://asciified.thelicato.io/api/v2/ascii?text=HERMES&font=Slant" | boxes -d stone
+```
+
+## Tool 5: TOIlet (Colored Text Art)
+
+Like pyfiglet but with ANSI color effects and visual filters. Great for terminal eye candy.
+
+### Setup
+
+```bash
+sudo apt install toilet toilet-fonts -y # Debian/Ubuntu
+# brew install toilet # macOS
+```
+
+### Usage
+
+```bash
+toilet "Hello World" # Basic text art
+toilet -f bigmono12 "Hello" # Specific font
+toilet --gay "Rainbow!" # Rainbow coloring
+toilet --metal "Metal!" # Metallic effect
+toilet -F border "Bordered" # Add border
+toilet -F border --gay "Fancy!" # Combined effects
+toilet -f pagga "Block" # Block-style font (unique to toilet)
+toilet -F list # List available filters
+```
+
+### Filters
+
+`crop`, `gay` (rainbow), `metal`, `flip`, `flop`, `180`, `left`, `right`, `border`
+
+**Note**: toilet outputs ANSI escape codes for colors — works in terminals but may not render in all contexts (e.g., plain text files, some chat platforms).
+
+## Tool 6: Image to ASCII Art
+
+Convert images (PNG, JPEG, GIF, WEBP) to ASCII art.
+
+### Option A: ascii-image-converter (recommended, modern)
+
+```bash
+# Install
+sudo snap install ascii-image-converter
+# OR: go install github.com/TheZoraiz/ascii-image-converter@latest
+```
+
+```bash
+ascii-image-converter image.png # Basic
+ascii-image-converter image.png -C # Color output
+ascii-image-converter image.png -d 60,30 # Set dimensions
+ascii-image-converter image.png -b # Braille characters
+ascii-image-converter image.png -n # Negative/inverted
+ascii-image-converter https://url/image.jpg # Direct URL
+ascii-image-converter image.png --save-txt out # Save as text
+```
+
+### Option B: jp2a (lightweight, JPEG only)
+
+```bash
+sudo apt install jp2a -y
+jp2a --width=80 image.jpg
+jp2a --colors image.jpg # Colorized
+```
+
+## Tool 7: Search Pre-Made ASCII Art
+
+Search curated ASCII art from the web. Use `terminal` with `curl`.
+
+### Source A: ascii.co.uk (recommended for pre-made art)
+
+Large collection of classic ASCII art organized by subject. Art is inside HTML `
` tags. Fetch the page with curl, then extract art with a small Python snippet.
+
+**URL pattern:** `https://ascii.co.uk/art/{subject}`
+
+**Step 1 — Fetch the page:**
+
+```bash
+curl -s 'https://ascii.co.uk/art/cat' -o /tmp/ascii_art.html
+```
+
+**Step 2 — Extract art from pre tags:**
+
+```python
+import re, html
+with open('/tmp/ascii_art.html') as f:
+ text = f.read()
+arts = re.findall(r'
]*>(.*?)
', text, re.DOTALL)
+for art in arts:
+ clean = re.sub(r'<[^>]+>', '', art)
+ clean = html.unescape(clean).strip()
+ if len(clean) > 30:
+ print(clean)
+ print('\n---\n')
+```
+
+**Available subjects** (use as URL path):
+- Animals: `cat`, `dog`, `horse`, `bird`, `fish`, `dragon`, `snake`, `rabbit`, `elephant`, `dolphin`, `butterfly`, `owl`, `wolf`, `bear`, `penguin`, `turtle`
+- Objects: `car`, `ship`, `airplane`, `rocket`, `guitar`, `computer`, `coffee`, `beer`, `cake`, `house`, `castle`, `sword`, `crown`, `key`
+- Nature: `tree`, `flower`, `sun`, `moon`, `star`, `mountain`, `ocean`, `rainbow`
+- Characters: `skull`, `robot`, `angel`, `wizard`, `pirate`, `ninja`, `alien`
+- Holidays: `christmas`, `halloween`, `valentine`
+
+**Tips:**
+- Preserve artist signatures/initials — important etiquette
+- Multiple art pieces per page — pick the best one for the user
+- Works reliably via curl, no JavaScript needed
+
+### Source B: GitHub Octocat API (fun easter egg)
+
+Returns a random GitHub Octocat with a wise quote. No auth needed.
+
+```bash
+curl -s https://api.github.com/octocat
+```
+
+## Tool 8: Fun ASCII Utilities (via curl)
+
+These free services return ASCII art directly — great for fun extras.
+
+### QR Codes as ASCII Art
+
+```bash
+curl -s "qrenco.de/Hello+World"
+curl -s "qrenco.de/https://example.com"
+```
+
+### Weather as ASCII Art
+
+```bash
+curl -s "wttr.in/London" # Full weather report with ASCII graphics
+curl -s "wttr.in/Moon" # Moon phase in ASCII art
+curl -s "v2.wttr.in/London" # Detailed version
+```
+
+## Tool 9: LLM-Generated Custom Art (Fallback)
+
+When tools above don't have what's needed, generate ASCII art directly using these Unicode characters:
+
+### Character Palette
+
+**Box Drawing:** `╔ ╗ ╚ ╝ ║ ═ ╠ ╣ ╦ ╩ ╬ ┌ ┐ └ ┘ │ ─ ├ ┤ ┬ ┴ ┼ ╭ ╮ ╰ ╯`
+
+**Block Elements:** `░ ▒ ▓ █ ▄ ▀ ▌ ▐ ▖ ▗ ▘ ▝ ▚ ▞`
+
+**Geometric & Symbols:** `◆ ◇ ◈ ● ○ ◉ ■ □ ▲ △ ▼ ▽ ★ ☆ ✦ ✧ ◀ ▶ ◁ ▷ ⬡ ⬢ ⌂`
+
+### Rules
+
+- Max width: 60 characters per line (terminal-safe)
+- Max height: 15 lines for banners, 25 for scenes
+- Monospace only: output must render correctly in fixed-width fonts
+
+## Decision Flow
+
+1. **Text as a banner** → pyfiglet if installed, otherwise asciified API via curl
+2. **Wrap a message in fun character art** → cowsay
+3. **Add decorative border/frame** → boxes (can combine with pyfiglet/asciified)
+4. **Art of a specific thing** (cat, rocket, dragon) → ascii.co.uk via curl + parsing
+5. **Convert an image to ASCII** → ascii-image-converter or jp2a
+6. **QR code** → qrenco.de via curl
+7. **Weather/moon art** → wttr.in via curl
+8. **Something custom/creative** → LLM generation with Unicode palette
+9. **Any tool not installed** → install it, or fall back to next option
diff --git a/src/skills/bundled/ascii-video/README.md b/src/skills/bundled/ascii-video/README.md
new file mode 100644
index 0000000..9e17db0
--- /dev/null
+++ b/src/skills/bundled/ascii-video/README.md
@@ -0,0 +1,290 @@
+# ☤ ASCII Video
+
+Renders any content as colored ASCII character video. Audio, video, images, text, or pure math in, MP4/GIF/PNG sequence out. Full RGB color per character cell, 1080p 24fps default. No GPU.
+
+Built for [Hermes Agent](https://github.com/NousResearch/hermes-agent). Usable in any coding agent. Canonical source lives here; synced to [`NousResearch/hermes-agent/skills/creative/ascii-video`](https://github.com/NousResearch/hermes-agent/tree/main/skills/creative/ascii-video) via PR.
+
+## What this is
+
+A skill that teaches an agent how to build single-file Python renderers for ASCII video from scratch. The agent gets the full pipeline: grid system, font rasterization, effect library, shader chain, audio analysis, parallel encoding. It writes the renderer, runs it, gets video.
+
+The output is actual video. Not terminal escape codes. Frames are computed as grids of colored characters, composited onto pixel canvases with pre-rasterized font bitmaps, post-processed through shaders, piped to ffmpeg.
+
+## Modes
+
+| Mode | Input | Output |
+|------|-------|--------|
+| Video-to-ASCII | A video file | ASCII recreation of the footage |
+| Audio-reactive | An audio file | Visuals driven by frequency bands, beats, energy |
+| Generative | Nothing | Procedural animation from math |
+| Hybrid | Video + audio | ASCII video with audio-reactive overlays |
+| Lyrics/text | Audio + timed text (SRT) | Karaoke-style text with effects |
+| TTS narration | Text quotes + API key | Narrated video with typewriter text and generated speech |
+
+## Pipeline
+
+Every mode follows the same 6-stage path:
+
+```
+INPUT --> ANALYZE --> SCENE_FN --> TONEMAP --> SHADE --> ENCODE
+```
+
+1. **Input** loads source material (or nothing for generative).
+2. **Analyze** extracts per-frame features. Audio gets 6-band FFT, RMS, spectral centroid, flatness, flux, beat detection with exponential decay. Video gets luminance, edges, motion.
+3. **Scene function** returns a pixel canvas directly. Composes multiple character grids at different densities, value/hue fields, pixel blend modes. This is where the visuals happen.
+4. **Tonemap** does adaptive percentile-based brightness normalization with per-scene gamma. ASCII on black is inherently dark. Linear multipliers don't work. This does.
+5. **Shade** runs a `ShaderChain` (38 composable shaders) plus a `FeedbackBuffer` for temporal recursion with spatial transforms.
+6. **Encode** pipes raw RGB frames to ffmpeg for H.264 encoding. Segments concatenated, audio muxed.
+
+## Grid system
+
+Characters render on fixed-size grids. Layer multiple densities for depth.
+
+| Size | Font | Grid at 1080p | Use |
+|------|------|---------------|-----|
+| xs | 8px | 400x108 | Ultra-dense data fields |
+| sm | 10px | 320x83 | Rain, starfields |
+| md | 16px | 192x56 | Default balanced |
+| lg | 20px | 160x45 | Readable text |
+| xl | 24px | 137x37 | Large titles |
+| xxl | 40px | 80x22 | Giant minimal |
+
+Rendering the same scene on `sm` and `lg` then screen-blending them creates natural texture interference. Fine detail shows through gaps in coarse characters. Most scenes use two or three grids.
+
+## Character palettes (24)
+
+Each sorted dark-to-bright, each a different visual texture. Validated against the font at init so broken glyphs get dropped silently.
+
+| Family | Examples | Feel |
+|--------|----------|------|
+| Density ramps | ` .:-=+#@█` | Classic ASCII art gradient |
+| Block elements | ` ░▒▓█▄▀▐▌` | Chunky, digital |
+| Braille | ` ⠁⠂⠃...⠿` | Fine-grained pointillism |
+| Dots | ` ⋅∘∙●◉◎` | Smooth, organic |
+| Stars | ` ·✧✦✩✨★✶` | Sparkle, celestial |
+| Half-fills | ` ◔◑◕◐◒◓◖◗◙` | Directional fill progression |
+| Crosshatch | ` ▣▤▥▦▧▨▩` | Hatched density ramp |
+| Math | ` ·∘∙•°±×÷≈≠≡∞∫∑Ω` | Scientific, abstract |
+| Box drawing | ` ─│┌┐└┘├┤┬┴┼` | Structural, circuit-like |
+| Katakana | ` ·ヲァィゥェォャュ...` | Matrix rain |
+| Greek | ` αβγδεζηθ...ω` | Classical, academic |
+| Runes | ` ᚠᚢᚦᚱᚷᛁᛇᛒᛖᛚᛞᛟ` | Mystical, ancient |
+| Alchemical | ` ☉☽♀♂♃♄♅♆♇` | Esoteric |
+| Arrows | ` ←↑→↓↔↕↖↗↘↙` | Directional, kinetic |
+| Music | ` ♪♫♬♩♭♮♯○●` | Musical |
+| Project-specific | ` .·~=≈∞⚡☿✦★⊕◊◆▲▼●■` | Themed per project |
+
+Custom palettes are built per project to match the content.
+
+## Color strategies
+
+| Strategy | How it maps hue | Good for |
+|----------|----------------|----------|
+| Angle-mapped | Position angle from center | Rainbow radial effects |
+| Distance-mapped | Distance from center | Depth, tunnels |
+| Frequency-mapped | Audio spectral centroid | Timbral shifting |
+| Value-mapped | Brightness level | Heat maps, fire |
+| Time-cycled | Slow rotation over time | Ambient, chill |
+| Source-sampled | Original video pixel colors | Video-to-ASCII |
+| Palette-indexed | Discrete lookup table | Retro, flat graphic |
+| Temperature | Warm-to-cool blend | Emotional tone |
+| Complementary | Hue + opposite | Bold, dramatic |
+| Triadic | Three equidistant hues | Psychedelic, vibrant |
+| Analogous | Neighboring hues | Harmonious, subtle |
+| Monochrome | Fixed hue, vary S/V | Noir, focused |
+
+Plus 10 discrete RGB palettes (neon, pastel, cyberpunk, vaporwave, earth, ice, blood, forest, mono-green, mono-amber).
+
+Full OKLAB/OKLCH color system: sRGB↔linear↔OKLAB conversion pipeline, perceptually uniform gradient interpolation, and color harmony generation (complementary, triadic, analogous, split-complementary, tetradic).
+
+## Value field generators (21)
+
+Value fields are the core visual building blocks. Each produces a 2D float array in [0, 1] mapping every grid cell to a brightness value.
+
+### Trigonometric (12)
+
+| Field | Description |
+|-------|-------------|
+| Sine field | Layered multi-sine interference, general-purpose background |
+| Smooth noise | Multi-octave sine approximation of Perlin noise |
+| Rings | Concentric rings, bass-driven count and wobble |
+| Spiral | Logarithmic spiral arms, configurable arm count/tightness |
+| Tunnel | Infinite depth perspective (inverse distance) |
+| Vortex | Twisting radial pattern, distance modulates angle |
+| Interference | N overlapping sine waves creating moire |
+| Aurora | Horizontal flowing bands |
+| Ripple | Concentric waves from configurable source points |
+| Plasma | Sum of sines at multiple orientations/speeds |
+| Diamond | Diamond/checkerboard pattern |
+| Noise/static | Random per-cell per-frame flicker |
+
+### Noise-based (4)
+
+| Field | Description |
+|-------|-------------|
+| Value noise | Smooth organic noise, no axis-alignment artifacts |
+| fBM | Fractal Brownian Motion — octaved noise for clouds, terrain, smoke |
+| Domain warp | Inigo Quilez technique — fBM-driven coordinate distortion for flowing organic forms |
+| Voronoi | Moving seed points with distance, edge, and cell-ID output modes |
+
+### Simulation-based (4)
+
+| Field | Description |
+|-------|-------------|
+| Reaction-diffusion | Gray-Scott with 7 presets: coral, spots, worms, labyrinths, mitosis, pulsating, chaos |
+| Cellular automata | Game of Life + 4 rule variants with analog fade trails |
+| Strange attractors | Clifford, De Jong, Bedhead — iterated point systems binned to density fields |
+| Temporal noise | 3D noise that morphs in-place without directional drift |
+
+### SDF-based
+
+7 signed distance field primitives (circle, box, ring, line, triangle, star, heart) with smooth boolean combinators (union, intersection, subtraction, smooth union/subtraction) and infinite tiling. Render as solid fills or glowing outlines.
+
+## Hue field generators (9)
+
+Determine per-cell color independent of brightness: fixed hue, angle-mapped rainbow, distance gradient, time-cycled rotation, audio spectral centroid, horizontal/vertical gradients, plasma variation, perceptually uniform OKLCH rainbow.
+
+## Coordinate transforms (11)
+
+UV-space transforms applied before effect evaluation: rotate, scale, skew, tile (with mirror seaming), polar, inverse-polar, twist (rotation increasing with distance), fisheye, wave displacement, Möbius conformal transformation. `make_tgrid()` wraps transformed coordinates into a grid object.
+
+## Particle systems (9)
+
+| Type | Behavior |
+|------|----------|
+| Explosion | Beat-triggered radial burst with gravity and life decay |
+| Embers | Rising from bottom with horizontal drift |
+| Dissolving cloud | Spreading outward with accelerating fade |
+| Starfield | 3D projected, Z-depth stars approaching with streak trails |
+| Orbit | Circular/elliptical paths around center |
+| Gravity well | Attracted toward configurable point sources |
+| Boid flocking | Separation/alignment/cohesion with spatial hash for O(n) neighbors |
+| Flow-field | Steered by gradient of any value field |
+| Trail particles | Fading lines between current and previous positions |
+
+14 themed particle character sets (energy, spark, leaf, snow, rain, bubble, data, hex, binary, rune, zodiac, dot, dash).
+
+## Temporal coherence
+
+10 easing functions (linear, quad, cubic, expo, elastic, bounce — in/out/in-out). Keyframe interpolation with eased transitions. Value field morphing (smooth crossfade between fields). Value field sequencing (cycle through fields with crossfade). Temporal noise (3D noise evolving smoothly in-place).
+
+## Shader pipeline
+
+38 composable shaders, applied to the pixel canvas after character rendering. Configurable per section.
+
+| Category | Shaders |
+|----------|---------|
+| Geometry | CRT barrel, pixelate, wave distort, displacement map, kaleidoscope, mirror (h/v/quad/diag) |
+| Channel | Chromatic aberration (beat-reactive), channel shift, channel swap, RGB split radial |
+| Color | Invert, posterize, threshold, solarize, hue rotate, saturation, color grade, color wobble, color ramp |
+| Glow/Blur | Bloom, edge glow, soft focus, radial blur |
+| Noise | Film grain (beat-reactive), static noise |
+| Lines/Patterns | Scanlines, halftone |
+| Tone | Vignette, contrast, gamma, levels, brightness |
+| Glitch/Data | Glitch bands (beat-reactive), block glitch, pixel sort, data bend |
+
+12 color tint presets: warm, cool, matrix green, amber, sepia, neon pink, ice, blood, forest, void, sunset, neutral.
+
+7 mood presets for common shader combos:
+
+| Mood | Shaders |
+|------|---------|
+| Retro terminal | CRT + scanlines + grain + amber/green tint |
+| Clean modern | Light bloom + subtle vignette |
+| Glitch art | Heavy chromatic + glitch bands + color wobble |
+| Cinematic | Bloom + vignette + grain + color grade |
+| Dreamy | Heavy bloom + soft focus + color wobble |
+| Harsh/industrial | High contrast + grain + scanlines, no bloom |
+| Psychedelic | Color wobble + chromatic + kaleidoscope mirror |
+
+## Blend modes and composition
+
+20 pixel blend modes for layering canvases: normal, add, subtract, multiply, screen, overlay, softlight, hardlight, difference, exclusion, colordodge, colorburn, linearlight, vividlight, pin_light, hard_mix, lighten, darken, grain_extract, grain_merge. Both sRGB and linear-light blending supported.
+
+**Feedback buffer.** Temporal recursion — each frame blends with a transformed version of the previous frame. 7 spatial transforms: zoom, shrink, rotate CW/CCW, shift up/down, mirror. Optional per-frame hue shift for rainbow trails. Configurable decay, blend mode, and opacity per scene.
+
+**Masking.** 16 mask types for spatial compositing: shape masks (circle, rect, ring, gradients), procedural masks (any value field as a mask, text stencils), animated masks (iris open/close, wipe, dissolve), boolean operations (union, intersection, subtraction, invert).
+
+**Transitions.** Crossfade, directional wipe, radial wipe, dissolve, glitch cut.
+
+## Scene design patterns
+
+Compositional patterns for making scenes that look intentional rather than random.
+
+**Layer hierarchy.** Background (dim atmosphere, dense grid), content (main visual, standard grid), accent (sparse highlights, coarse grid). Three distinct roles, not three competing layers.
+
+**Directional parameter arcs.** The defining parameter of each scene ramps, accelerates, or builds over its duration. Progress-based formulas (linear, ease-out, step reveal) replace aimless `sin(t)` oscillation.
+
+**Scene concepts.** Scenes built around visual metaphors (emergence, descent, collision, entropy) with motivated layer/palette/feedback choices. Not named after their effects.
+
+**Compositional techniques.** Counter-rotating dual systems, wave collision, progressive fragmentation (voronoi cells multiplying over time), entropy (geometry consumed by reaction-diffusion), staggered layer entry (crescendo buildup).
+
+## Hardware adaptation
+
+Auto-detects CPU count, RAM, platform, ffmpeg. Adapts worker count, resolution, FPS.
+
+| Profile | Resolution | FPS | When |
+|---------|-----------|-----|------|
+| `draft` | 960x540 | 12 | Check timing/layout |
+| `preview` | 1280x720 | 15 | Review effects |
+| `production` | 1920x1080 | 24 | Final output |
+| `max` | 3840x2160 | 30 | Ultra-high |
+| `auto` | Detected | 24 | Adapts to hardware + duration |
+
+`auto` estimates render time and downgrades if it would take over an hour. Low-memory systems drop to 720p automatically.
+
+### Render times (1080p 24fps, ~180ms/frame/worker)
+
+| Duration | 4 workers | 8 workers | 16 workers |
+|----------|-----------|-----------|------------|
+| 30s | ~3 min | ~2 min | ~1 min |
+| 2 min | ~13 min | ~7 min | ~4 min |
+| 5 min | ~33 min | ~17 min | ~9 min |
+| 10 min | ~65 min | ~33 min | ~17 min |
+
+720p roughly halves these. 4K roughly quadruples them.
+
+## Known pitfalls
+
+**Brightness.** ASCII characters are small bright dots on black. Most frame pixels are background. Linear `* N` multipliers clip highlights and wash out. Use `tonemap()` with per-scene gamma instead. Default gamma 0.75, solarize scenes 0.55, posterize 0.50.
+
+**Render bottleneck.** The per-cell Python loop compositing font bitmaps runs at ~100-150ms/frame. Unavoidable without Cython/C. Everything else must be vectorized numpy. Python for-loops over rows/cols in effect functions will tank performance.
+
+**ffmpeg deadlock.** Never `stderr=subprocess.PIPE` on long-running encodes. Buffer fills at ~64KB, process hangs. Redirect stderr to a file.
+
+**Font cell height.** Pillow's `textbbox()` returns wrong height on macOS. Use `font.getmetrics()` for `ascent + descent`.
+
+**Font compatibility.** Not all Unicode renders in all fonts. Palettes validated at init, blank glyphs silently removed.
+
+## Requirements
+
+◆ Python 3.10+
+◆ NumPy, Pillow, SciPy (audio modes)
+◆ ffmpeg on PATH
+◆ A monospace font (Menlo, Courier, Monaco, auto-detected)
+◆ Optional: OpenCV, ElevenLabs API key (TTS mode)
+
+## File structure
+
+```
+├── SKILL.md # Modes, workflow, creative direction
+├── README.md # This file
+└── references/
+ ├── architecture.md # Grid system, fonts, palettes, color, _render_vf()
+ ├── effects.md # Value fields, hue fields, backgrounds, particles
+ ├── shaders.md # 38 shaders, ShaderChain, tint presets, transitions
+ ├── composition.md # Blend modes, multi-grid, tonemap, FeedbackBuffer
+ ├── scenes.md # Scene protocol, SCENES table, render_clip(), examples
+ ├── design-patterns.md # Layer hierarchy, directional arcs, scene concepts
+ ├── inputs.md # Audio analysis, video sampling, text, TTS
+ ├── optimization.md # Hardware detection, vectorized patterns, parallelism
+ └── troubleshooting.md # Broadcasting traps, blend pitfalls, diagnostics
+```
+
+## Projects built with this
+
+✦ 85-second highlight reel. 15 scenes (14×5s + 15s crescendo finale), randomized order, directional parameter arcs, layer hierarchy composition. Showcases the full effect vocabulary: fBM, voronoi fragmentation, reaction-diffusion, cellular automata, dual counter-rotating spirals, wave collision, domain warping, tunnel descent, kaleidoscope symmetry, boid flocking, fire simulation, glitch corruption, and a 7-layer crescendo buildup.
+
+✦ Audio-reactive music visualizer. 3.5 min, 8 sections with distinct effects, beat-triggered particles and glitch, cycling palettes.
+
+✦ TTS narrated testimonial video. 23 quotes, per-quote ElevenLabs voices, background music at 15% wide stereo, per-clip re-rendering for iterative editing.
diff --git a/src/skills/bundled/ascii-video/SKILL.md b/src/skills/bundled/ascii-video/SKILL.md
new file mode 100644
index 0000000..8534dfb
--- /dev/null
+++ b/src/skills/bundled/ascii-video/SKILL.md
@@ -0,0 +1,248 @@
+---
+name: ascii-video
+description: "ASCII video: convert video/audio to colored ASCII MP4/GIF."
+version: 1.0.0
+author: SHL0MS, Hermes Agent
+license: MIT
+platforms: [linux, macos, windows]
+metadata:
+ hermes:
+ tags: [ASCII, Video, FFmpeg, Terminal-Art]
+ related_skills: []
+---
+
+# ASCII Video Production Pipeline
+
+## When to use
+
+Use when users request: ASCII video, text art video, terminal-style video, character art animation, retro text visualization, audio visualizer in ASCII, converting video to ASCII art, matrix-style effects, or any animated ASCII output.
+
+## What's inside
+
+Production pipeline for ASCII art video — any format. Converts video/audio/images/generative input into colored ASCII character video output (MP4, GIF, image sequence). Covers: video-to-ASCII conversion, audio-reactive music visualizers, generative ASCII art animations, hybrid video+audio reactive, text/lyrics overlays, real-time terminal rendering.
+
+## Creative Standard
+
+This is visual art. ASCII characters are the medium; cinema is the standard.
+
+**Before writing a single line of code**, articulate the creative concept. What is the mood? What visual story does this tell? What makes THIS project different from every other ASCII video? The user's prompt is a starting point — interpret it with creative ambition, not literal transcription.
+
+**First-render excellence is non-negotiable.** The output must be visually striking without requiring revision rounds. If something looks generic, flat, or like "AI-generated ASCII art," it is wrong — rethink the creative concept before shipping.
+
+**Go beyond the reference vocabulary.** The effect catalogs, shader presets, and palette libraries in the references are a starting vocabulary. For every project, combine, modify, and invent new patterns. The catalog is a palette of paints — you write the painting.
+
+**Be proactively creative.** Extend the skill's vocabulary when the project calls for it. If the references don't have what the vision demands, build it. Include at least one visual moment the user didn't ask for but will appreciate — a transition, an effect, a color choice that elevates the whole piece.
+
+**Cohesive aesthetic over technical correctness.** All scenes in a video must feel connected by a unifying visual language — shared color temperature, related character palettes, consistent motion vocabulary. A technically correct video where every scene uses a random different effect is an aesthetic failure.
+
+**Dense, layered, considered.** Every frame should reward viewing. Never flat black backgrounds. Always multi-grid composition. Always per-scene variation. Always intentional color.
+
+## Modes
+
+| Mode | Input | Output | Reference |
+|------|-------|--------|-----------|
+| **Video-to-ASCII** | Video file | ASCII recreation of source footage | `references/inputs.md` § Video Sampling |
+| **Audio-reactive** | Audio file | Generative visuals driven by audio features | `references/inputs.md` § Audio Analysis |
+| **Generative** | None (or seed params) | Procedural ASCII animation | `references/effects.md` |
+| **Hybrid** | Video + audio | ASCII video with audio-reactive overlays | Both input refs |
+| **Lyrics/text** | Audio + text/SRT | Timed text with visual effects | `references/inputs.md` § Text/Lyrics |
+| **TTS narration** | Text quotes + TTS API | Narrated testimonial/quote video with typed text | `references/inputs.md` § TTS Integration |
+
+## Stack
+
+Single self-contained Python script per project. No GPU required.
+
+| Layer | Tool | Purpose |
+|-------|------|---------|
+| Core | Python 3.10+, NumPy | Math, array ops, vectorized effects |
+| Signal | SciPy | FFT, peak detection (audio modes) |
+| Imaging | Pillow (PIL) | Font rasterization, frame decoding, image I/O |
+| Video I/O | ffmpeg (CLI) | Decode input, encode output, mux audio |
+| Parallel | concurrent.futures | N workers for batch/clip rendering |
+| TTS | ElevenLabs API (optional) | Generate narration clips |
+| Optional | OpenCV | Video frame sampling, edge detection |
+
+## Pipeline Architecture
+
+Every mode follows the same 6-stage pipeline:
+
+```
+INPUT → ANALYZE → SCENE_FN → TONEMAP → SHADE → ENCODE
+```
+
+1. **INPUT** — Load/decode source material (video frames, audio samples, images, or nothing)
+2. **ANALYZE** — Extract per-frame features (audio bands, video luminance/edges, motion vectors)
+3. **SCENE_FN** — Scene function renders to pixel canvas (`uint8 H,W,3`). Composes multiple character grids via `_render_vf()` + pixel blend modes. See `references/composition.md`
+4. **TONEMAP** — Percentile-based adaptive brightness normalization. See `references/composition.md` § Adaptive Tonemap
+5. **SHADE** — Post-processing via `ShaderChain` + `FeedbackBuffer`. See `references/shaders.md`
+6. **ENCODE** — Pipe raw RGB frames to ffmpeg for H.264/GIF encoding
+
+## Creative Direction
+
+### Aesthetic Dimensions
+
+| Dimension | Options | Reference |
+|-----------|---------|-----------|
+| **Character palette** | Density ramps, block elements, symbols, scripts (katakana, Greek, runes, braille), project-specific | `architecture.md` § Palettes |
+| **Color strategy** | HSV, OKLAB/OKLCH, discrete RGB palettes, auto-generated harmony, monochrome, temperature | `architecture.md` § Color System |
+| **Background texture** | Sine fields, fBM noise, domain warp, voronoi, reaction-diffusion, cellular automata, video | `effects.md` |
+| **Primary effects** | Rings, spirals, tunnel, vortex, waves, interference, aurora, fire, SDFs, strange attractors | `effects.md` |
+| **Particles** | Sparks, snow, rain, bubbles, runes, orbits, flocking boids, flow-field followers, trails | `effects.md` § Particles |
+| **Shader mood** | Retro CRT, clean modern, glitch art, cinematic, dreamy, industrial, psychedelic | `shaders.md` |
+| **Grid density** | xs(8px) through xxl(40px), mixed per layer | `architecture.md` § Grid System |
+| **Coordinate space** | Cartesian, polar, tiled, rotated, fisheye, Möbius, domain-warped | `effects.md` § Transforms |
+| **Feedback** | Zoom tunnel, rainbow trails, ghostly echo, rotating mandala, color evolution | `composition.md` § Feedback |
+| **Masking** | Circle, ring, gradient, text stencil, animated iris/wipe/dissolve | `composition.md` § Masking |
+| **Transitions** | Crossfade, wipe, dissolve, glitch cut, iris, mask-based reveal | `shaders.md` § Transitions |
+
+### Per-Section Variation
+
+Never use the same config for the entire video. For each section/scene:
+- **Different background effect** (or compose 2-3)
+- **Different character palette** (match the mood)
+- **Different color strategy** (or at minimum a different hue)
+- **Vary shader intensity** (more bloom during peaks, more grain during quiet)
+- **Different particle types** if particles are active
+
+### Project-Specific Invention
+
+For every project, invent at least one of:
+- A custom character palette matching the theme
+- A custom background effect (combine/modify existing building blocks)
+- A custom color palette (discrete RGB set matching the brand/mood)
+- A custom particle character set
+- A novel scene transition or visual moment
+
+Don't just pick from the catalog. The catalog is vocabulary — you write the poem.
+
+## Workflow
+
+### Step 1: Creative Vision
+
+Before any code, articulate the creative concept:
+
+- **Mood/atmosphere**: What should the viewer feel? Energetic, meditative, chaotic, elegant, ominous?
+- **Visual story**: What happens over the duration? Build tension? Transform? Dissolve?
+- **Color world**: Warm/cool? Monochrome? Neon? Earth tones? What's the dominant hue?
+- **Character texture**: Dense data? Sparse stars? Organic dots? Geometric blocks?
+- **What makes THIS different**: What's the one thing that makes this project unique?
+- **Emotional arc**: How do scenes progress? Open with energy, build to climax, resolve?
+
+Map the user's prompt to aesthetic choices. A "chill lo-fi visualizer" demands different everything from a "glitch cyberpunk data stream."
+
+### Step 2: Technical Design
+
+- **Mode** — which of the 6 modes above
+- **Resolution** — landscape 1920x1080 (default), portrait 1080x1920, square 1080x1080 @ 24fps
+- **Hardware detection** — auto-detect cores/RAM, set quality profile. See `references/optimization.md`
+- **Sections** — map timestamps to scene functions, each with its own effect/palette/color/shader config
+- **Output format** — MP4 (default), GIF (640x360 @ 15fps), PNG sequence
+
+### Step 3: Build the Script
+
+Single Python file. Components (with references):
+
+1. **Hardware detection + quality profile** — `references/optimization.md`
+2. **Input loader** — mode-dependent; `references/inputs.md`
+3. **Feature analyzer** — audio FFT, video luminance, or synthetic
+4. **Grid + renderer** — multi-density grids with bitmap cache; `references/architecture.md`
+5. **Character palettes** — multiple per project; `references/architecture.md` § Palettes
+6. **Color system** — HSV + discrete RGB + harmony generation; `references/architecture.md` § Color
+7. **Scene functions** — each returns `canvas (uint8 H,W,3)`; `references/scenes.md`
+8. **Tonemap** — adaptive brightness normalization; `references/composition.md`
+9. **Shader pipeline** — `ShaderChain` + `FeedbackBuffer`; `references/shaders.md`
+10. **Scene table + dispatcher** — time → scene function + config; `references/scenes.md`
+11. **Parallel encoder** — N-worker clip rendering with ffmpeg pipes
+12. **Main** — orchestrate full pipeline
+
+### Step 4: Quality Verification
+
+- **Test frames first**: render single frames at key timestamps before full render
+- **Brightness check**: `canvas.mean() > 8` for all ASCII content. If dark, lower gamma
+- **Visual coherence**: do all scenes feel like they belong to the same video?
+- **Creative vision check**: does the output match the concept from Step 1? If it looks generic, go back
+
+## Critical Implementation Notes
+
+### Brightness — Use `tonemap()`, Not Linear Multipliers
+
+This is the #1 visual issue. ASCII on black is inherently dark. **Never use `canvas * N` multipliers** — they clip highlights. Use adaptive tonemap:
+
+```python
+def tonemap(canvas, gamma=0.75):
+ f = canvas.astype(np.float32)
+ lo, hi = np.percentile(f[::4, ::4], [1, 99.5])
+ if hi - lo < 10: hi = lo + 10
+ f = np.clip((f - lo) / (hi - lo), 0, 1) ** gamma
+ return (f * 255).astype(np.uint8)
+```
+
+Pipeline: `scene_fn() → tonemap() → FeedbackBuffer → ShaderChain → ffmpeg`
+
+Per-scene gamma: default 0.75, solarize 0.55, posterize 0.50, bright scenes 0.85. Use `screen` blend (not `overlay`) for dark layers.
+
+### Font Cell Height
+
+macOS Pillow: `textbbox()` returns wrong height. Use `font.getmetrics()`: `cell_height = ascent + descent`. See `references/troubleshooting.md`.
+
+### ffmpeg Pipe Deadlock
+
+Never `stderr=subprocess.PIPE` with long-running ffmpeg — buffer fills at 64KB and deadlocks. Redirect to file. See `references/troubleshooting.md`.
+
+### Font Compatibility
+
+Not all Unicode chars render in all fonts. Validate palettes at init — render each char, check for blank output. See `references/troubleshooting.md`.
+
+### Per-Clip Architecture
+
+For segmented videos (quotes, scenes, chapters), render each as a separate clip file for parallel rendering and selective re-rendering. See `references/scenes.md`.
+
+## Performance Targets
+
+| Component | Budget |
+|-----------|--------|
+| Feature extraction | 1-5ms |
+| Effect function | 2-15ms |
+| Character render | 80-150ms (bottleneck) |
+| Shader pipeline | 5-25ms |
+| **Total** | ~100-200ms/frame |
+
+## References
+
+| File | Contents |
+|------|----------|
+| `references/architecture.md` | Grid system, resolution presets, font selection, character palettes (20+), color system (HSV + OKLAB + discrete RGB + harmony generation), `_render_vf()` helper, GridLayer class |
+| `references/composition.md` | Pixel blend modes (20 modes), `blend_canvas()`, multi-grid composition, adaptive `tonemap()`, `FeedbackBuffer`, `PixelBlendStack`, masking/stencil system |
+| `references/effects.md` | Effect building blocks: value field generators, hue fields, noise/fBM/domain warp, voronoi, reaction-diffusion, cellular automata, SDFs, strange attractors, particle systems, coordinate transforms, temporal coherence |
+| `references/shaders.md` | `ShaderChain`, `_apply_shader_step()` dispatch, 38 shader catalog, audio-reactive scaling, transitions, tint presets, output format encoding, terminal rendering |
+| `references/scenes.md` | Scene protocol, `Renderer` class, `SCENES` table, `render_clip()`, beat-synced cutting, parallel rendering, design patterns (layer hierarchy, directional arcs, visual metaphors, compositional techniques), complete scene examples at every complexity level, scene design checklist |
+| `references/inputs.md` | Audio analysis (FFT, bands, beats), video sampling, image conversion, text/lyrics, TTS integration (ElevenLabs, voice assignment, audio mixing) |
+| `references/optimization.md` | Hardware detection, quality profiles, vectorized patterns, parallel rendering, memory management, performance budgets |
+| `references/troubleshooting.md` | NumPy broadcasting traps, blend mode pitfalls, multiprocessing/pickling, brightness diagnostics, ffmpeg issues, font problems, common mistakes |
+
+---
+
+## Creative Divergence (use only when user requests experimental/creative/unique output)
+
+If the user asks for creative, experimental, surprising, or unconventional output, select the strategy that best fits and reason through its steps BEFORE generating code.
+
+- **Forced Connections** — when the user wants cross-domain inspiration ("make it look organic," "industrial aesthetic")
+- **Conceptual Blending** — when the user names two things to combine ("ocean meets music," "space + calligraphy")
+- **Oblique Strategies** — when the user is maximally open ("surprise me," "something I've never seen")
+
+### Forced Connections
+1. Pick a domain unrelated to the visual goal (weather systems, microbiology, architecture, fluid dynamics, textile weaving)
+2. List its core visual/structural elements (erosion → gradual reveal; mitosis → splitting duplication; weaving → interlocking patterns)
+3. Map those elements onto ASCII characters and animation patterns
+4. Synthesize — what does "erosion" or "crystallization" look like in a character grid?
+
+### Conceptual Blending
+1. Name two distinct visual/conceptual spaces (e.g., ocean waves + sheet music)
+2. Map correspondences (crests = high notes, troughs = rests, foam = staccato)
+3. Blend selectively — keep the most interesting mappings, discard forced ones
+4. Develop emergent properties that exist only in the blend
+
+### Oblique Strategies
+1. Draw one: "Honor thy error as a hidden intention" / "Use an old idea" / "What would your closest friend do?" / "Emphasize the flaws" / "Turn it upside down" / "Only a part, not the whole" / "Reverse"
+2. Interpret the directive against the current ASCII animation challenge
+3. Apply the lateral insight to the visual design before writing code
diff --git a/src/skills/bundled/ascii-video/references/architecture.md b/src/skills/bundled/ascii-video/references/architecture.md
new file mode 100644
index 0000000..16a15ae
--- /dev/null
+++ b/src/skills/bundled/ascii-video/references/architecture.md
@@ -0,0 +1,802 @@
+# Architecture Reference
+
+> **See also:** composition.md · effects.md · scenes.md · shaders.md · inputs.md · optimization.md · troubleshooting.md
+
+## Grid System
+
+### Resolution Presets
+
+```python
+RESOLUTION_PRESETS = {
+ "landscape": (1920, 1080), # 16:9 — YouTube, default
+ "portrait": (1080, 1920), # 9:16 — TikTok, Reels, Stories
+ "square": (1080, 1080), # 1:1 — Instagram feed
+ "ultrawide": (2560, 1080), # 21:9 — cinematic
+ "landscape4k":(3840, 2160), # 16:9 — 4K
+ "portrait4k": (2160, 3840), # 9:16 — 4K portrait
+}
+
+def get_resolution(preset="landscape", custom=None):
+ """Returns (VW, VH) tuple."""
+ if custom:
+ return custom
+ return RESOLUTION_PRESETS.get(preset, RESOLUTION_PRESETS["landscape"])
+```
+
+### Multi-Density Grids
+
+Pre-initialize multiple grid sizes. Switch per section for visual variety. Grid dimensions auto-compute from resolution:
+
+**Landscape (1920x1080):**
+
+| Key | Font Size | Grid (cols x rows) | Use |
+|-----|-----------|-------------------|-----|
+| xs | 8 | 400x108 | Ultra-dense data fields |
+| sm | 10 | 320x83 | Dense detail, rain, starfields |
+| md | 16 | 192x56 | Default balanced, transitions |
+| lg | 20 | 160x45 | Quote/lyric text (readable at 1080p) |
+| xl | 24 | 137x37 | Short quotes, large titles |
+| xxl | 40 | 80x22 | Giant text, minimal |
+
+**Portrait (1080x1920):**
+
+| Key | Font Size | Grid (cols x rows) | Use |
+|-----|-----------|-------------------|-----|
+| xs | 8 | 225x192 | Ultra-dense, tall data columns |
+| sm | 10 | 180x148 | Dense detail, vertical rain |
+| md | 16 | 112x100 | Default balanced |
+| lg | 20 | 90x80 | Readable text (~30 chars/line centered) |
+| xl | 24 | 75x66 | Short quotes, stacked |
+| xxl | 40 | 45x39 | Giant text, minimal |
+
+**Square (1080x1080):**
+
+| Key | Font Size | Grid (cols x rows) | Use |
+|-----|-----------|-------------------|-----|
+| sm | 10 | 180x83 | Dense detail |
+| md | 16 | 112x56 | Default balanced |
+| lg | 20 | 90x45 | Readable text |
+
+**Key differences in portrait mode:**
+- Fewer columns (90 at `lg` vs 160) — lines must be shorter or wrap
+- Many more rows (80 at `lg` vs 45) — vertical stacking is natural
+- Aspect ratio correction flips: `asp = cw / ch` still works but the visual emphasis is vertical
+- Radial effects appear as tall ellipses unless corrected
+- Vertical effects (rain, embers, fire columns) are naturally enhanced
+- Horizontal effects (spectrum bars, waveforms) need rotation or compression
+
+**Grid sizing for text in portrait**: Use `lg` (20px) for 2-3 word lines. Max comfortable line length is ~25-30 chars. For longer quotes, break aggressively into many short lines stacked vertically — portrait has vertical space to spare. `xl` (24px) works for single words or very short phrases.
+
+Grid dimensions: `cols = VW // cell_width`, `rows = VH // cell_height`.
+
+### Font Selection
+
+Don't hardcode a single font. Choose fonts to match the project's mood. Monospace fonts are required for grid alignment but vary widely in personality:
+
+| Font | Personality | Platform |
+|------|-------------|----------|
+| Menlo | Clean, neutral, Apple-native | macOS |
+| Monaco | Retro terminal, compact | macOS |
+| Courier New | Classic typewriter, wide | Cross-platform |
+| SF Mono | Modern, tight spacing | macOS |
+| Consolas | Windows native, clean | Windows |
+| JetBrains Mono | Developer, ligature-ready | Install |
+| Fira Code | Geometric, modern | Install |
+| IBM Plex Mono | Corporate, authoritative | Install |
+| Source Code Pro | Adobe, balanced | Install |
+
+**Font detection at init**: probe available fonts and fall back gracefully:
+
+```python
+import platform
+
+def find_font(preferences):
+ """Try fonts in order, return first that exists."""
+ for name, path in preferences:
+ if os.path.exists(path):
+ return path
+ raise FileNotFoundError(f"No monospace font found. Tried: {[p for _,p in preferences]}")
+
+FONT_PREFS_MACOS = [
+ ("Menlo", "/System/Library/Fonts/Menlo.ttc"),
+ ("Monaco", "/System/Library/Fonts/Monaco.ttf"),
+ ("SF Mono", "/System/Library/Fonts/SFNSMono.ttf"),
+ ("Courier", "/System/Library/Fonts/Courier.ttc"),
+]
+FONT_PREFS_LINUX = [
+ ("DejaVu Sans Mono", "/usr/share/fonts/truetype/dejavu/DejaVuSansMono.ttf"),
+ ("Liberation Mono", "/usr/share/fonts/truetype/liberation/LiberationMono-Regular.ttf"),
+ ("Noto Sans Mono", "/usr/share/fonts/truetype/noto/NotoSansMono-Regular.ttf"),
+ ("Ubuntu Mono", "/usr/share/fonts/truetype/ubuntu/UbuntuMono-R.ttf"),
+]
+FONT_PREFS_WINDOWS = [
+ ("Consolas", r"C:\Windows\Fonts\consola.ttf"),
+ ("Courier New", r"C:\Windows\Fonts\cour.ttf"),
+ ("Lucida Console", r"C:\Windows\Fonts\lucon.ttf"),
+ ("Cascadia Code", os.path.expandvars(r"%LOCALAPPDATA%\Microsoft\Windows\Fonts\CascadiaCode.ttf")),
+ ("Cascadia Mono", os.path.expandvars(r"%LOCALAPPDATA%\Microsoft\Windows\Fonts\CascadiaMono.ttf")),
+]
+
+def _get_font_prefs():
+ s = platform.system()
+ if s == "Darwin":
+ return FONT_PREFS_MACOS
+ elif s == "Windows":
+ return FONT_PREFS_WINDOWS
+ return FONT_PREFS_LINUX
+
+FONT_PREFS = _get_font_prefs()
+```
+
+**Multi-font rendering**: use different fonts for different layers (e.g., monospace for background, a bolder variant for overlay text). Each GridLayer owns its own font:
+
+```python
+grid_bg = GridLayer(find_font(FONT_PREFS), 16) # background
+grid_text = GridLayer(find_font(BOLD_PREFS), 20) # readable text
+```
+
+### Collecting All Characters
+
+Before initializing grids, gather all characters that need bitmap pre-rasterization:
+
+```python
+all_chars = set()
+for pal in [PAL_DEFAULT, PAL_DENSE, PAL_BLOCKS, PAL_RUNE, PAL_KATA,
+ PAL_GREEK, PAL_MATH, PAL_DOTS, PAL_BRAILLE, PAL_STARS,
+ PAL_HALFFILL, PAL_HATCH, PAL_BINARY, PAL_MUSIC, PAL_BOX,
+ PAL_CIRCUIT, PAL_ARROWS, PAL_HERMES]: # ... all palettes used in project
+ all_chars.update(pal)
+# Add any overlay text characters
+all_chars.update("ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789 .,-:;!?/|")
+all_chars.discard(" ") # space is never rendered
+```
+
+### GridLayer Initialization
+
+Each grid pre-computes coordinate arrays for vectorized effect math. The grid automatically adapts to any resolution (landscape, portrait, square):
+
+```python
+class GridLayer:
+ def __init__(self, font_path, font_size, vw=None, vh=None):
+ """Initialize grid for any resolution.
+ vw, vh: video width/height in pixels. Defaults to global VW, VH."""
+ vw = vw or VW; vh = vh or VH
+ self.vw = vw; self.vh = vh
+
+ self.font = ImageFont.truetype(font_path, font_size)
+ asc, desc = self.font.getmetrics()
+ bbox = self.font.getbbox("M")
+ self.cw = bbox[2] - bbox[0] # character cell width
+ self.ch = asc + desc # CRITICAL: not textbbox height
+
+ self.cols = vw // self.cw
+ self.rows = vh // self.ch
+ self.ox = (vw - self.cols * self.cw) // 2 # centering
+ self.oy = (vh - self.rows * self.ch) // 2
+
+ # Aspect ratio metadata
+ self.aspect = vw / vh # >1 = landscape, <1 = portrait, 1 = square
+ self.is_portrait = vw < vh
+ self.is_landscape = vw > vh
+
+ # Index arrays
+ self.rr = np.arange(self.rows, dtype=np.float32)[:, None]
+ self.cc = np.arange(self.cols, dtype=np.float32)[None, :]
+
+ # Polar coordinates (aspect-corrected)
+ cx, cy = self.cols / 2.0, self.rows / 2.0
+ asp = self.cw / self.ch
+ self.dx = self.cc - cx
+ self.dy = (self.rr - cy) * asp
+ self.dist = np.sqrt(self.dx**2 + self.dy**2)
+ self.angle = np.arctan2(self.dy, self.dx)
+
+ # Normalized (0-1 range) -- for distance falloff
+ self.dx_n = (self.cc - cx) / max(self.cols, 1)
+ self.dy_n = (self.rr - cy) / max(self.rows, 1) * asp
+ self.dist_n = np.sqrt(self.dx_n**2 + self.dy_n**2)
+
+ # Pre-rasterize all characters to float32 bitmaps
+ self.bm = {}
+ for c in all_chars:
+ img = Image.new("L", (self.cw, self.ch), 0)
+ ImageDraw.Draw(img).text((0, 0), c, fill=255, font=self.font)
+ self.bm[c] = np.array(img, dtype=np.float32) / 255.0
+```
+
+### Character Render Loop
+
+The bottleneck. Composites pre-rasterized bitmaps onto pixel canvas:
+
+```python
+def render(self, chars, colors, canvas=None):
+ if canvas is None:
+ canvas = np.zeros((VH, VW, 3), dtype=np.uint8)
+ for row in range(self.rows):
+ y = self.oy + row * self.ch
+ if y + self.ch > VH: break
+ for col in range(self.cols):
+ c = chars[row, col]
+ if c == " ": continue
+ x = self.ox + col * self.cw
+ if x + self.cw > VW: break
+ a = self.bm[c] # float32 bitmap
+ canvas[y:y+self.ch, x:x+self.cw] = np.maximum(
+ canvas[y:y+self.ch, x:x+self.cw],
+ (a[:, :, None] * colors[row, col]).astype(np.uint8))
+ return canvas
+```
+
+Use `np.maximum` for additive blending (brighter chars overwrite dimmer ones, never darken).
+
+### Multi-Layer Rendering
+
+Render multiple grids onto the same canvas for depth:
+
+```python
+canvas = np.zeros((VH, VW, 3), dtype=np.uint8)
+canvas = grid_lg.render(bg_chars, bg_colors, canvas) # background layer
+canvas = grid_md.render(main_chars, main_colors, canvas) # main layer
+canvas = grid_sm.render(detail_chars, detail_colors, canvas) # detail overlay
+```
+
+---
+
+## Character Palettes
+
+### Design Principles
+
+Character palettes are the primary visual texture of ASCII video. They control not just brightness mapping but the entire visual feel. Design palettes intentionally:
+
+- **Visual weight**: characters sorted by the amount of ink/pixels they fill. Space is always index 0.
+- **Coherence**: characters within a palette should belong to the same visual family.
+- **Density curve**: the brightness-to-character mapping is nonlinear. Dense palettes (many chars) give smoother gradients; sparse palettes (5-8 chars) give posterized/graphic looks.
+- **Rendering compatibility**: every character in the palette must exist in the font. Test at init and remove missing glyphs.
+
+### Palette Library
+
+Organized by visual family. Mix and match per project -- don't default to PAL_DEFAULT for everything.
+
+#### Density / Brightness Palettes
+```python
+PAL_DEFAULT = " .`'-:;!><=+*^~?/|(){}[]#&$@%" # classic ASCII art
+PAL_DENSE = " .:;+=xX$#@\u2588" # simple 11-level ramp
+PAL_MINIMAL = " .:-=+#@" # 8-level, graphic
+PAL_BINARY = " \u2588" # 2-level, extreme contrast
+PAL_GRADIENT = " \u2591\u2592\u2593\u2588" # 4-level block gradient
+```
+
+#### Unicode Block Elements
+```python
+PAL_BLOCKS = " \u2591\u2592\u2593\u2588\u2584\u2580\u2590\u258c" # standard blocks
+PAL_BLOCKS_EXT = " \u2596\u2597\u2598\u2599\u259a\u259b\u259c\u259d\u259e\u259f\u2591\u2592\u2593\u2588" # quadrant blocks (more detail)
+PAL_SHADE = " \u2591\u2592\u2593\u2588\u2587\u2586\u2585\u2584\u2583\u2582\u2581" # vertical fill progression
+```
+
+#### Symbolic / Thematic
+```python
+PAL_MATH = " \u00b7\u2218\u2219\u2022\u00b0\u00b1\u2213\u00d7\u00f7\u2248\u2260\u2261\u2264\u2265\u221e\u222b\u2211\u220f\u221a\u2207\u2202\u2206\u03a9" # math symbols
+PAL_BOX = " \u2500\u2502\u250c\u2510\u2514\u2518\u251c\u2524\u252c\u2534\u253c\u2550\u2551\u2554\u2557\u255a\u255d\u2560\u2563\u2566\u2569\u256c" # box drawing
+PAL_CIRCUIT = " .\u00b7\u2500\u2502\u250c\u2510\u2514\u2518\u253c\u25cb\u25cf\u25a1\u25a0\u2206\u2207\u2261" # circuit board
+PAL_RUNE = " .\u16a0\u16a2\u16a6\u16b1\u16b7\u16c1\u16c7\u16d2\u16d6\u16da\u16de\u16df" # elder futhark runes
+PAL_ALCHEMIC = " \u2609\u263d\u2640\u2642\u2643\u2644\u2645\u2646\u2647\u2648\u2649\u264a\u264b" # planetary/alchemical symbols
+PAL_ZODIAC = " \u2648\u2649\u264a\u264b\u264c\u264d\u264e\u264f\u2650\u2651\u2652\u2653" # zodiac
+PAL_ARROWS = " \u2190\u2191\u2192\u2193\u2194\u2195\u2196\u2197\u2198\u2199\u21a9\u21aa\u21bb\u27a1" # directional arrows
+PAL_MUSIC = " \u266a\u266b\u266c\u2669\u266d\u266e\u266f\u25cb\u25cf" # musical notation
+```
+
+#### Script / Writing System
+```python
+PAL_KATA = " \u00b7\uff66\uff67\uff68\uff69\uff6a\uff6b\uff6c\uff6d\uff6e\uff6f\uff70\uff71\uff72\uff73\uff74\uff75\uff76\uff77" # katakana halfwidth (matrix rain)
+PAL_GREEK = " \u03b1\u03b2\u03b3\u03b4\u03b5\u03b6\u03b7\u03b8\u03b9\u03ba\u03bb\u03bc\u03bd\u03be\u03c0\u03c1\u03c3\u03c4\u03c6\u03c8\u03c9" # Greek lowercase
+PAL_CYRILLIC = " \u0430\u0431\u0432\u0433\u0434\u0435\u0436\u0437\u0438\u043a\u043b\u043c\u043d\u043e\u043f\u0440\u0441\u0442\u0443\u0444\u0445\u0446\u0447\u0448" # Cyrillic lowercase
+PAL_ARABIC = " \u0627\u0628\u062a\u062b\u062c\u062d\u062e\u062f\u0630\u0631\u0632\u0633\u0634\u0635\u0636\u0637" # Arabic letters (isolated forms)
+```
+
+#### Dot / Point Progressions
+```python
+PAL_DOTS = " ⋅∘∙●◉◎◆✦★" # dot size progression
+PAL_BRAILLE = " ⠁⠂⠃⠄⠅⠆⠇⠈⠉⠊⠋⠌⠍⠎⠏⠐⠑⠒⠓⠔⠕⠖⠗⠘⠙⠚⠛⠜⠝⠞⠟⠿" # braille patterns
+PAL_STARS = " ·✧✦✩✨★✶✳✸" # star progression
+PAL_HALFFILL = " ◔◑◕◐◒◓◖◗◙" # directional half-fill progression
+PAL_HATCH = " ▣▤▥▦▧▨▩" # crosshatch density ramp
+```
+
+#### Project-Specific (examples -- invent new ones per project)
+```python
+PAL_HERMES = " .\u00b7~=\u2248\u221e\u26a1\u263f\u2726\u2605\u2295\u25ca\u25c6\u25b2\u25bc\u25cf\u25a0" # mythology/tech blend
+PAL_OCEAN = " ~\u2248\u2248\u2248\u223c\u2307\u2248\u224b\u224c\u2248" # water/wave characters
+PAL_ORGANIC = " .\u00b0\u2218\u2022\u25e6\u25c9\u2742\u273f\u2741\u2743" # growing/botanical
+PAL_MACHINE = " _\u2500\u2502\u250c\u2510\u253c\u2261\u25a0\u2588\u2593\u2592\u2591" # mechanical/industrial
+```
+
+### Creating Custom Palettes
+
+When designing for a project, build palettes from the content's theme:
+
+1. **Choose a visual family** (dots, blocks, symbols, script)
+2. **Sort by visual weight** -- render each char at target font size, count lit pixels, sort ascending
+3. **Test at target grid size** -- some chars collapse to blobs at small sizes
+4. **Validate in font** -- remove chars the font can't render:
+
+```python
+def validate_palette(pal, font):
+ """Remove characters the font can't render."""
+ valid = []
+ for c in pal:
+ if c == " ":
+ valid.append(c)
+ continue
+ img = Image.new("L", (20, 20), 0)
+ ImageDraw.Draw(img).text((0, 0), c, fill=255, font=font)
+ if np.array(img).max() > 0: # char actually rendered something
+ valid.append(c)
+ return "".join(valid)
+```
+
+### Mapping Values to Characters
+
+```python
+def val2char(v, mask, pal=PAL_DEFAULT):
+ """Map float array (0-1) to character array using palette."""
+ n = len(pal)
+ idx = np.clip((v * n).astype(int), 0, n - 1)
+ out = np.full(v.shape, " ", dtype="U1")
+ for i, ch in enumerate(pal):
+ out[mask & (idx == i)] = ch
+ return out
+```
+
+**Nonlinear mapping** for different visual curves:
+
+```python
+def val2char_gamma(v, mask, pal, gamma=1.0):
+ """Gamma-corrected palette mapping. gamma<1 = brighter, gamma>1 = darker."""
+ v_adj = np.power(np.clip(v, 0, 1), gamma)
+ return val2char(v_adj, mask, pal)
+
+def val2char_step(v, mask, pal, thresholds):
+ """Custom threshold mapping. thresholds = list of float breakpoints."""
+ out = np.full(v.shape, pal[0], dtype="U1")
+ for i, thr in enumerate(thresholds):
+ out[mask & (v > thr)] = pal[min(i + 1, len(pal) - 1)]
+ return out
+```
+
+---
+
+## Color System
+
+### HSV->RGB (Vectorized)
+
+All color computation in HSV for intuitive control, converted at render time:
+
+```python
+def hsv2rgb(h, s, v):
+ """Vectorized HSV->RGB. h,s,v are numpy arrays. Returns (R,G,B) uint8 arrays."""
+ h = h % 1.0
+ c = v * s; x = c * (1 - np.abs((h*6) % 2 - 1)); m = v - c
+ # ... 6 sector assignment ...
+ return (np.clip((r+m)*255, 0, 255).astype(np.uint8),
+ np.clip((g+m)*255, 0, 255).astype(np.uint8),
+ np.clip((b+m)*255, 0, 255).astype(np.uint8))
+```
+
+### Color Mapping Strategies
+
+Don't default to a single strategy. Choose based on the visual intent:
+
+| Strategy | Hue source | Effect | Good for |
+|----------|------------|--------|----------|
+| Angle-mapped | `g.angle / (2*pi)` | Rainbow around center | Radial effects, kaleidoscopes |
+| Distance-mapped | `g.dist_n * 0.3` | Gradient from center | Tunnels, depth effects |
+| Frequency-mapped | `f["cent"] * 0.2` | Timbral color shifting | Audio-reactive |
+| Value-mapped | `val * 0.15` | Brightness-dependent hue | Fire, heat maps |
+| Time-cycled | `t * rate` | Slow color rotation | Ambient, chill |
+| Source-sampled | Video frame pixel colors | Preserve original color | Video-to-ASCII |
+| Palette-indexed | Discrete color lookup | Flat graphic style | Retro, pixel art |
+| Temperature | Blend between warm/cool | Emotional tone | Mood-driven scenes |
+| Complementary | `hue` and `hue + 0.5` | High contrast | Bold, dramatic |
+| Triadic | `hue`, `hue + 0.33`, `hue + 0.66` | Vibrant, balanced | Psychedelic |
+| Analogous | `hue +/- 0.08` | Harmonious, subtle | Elegant, cohesive |
+| Monochrome | Fixed hue, vary S and V | Restrained, focused | Noir, minimal |
+
+### Color Palettes (Discrete RGB)
+
+For non-HSV workflows -- direct RGB color sets for graphic/retro looks:
+
+```python
+# Named color palettes -- use for flat/graphic styles or per-character coloring
+COLORS_NEON = [(255,0,102), (0,255,153), (102,0,255), (255,255,0), (0,204,255)]
+COLORS_PASTEL = [(255,179,186), (255,223,186), (255,255,186), (186,255,201), (186,225,255)]
+COLORS_MONO_GREEN = [(0,40,0), (0,80,0), (0,140,0), (0,200,0), (0,255,0)]
+COLORS_MONO_AMBER = [(40,20,0), (80,50,0), (140,90,0), (200,140,0), (255,191,0)]
+COLORS_CYBERPUNK = [(255,0,60), (0,255,200), (180,0,255), (255,200,0)]
+COLORS_VAPORWAVE = [(255,113,206), (1,205,254), (185,103,255), (5,255,161)]
+COLORS_EARTH = [(86,58,26), (139,90,43), (189,154,91), (222,193,136), (245,230,193)]
+COLORS_ICE = [(200,230,255), (150,200,240), (100,170,230), (60,130,210), (30,80,180)]
+COLORS_BLOOD = [(80,0,0), (140,10,10), (200,20,20), (255,50,30), (255,100,80)]
+COLORS_FOREST = [(10,30,10), (20,60,15), (30,100,20), (50,150,30), (80,200,50)]
+
+def rgb_palette_map(val, mask, palette):
+ """Map float array (0-1) to RGB colors from a discrete palette."""
+ n = len(palette)
+ idx = np.clip((val * n).astype(int), 0, n - 1)
+ R = np.zeros(val.shape, dtype=np.uint8)
+ G = np.zeros(val.shape, dtype=np.uint8)
+ B = np.zeros(val.shape, dtype=np.uint8)
+ for i, (r, g, b) in enumerate(palette):
+ m = mask & (idx == i)
+ R[m] = r; G[m] = g; B[m] = b
+ return R, G, B
+```
+
+### OKLAB Color Space (Perceptually Uniform)
+
+HSV hue is perceptually non-uniform: green occupies far more visual range than blue. OKLAB / OKLCH provide perceptually even color steps — hue increments of 0.1 look equally different regardless of starting hue. Use OKLAB for:
+- Gradient interpolation (no unwanted intermediate hues)
+- Color harmony generation (perceptually balanced palettes)
+- Smooth color transitions over time
+
+```python
+# --- sRGB <-> Linear sRGB ---
+
+def srgb_to_linear(c):
+ """Convert sRGB [0,1] to linear light. c: float32 array."""
+ return np.where(c <= 0.04045, c / 12.92, ((c + 0.055) / 1.055) ** 2.4)
+
+def linear_to_srgb(c):
+ """Convert linear light to sRGB [0,1]."""
+ return np.where(c <= 0.0031308, c * 12.92, 1.055 * np.power(np.maximum(c, 0), 1/2.4) - 0.055)
+
+# --- Linear sRGB <-> OKLAB ---
+
+def linear_rgb_to_oklab(r, g, b):
+ """Linear sRGB to OKLAB. r,g,b: float32 arrays [0,1].
+ Returns (L, a, b) where L=[0,1], a,b=[-0.4, 0.4] approx."""
+ l_ = 0.4122214708 * r + 0.5363325363 * g + 0.0514459929 * b
+ m_ = 0.2119034982 * r + 0.6806995451 * g + 0.1073969566 * b
+ s_ = 0.0883024619 * r + 0.2817188376 * g + 0.6299787005 * b
+ l_c = np.cbrt(l_); m_c = np.cbrt(m_); s_c = np.cbrt(s_)
+ L = 0.2104542553 * l_c + 0.7936177850 * m_c - 0.0040720468 * s_c
+ a = 1.9779984951 * l_c - 2.4285922050 * m_c + 0.4505937099 * s_c
+ b_ = 0.0259040371 * l_c + 0.7827717662 * m_c - 0.8086757660 * s_c
+ return L, a, b_
+
+def oklab_to_linear_rgb(L, a, b):
+ """OKLAB to linear sRGB. Returns (r, g, b) float32 arrays [0,1]."""
+ l_ = L + 0.3963377774 * a + 0.2158037573 * b
+ m_ = L - 0.1055613458 * a - 0.0638541728 * b
+ s_ = L - 0.0894841775 * a - 1.2914855480 * b
+ l_c = l_ ** 3; m_c = m_ ** 3; s_c = s_ ** 3
+ r = +4.0767416621 * l_c - 3.3077115913 * m_c + 0.2309699292 * s_c
+ g = -1.2684380046 * l_c + 2.6097574011 * m_c - 0.3413193965 * s_c
+ b_ = -0.0041960863 * l_c - 0.7034186147 * m_c + 1.7076147010 * s_c
+ return np.clip(r, 0, 1), np.clip(g, 0, 1), np.clip(b_, 0, 1)
+
+# --- Convenience: sRGB uint8 <-> OKLAB ---
+
+def rgb_to_oklab(R, G, B):
+ """sRGB uint8 arrays to OKLAB."""
+ r = srgb_to_linear(R.astype(np.float32) / 255.0)
+ g = srgb_to_linear(G.astype(np.float32) / 255.0)
+ b = srgb_to_linear(B.astype(np.float32) / 255.0)
+ return linear_rgb_to_oklab(r, g, b)
+
+def oklab_to_rgb(L, a, b):
+ """OKLAB to sRGB uint8 arrays."""
+ r, g, b_ = oklab_to_linear_rgb(L, a, b)
+ R = np.clip(linear_to_srgb(r) * 255, 0, 255).astype(np.uint8)
+ G = np.clip(linear_to_srgb(g) * 255, 0, 255).astype(np.uint8)
+ B = np.clip(linear_to_srgb(b_) * 255, 0, 255).astype(np.uint8)
+ return R, G, B
+
+# --- OKLCH (cylindrical form of OKLAB) ---
+
+def oklab_to_oklch(L, a, b):
+ """OKLAB to OKLCH. Returns (L, C, H) where H is in [0, 1] (normalized)."""
+ C = np.sqrt(a**2 + b**2)
+ H = (np.arctan2(b, a) / (2 * np.pi)) % 1.0
+ return L, C, H
+
+def oklch_to_oklab(L, C, H):
+ """OKLCH to OKLAB. H in [0, 1]."""
+ angle = H * 2 * np.pi
+ a = C * np.cos(angle)
+ b = C * np.sin(angle)
+ return L, a, b
+```
+
+### Gradient Interpolation (OKLAB vs HSV)
+
+Interpolating colors through OKLAB avoids the hue detours that HSV produces:
+
+```python
+def lerp_oklab(color_a, color_b, t_array):
+ """Interpolate between two sRGB colors through OKLAB.
+ color_a, color_b: (R, G, B) tuples 0-255
+ t_array: float32 array [0,1] — interpolation parameter per pixel.
+ Returns (R, G, B) uint8 arrays."""
+ La, aa, ba = rgb_to_oklab(
+ np.full_like(t_array, color_a[0], dtype=np.uint8),
+ np.full_like(t_array, color_a[1], dtype=np.uint8),
+ np.full_like(t_array, color_a[2], dtype=np.uint8))
+ Lb, ab, bb = rgb_to_oklab(
+ np.full_like(t_array, color_b[0], dtype=np.uint8),
+ np.full_like(t_array, color_b[1], dtype=np.uint8),
+ np.full_like(t_array, color_b[2], dtype=np.uint8))
+ L = La + (Lb - La) * t_array
+ a = aa + (ab - aa) * t_array
+ b = ba + (bb - ba) * t_array
+ return oklab_to_rgb(L, a, b)
+
+def lerp_oklch(color_a, color_b, t_array, short_path=True):
+ """Interpolate through OKLCH (preserves chroma, smooth hue path).
+ short_path: take the shorter arc around the hue wheel."""
+ La, aa, ba = rgb_to_oklab(
+ np.full_like(t_array, color_a[0], dtype=np.uint8),
+ np.full_like(t_array, color_a[1], dtype=np.uint8),
+ np.full_like(t_array, color_a[2], dtype=np.uint8))
+ Lb, ab, bb = rgb_to_oklab(
+ np.full_like(t_array, color_b[0], dtype=np.uint8),
+ np.full_like(t_array, color_b[1], dtype=np.uint8),
+ np.full_like(t_array, color_b[2], dtype=np.uint8))
+ L1, C1, H1 = oklab_to_oklch(La, aa, ba)
+ L2, C2, H2 = oklab_to_oklch(Lb, ab, bb)
+ # Shortest hue path
+ if short_path:
+ dh = H2 - H1
+ dh = np.where(dh > 0.5, dh - 1.0, np.where(dh < -0.5, dh + 1.0, dh))
+ H = (H1 + dh * t_array) % 1.0
+ else:
+ H = H1 + (H2 - H1) * t_array
+ L = L1 + (L2 - L1) * t_array
+ C = C1 + (C2 - C1) * t_array
+ Lout, aout, bout = oklch_to_oklab(L, C, H)
+ return oklab_to_rgb(Lout, aout, bout)
+```
+
+### Color Harmony Generation
+
+Auto-generate harmonious palettes from a seed color:
+
+```python
+def harmony_complementary(seed_rgb):
+ """Two colors: seed + opposite hue."""
+ L, a, b = rgb_to_oklab(np.array([seed_rgb[0]]), np.array([seed_rgb[1]]), np.array([seed_rgb[2]]))
+ _, C, H = oklab_to_oklch(L, a, b)
+ return [seed_rgb, _oklch_to_srgb_tuple(L[0], C[0], (H[0] + 0.5) % 1.0)]
+
+def harmony_triadic(seed_rgb):
+ """Three colors: seed + two at 120-degree offsets."""
+ L, a, b = rgb_to_oklab(np.array([seed_rgb[0]]), np.array([seed_rgb[1]]), np.array([seed_rgb[2]]))
+ _, C, H = oklab_to_oklch(L, a, b)
+ return [seed_rgb,
+ _oklch_to_srgb_tuple(L[0], C[0], (H[0] + 0.333) % 1.0),
+ _oklch_to_srgb_tuple(L[0], C[0], (H[0] + 0.667) % 1.0)]
+
+def harmony_analogous(seed_rgb, spread=0.08, n=5):
+ """N colors spread evenly around seed hue."""
+ L, a, b = rgb_to_oklab(np.array([seed_rgb[0]]), np.array([seed_rgb[1]]), np.array([seed_rgb[2]]))
+ _, C, H = oklab_to_oklch(L, a, b)
+ offsets = np.linspace(-spread * (n-1)/2, spread * (n-1)/2, n)
+ return [_oklch_to_srgb_tuple(L[0], C[0], (H[0] + off) % 1.0) for off in offsets]
+
+def harmony_split_complementary(seed_rgb, split=0.08):
+ """Three colors: seed + two flanking the complement."""
+ L, a, b = rgb_to_oklab(np.array([seed_rgb[0]]), np.array([seed_rgb[1]]), np.array([seed_rgb[2]]))
+ _, C, H = oklab_to_oklch(L, a, b)
+ comp = (H[0] + 0.5) % 1.0
+ return [seed_rgb,
+ _oklch_to_srgb_tuple(L[0], C[0], (comp - split) % 1.0),
+ _oklch_to_srgb_tuple(L[0], C[0], (comp + split) % 1.0)]
+
+def harmony_tetradic(seed_rgb):
+ """Four colors: two complementary pairs at 90-degree offset."""
+ L, a, b = rgb_to_oklab(np.array([seed_rgb[0]]), np.array([seed_rgb[1]]), np.array([seed_rgb[2]]))
+ _, C, H = oklab_to_oklch(L, a, b)
+ return [seed_rgb,
+ _oklch_to_srgb_tuple(L[0], C[0], (H[0] + 0.25) % 1.0),
+ _oklch_to_srgb_tuple(L[0], C[0], (H[0] + 0.5) % 1.0),
+ _oklch_to_srgb_tuple(L[0], C[0], (H[0] + 0.75) % 1.0)]
+
+def _oklch_to_srgb_tuple(L, C, H):
+ """Helper: single OKLCH -> sRGB (R,G,B) int tuple."""
+ La = np.array([L]); Ca = np.array([C]); Ha = np.array([H])
+ Lo, ao, bo = oklch_to_oklab(La, Ca, Ha)
+ R, G, B = oklab_to_rgb(Lo, ao, bo)
+ return (int(R[0]), int(G[0]), int(B[0]))
+```
+
+### OKLAB Hue Fields
+
+Drop-in replacements for `hf_*` generators that produce perceptually uniform hue variation:
+
+```python
+def hf_oklch_angle(offset=0.0, chroma=0.12, lightness=0.7):
+ """OKLCH hue mapped to angle from center. Perceptually uniform rainbow.
+ Returns (R, G, B) uint8 color array instead of a float hue.
+ NOTE: Use with _render_vf_rgb() variant, not standard _render_vf()."""
+ def fn(g, f, t, S):
+ H = (g.angle / (2 * np.pi) + offset + t * 0.05) % 1.0
+ L = np.full_like(H, lightness)
+ C = np.full_like(H, chroma)
+ Lo, ao, bo = oklch_to_oklab(L, C, H)
+ R, G, B = oklab_to_rgb(Lo, ao, bo)
+ return mkc(R, G, B, g.rows, g.cols)
+ return fn
+```
+
+### Compositing Helpers
+
+```python
+def mkc(R, G, B, rows, cols):
+ """Pack 3 uint8 arrays into (rows, cols, 3) color array."""
+ o = np.zeros((rows, cols, 3), dtype=np.uint8)
+ o[:,:,0] = R; o[:,:,1] = G; o[:,:,2] = B
+ return o
+
+def layer_over(base_ch, base_co, top_ch, top_co):
+ """Composite top layer onto base. Non-space chars overwrite."""
+ m = top_ch != " "
+ base_ch[m] = top_ch[m]; base_co[m] = top_co[m]
+ return base_ch, base_co
+
+def layer_blend(base_co, top_co, alpha):
+ """Alpha-blend top color layer onto base. alpha is float array (0-1) or scalar."""
+ if isinstance(alpha, (int, float)):
+ alpha = np.full(base_co.shape[:2], alpha, dtype=np.float32)
+ a = alpha[:,:,None]
+ return np.clip(base_co * (1 - a) + top_co * a, 0, 255).astype(np.uint8)
+
+def stamp(ch, co, text, row, col, color=(255,255,255)):
+ """Write text string at position."""
+ for i, c in enumerate(text):
+ cc = col + i
+ if 0 <= row < ch.shape[0] and 0 <= cc < ch.shape[1]:
+ ch[row, cc] = c; co[row, cc] = color
+```
+
+---
+
+## Section System
+
+Map time ranges to effect functions + shader configs + grid sizes:
+
+```python
+SECTIONS = [
+ (0.0, "void"), (3.94, "starfield"), (21.0, "matrix"),
+ (46.0, "drop"), (130.0, "glitch"), (187.0, "outro"),
+]
+
+FX_DISPATCH = {"void": fx_void, "starfield": fx_starfield, ...}
+SECTION_FX = {"void": {"vignette": 0.3, "bloom": 170}, ...}
+SECTION_GRID = {"void": "md", "starfield": "sm", "drop": "lg", ...}
+SECTION_MIRROR = {"drop": "h", "bass_rings": "quad"}
+
+def get_section(t):
+ sec = SECTIONS[0][1]
+ for ts, name in SECTIONS:
+ if t >= ts: sec = name
+ return sec
+```
+
+---
+
+## Parallel Encoding
+
+Split frames across N workers. Each pipes raw RGB to its own ffmpeg subprocess:
+
+```python
+def render_batch(batch_id, frame_start, frame_end, features, seg_path):
+ r = Renderer()
+ cmd = ["ffmpeg", "-y", "-f", "rawvideo", "-pix_fmt", "rgb24",
+ "-s", f"{VW}x{VH}", "-r", str(FPS), "-i", "pipe:0",
+ "-c:v", "libx264", "-preset", "fast", "-crf", "18",
+ "-pix_fmt", "yuv420p", seg_path]
+
+ # CRITICAL: stderr to file, not pipe
+ stderr_fh = open(os.path.join(workdir, f"err_{batch_id:02d}.log"), "w")
+ pipe = subprocess.Popen(cmd, stdin=subprocess.PIPE,
+ stdout=subprocess.DEVNULL, stderr=stderr_fh)
+
+ for fi in range(frame_start, frame_end):
+ t = fi / FPS
+ sec = get_section(t)
+ f = {k: float(features[k][fi]) for k in features}
+ ch, co = FX_DISPATCH[sec](r, f, t)
+ canvas = r.render(ch, co)
+ canvas = apply_mirror(canvas, sec, f)
+ canvas = apply_shaders(canvas, sec, f, t)
+ pipe.stdin.write(canvas.tobytes())
+
+ pipe.stdin.close()
+ pipe.wait()
+ stderr_fh.close()
+```
+
+Concatenate segments + mux audio:
+
+```python
+# Write concat file
+with open(concat_path, "w") as cf:
+ for seg in segments:
+ cf.write(f"file '{seg}'\n")
+
+subprocess.run(["ffmpeg", "-y", "-f", "concat", "-safe", "0", "-i", concat_path,
+ "-i", audio_path, "-c:v", "copy", "-c:a", "aac", "-b:a", "192k",
+ "-shortest", output_path])
+```
+
+## Effect Function Contract
+
+### v2 Protocol (Current)
+
+Every scene function: `(r, f, t, S) -> canvas_uint8` — where `r` = Renderer, `f` = features dict, `t` = time float, `S` = persistent state dict
+
+```python
+def fx_example(r, f, t, S):
+ """Scene function returns a full pixel canvas (uint8 H,W,3).
+ Scenes have full control over multi-grid rendering and pixel-level composition.
+ """
+ # Render multiple layers at different grid densities
+ canvas_a = _render_vf(r, "md", vf_plasma, hf_angle(0.0), PAL_DENSE, f, t, S)
+ canvas_b = _render_vf(r, "sm", vf_vortex, hf_time_cycle(0.1), PAL_RUNE, f, t, S)
+
+ # Pixel-level blend
+ result = blend_canvas(canvas_a, canvas_b, "screen", 0.8)
+ return result
+```
+
+See `references/scenes.md` for the full scene protocol, the Renderer class, `_render_vf()` helper, and complete scene examples.
+
+See `references/composition.md` for blend modes, tone mapping, feedback buffers, and multi-grid composition.
+
+### v1 Protocol (Legacy)
+
+Simple scenes that use a single grid can still return `(chars, colors)` and let the caller handle rendering, but the v2 canvas protocol is preferred for all new code.
+
+```python
+def fx_simple(r, f, t, S):
+ g = r.get_grid("md")
+ val = np.sin(g.dist * 0.1 - t * 3) * f.get("bass", 0.3) * 2
+ val = np.clip(val, 0, 1); mask = val > 0.03
+ ch = val2char(val, mask, PAL_DEFAULT)
+ R, G, B = hsv2rgb(np.full_like(val, 0.6), np.full_like(val, 0.7), val)
+ co = mkc(R, G, B, g.rows, g.cols)
+ return g.render(ch, co) # returns canvas directly
+```
+
+### Persistent State
+
+Effects that need state across frames (particles, rain columns) use the `S` dict parameter (which is `r.S` — same object, but passed explicitly for clarity):
+
+```python
+def fx_with_state(r, f, t, S):
+ if "particles" not in S:
+ S["particles"] = initialize_particles()
+ update_particles(S["particles"])
+ # ...
+```
+
+State persists across frames within a single scene/clip. Each worker process (and each scene) gets its own independent state.
+
+### Helper Functions
+
+```python
+def hsv2rgb_scalar(h, s, v):
+ """Single-value HSV to RGB. Returns (R, G, B) tuple of ints 0-255."""
+ h = h % 1.0
+ c = v * s; x = c * (1 - abs((h * 6) % 2 - 1)); m = v - c
+ if h * 6 < 1: r, g, b = c, x, 0
+ elif h * 6 < 2: r, g, b = x, c, 0
+ elif h * 6 < 3: r, g, b = 0, c, x
+ elif h * 6 < 4: r, g, b = 0, x, c
+ elif h * 6 < 5: r, g, b = x, 0, c
+ else: r, g, b = c, 0, x
+ return (int((r+m)*255), int((g+m)*255), int((b+m)*255))
+
+def log(msg):
+ """Print timestamped log message."""
+ print(msg, flush=True)
+```
diff --git a/src/skills/bundled/ascii-video/references/composition.md b/src/skills/bundled/ascii-video/references/composition.md
new file mode 100644
index 0000000..f7e6eff
--- /dev/null
+++ b/src/skills/bundled/ascii-video/references/composition.md
@@ -0,0 +1,892 @@
+# Composition & Brightness Reference
+
+The composable system is the core of visual complexity. It operates at three levels: pixel-level blend modes, multi-grid composition, and adaptive brightness management. This document covers all three, plus the masking/stencil system for spatial control.
+
+> **See also:** architecture.md · effects.md · scenes.md · shaders.md · troubleshooting.md
+
+## Pixel-Level Blend Modes
+
+### The `blend_canvas()` Function
+
+All blending operates on full pixel canvases (`uint8 H,W,3`). Internally converts to float32 [0,1] for precision, blends, lerps by opacity, converts back.
+
+```python
+def blend_canvas(base, top, mode="normal", opacity=1.0):
+ af = base.astype(np.float32) / 255.0
+ bf = top.astype(np.float32) / 255.0
+ fn = BLEND_MODES.get(mode, BLEND_MODES["normal"])
+ result = fn(af, bf)
+ if opacity < 1.0:
+ result = af * (1 - opacity) + result * opacity
+ return np.clip(result * 255, 0, 255).astype(np.uint8)
+```
+
+### 20 Blend Modes
+
+```python
+BLEND_MODES = {
+ # Basic arithmetic
+ "normal": lambda a, b: b,
+ "add": lambda a, b: np.clip(a + b, 0, 1),
+ "subtract": lambda a, b: np.clip(a - b, 0, 1),
+ "multiply": lambda a, b: a * b,
+ "screen": lambda a, b: 1 - (1 - a) * (1 - b),
+
+ # Contrast
+ "overlay": lambda a, b: np.where(a < 0.5, 2*a*b, 1 - 2*(1-a)*(1-b)),
+ "softlight": lambda a, b: (1 - 2*b)*a*a + 2*b*a,
+ "hardlight": lambda a, b: np.where(b < 0.5, 2*a*b, 1 - 2*(1-a)*(1-b)),
+
+ # Difference
+ "difference": lambda a, b: np.abs(a - b),
+ "exclusion": lambda a, b: a + b - 2*a*b,
+
+ # Dodge / burn
+ "colordodge": lambda a, b: np.clip(a / (1 - b + 1e-6), 0, 1),
+ "colorburn": lambda a, b: np.clip(1 - (1 - a) / (b + 1e-6), 0, 1),
+
+ # Light
+ "linearlight": lambda a, b: np.clip(a + 2*b - 1, 0, 1),
+ "vividlight": lambda a, b: np.where(b < 0.5,
+ np.clip(1 - (1-a)/(2*b + 1e-6), 0, 1),
+ np.clip(a / (2*(1-b) + 1e-6), 0, 1)),
+ "pin_light": lambda a, b: np.where(b < 0.5,
+ np.minimum(a, 2*b), np.maximum(a, 2*b - 1)),
+ "hard_mix": lambda a, b: np.where(a + b >= 1.0, 1.0, 0.0),
+
+ # Compare
+ "lighten": lambda a, b: np.maximum(a, b),
+ "darken": lambda a, b: np.minimum(a, b),
+
+ # Grain
+ "grain_extract": lambda a, b: np.clip(a - b + 0.5, 0, 1),
+ "grain_merge": lambda a, b: np.clip(a + b - 0.5, 0, 1),
+}
+```
+
+### Blend Mode Selection Guide
+
+**Modes that brighten** (safe for dark inputs):
+- `screen` — always brightens. Two 50% gray layers screen to 75%. The go-to safe blend.
+- `add` — simple addition, clips at white. Good for sparkles, glows, particle overlays.
+- `colordodge` — extreme brightening at overlap zones. Can blow out. Use low opacity (0.3-0.5).
+- `linearlight` — aggressive brightening. Similar to add but with offset.
+
+**Modes that darken** (avoid with dark inputs):
+- `multiply` — darkens everything. Only use when both layers are already bright.
+- `overlay` — darkens when base < 0.5, brightens when base > 0.5. Crushes dark inputs: `2 * 0.12 * 0.12 = 0.03`. Use `screen` instead for dark material.
+- `colorburn` — extreme darkening at overlap zones.
+
+**Modes that create contrast**:
+- `softlight` — gentle contrast. Good for subtle texture overlay.
+- `hardlight` — strong contrast. Like overlay but keyed on the top layer.
+- `vividlight` — very aggressive contrast. Use sparingly.
+
+**Modes that create color effects**:
+- `difference` — XOR-like patterns. Two identical layers difference to black; offset layers create wild colors. Great for psychedelic looks.
+- `exclusion` — softer version of difference. Creates complementary color patterns.
+- `hard_mix` — posterizes to pure black/white/saturated color at intersections.
+
+**Modes for texture blending**:
+- `grain_extract` / `grain_merge` — extract a texture from one layer, apply it to another.
+
+### Multi-Layer Chaining
+
+```python
+# Pattern: render layers -> blend sequentially
+canvas_a = _render_vf(r, "md", vf_plasma, hf_angle(0.0), PAL_DENSE, f, t, S)
+canvas_b = _render_vf(r, "sm", vf_vortex, hf_time_cycle(0.1), PAL_RUNE, f, t, S)
+canvas_c = _render_vf(r, "lg", vf_rings, hf_distance(), PAL_BLOCKS, f, t, S)
+
+result = blend_canvas(canvas_a, canvas_b, "screen", 0.8)
+result = blend_canvas(result, canvas_c, "difference", 0.6)
+```
+
+Order matters: `screen(A, B)` is commutative, but `difference(screen(A,B), C)` differs from `difference(A, screen(B,C))`.
+
+### Linear-Light Blend Modes
+
+Standard `blend_canvas()` operates in sRGB space — the raw byte values. This is fine for most uses, but sRGB is perceptually non-linear: blending in sRGB darkens midtones and shifts hues slightly. For physically accurate blending (matching how light actually combines), convert to linear light first.
+
+Uses `srgb_to_linear()` / `linear_to_srgb()` from `architecture.md` § OKLAB Color System.
+
+```python
+def blend_canvas_linear(base, top, mode="normal", opacity=1.0):
+ """Blend in linear light space for physically accurate results.
+
+ Identical API to blend_canvas(), but converts sRGB → linear before
+ blending and linear → sRGB after. More expensive (~2x) due to the
+ gamma conversions, but produces correct results for additive blending,
+ screen, and any mode where brightness matters.
+ """
+ af = srgb_to_linear(base.astype(np.float32) / 255.0)
+ bf = srgb_to_linear(top.astype(np.float32) / 255.0)
+ fn = BLEND_MODES.get(mode, BLEND_MODES["normal"])
+ result = fn(af, bf)
+ if opacity < 1.0:
+ result = af * (1 - opacity) + result * opacity
+ result = linear_to_srgb(np.clip(result, 0, 1))
+ return np.clip(result * 255, 0, 255).astype(np.uint8)
+```
+
+**When to use `blend_canvas_linear()` vs `blend_canvas()`:**
+
+| Scenario | Use | Why |
+|----------|-----|-----|
+| Screen-blending two bright layers | `linear` | sRGB screen over-brightens highlights |
+| Add mode for glow/bloom effects | `linear` | Additive light follows linear physics |
+| Blending text overlay at low opacity | `srgb` | Perceptual blending looks more natural for text |
+| Multiply for shadow/darkening | `srgb` | Differences are minimal for darken ops |
+| Color-critical work (matching reference) | `linear` | Avoids sRGB hue shifts in midtones |
+| Performance-critical inner loop | `srgb` | ~2x faster, good enough for most ASCII art |
+
+**Batch version** for compositing many layers (converts once, blends multiple, converts back):
+
+```python
+def blend_many_linear(layers, modes, opacities):
+ """Blend a stack of layers in linear light space.
+
+ Args:
+ layers: list of uint8 (H,W,3) canvases
+ modes: list of blend mode strings (len = len(layers) - 1)
+ opacities: list of floats (len = len(layers) - 1)
+ Returns:
+ uint8 (H,W,3) canvas
+ """
+ # Convert all to linear at once
+ linear = [srgb_to_linear(l.astype(np.float32) / 255.0) for l in layers]
+ result = linear[0]
+ for i in range(1, len(linear)):
+ fn = BLEND_MODES.get(modes[i-1], BLEND_MODES["normal"])
+ blended = fn(result, linear[i])
+ op = opacities[i-1]
+ if op < 1.0:
+ blended = result * (1 - op) + blended * op
+ result = np.clip(blended, 0, 1)
+ result = linear_to_srgb(result)
+ return np.clip(result * 255, 0, 255).astype(np.uint8)
+```
+
+---
+
+## Multi-Grid Composition
+
+This is the core visual technique. Rendering the same conceptual scene at different grid densities (character sizes) creates natural texture interference, because characters at different scales overlap at different spatial frequencies.
+
+### Why It Works
+
+- `sm` grid (10pt font): 320x83 characters. Fine detail, dense texture.
+- `md` grid (16pt): 192x56 characters. Medium density.
+- `lg` grid (20pt): 160x45 characters. Coarse, chunky characters.
+
+When you render a plasma field on `sm` and a vortex on `lg`, then screen-blend them, the fine plasma texture shows through the gaps in the coarse vortex characters. The result has more visual complexity than either layer alone.
+
+### The `_render_vf()` Helper
+
+This is the workhorse function. It takes a value field + hue field + palette + grid, renders to a complete pixel canvas:
+
+```python
+def _render_vf(r, grid_key, val_fn, hue_fn, pal, f, t, S, sat=0.8, threshold=0.03):
+ """Render a value field + hue field to a pixel canvas via a named grid.
+
+ Args:
+ r: Renderer instance (has .get_grid())
+ grid_key: "xs", "sm", "md", "lg", "xl", "xxl"
+ val_fn: (g, f, t, S) -> float32 [0,1] array (rows, cols)
+ hue_fn: callable (g, f, t, S) -> float32 hue array, OR float scalar
+ pal: character palette string
+ f: feature dict
+ t: time in seconds
+ S: persistent state dict
+ sat: HSV saturation (0-1)
+ threshold: minimum value to render (below = space)
+
+ Returns:
+ uint8 array (VH, VW, 3) — full pixel canvas
+ """
+ g = r.get_grid(grid_key)
+ val = np.clip(val_fn(g, f, t, S), 0, 1)
+ mask = val > threshold
+ ch = val2char(val, mask, pal)
+
+ # Hue: either a callable or a fixed float
+ if callable(hue_fn):
+ h = hue_fn(g, f, t, S) % 1.0
+ else:
+ h = np.full((g.rows, g.cols), float(hue_fn), dtype=np.float32)
+
+ # CRITICAL: broadcast to full shape and copy (see Troubleshooting)
+ h = np.broadcast_to(h, (g.rows, g.cols)).copy()
+
+ R, G, B = hsv2rgb(h, np.full_like(val, sat), val)
+ co = mkc(R, G, B, g.rows, g.cols)
+ return g.render(ch, co)
+```
+
+### Grid Combination Strategies
+
+| Combination | Effect | Good For |
+|-------------|--------|----------|
+| `sm` + `lg` | Maximum contrast between fine detail and chunky blocks | Bold, graphic looks |
+| `sm` + `md` | Subtle texture layering, similar scales | Organic, flowing looks |
+| `md` + `lg` + `xs` | Three-scale interference, maximum complexity | Psychedelic, dense |
+| `sm` + `sm` (different effects) | Same scale, pattern interference only | Moire, interference |
+
+### Complete Multi-Grid Scene Example
+
+```python
+def fx_psychedelic(r, f, t, S):
+ """Three-layer multi-grid scene with beat-reactive kaleidoscope."""
+ # Layer A: plasma on medium grid with rainbow hue
+ canvas_a = _render_vf(r, "md",
+ lambda g, f, t, S: vf_plasma(g, f, t, S) * 1.3,
+ hf_angle(0.0), PAL_DENSE, f, t, S, sat=0.8)
+
+ # Layer B: vortex on small grid with cycling hue
+ canvas_b = _render_vf(r, "sm",
+ lambda g, f, t, S: vf_vortex(g, f, t, S, twist=5.0) * 1.2,
+ hf_time_cycle(0.1), PAL_RUNE, f, t, S, sat=0.7)
+
+ # Layer C: rings on large grid with distance hue
+ canvas_c = _render_vf(r, "lg",
+ lambda g, f, t, S: vf_rings(g, f, t, S, n_base=8, spacing_base=3) * 1.4,
+ hf_distance(0.3, 0.02), PAL_BLOCKS, f, t, S, sat=0.9)
+
+ # Blend: A screened with B, then difference with C
+ result = blend_canvas(canvas_a, canvas_b, "screen", 0.8)
+ result = blend_canvas(result, canvas_c, "difference", 0.6)
+
+ # Beat-triggered kaleidoscope
+ if f.get("bdecay", 0) > 0.3:
+ result = sh_kaleidoscope(result.copy(), folds=6)
+
+ return result
+```
+
+---
+
+## Adaptive Tone Mapping
+
+### The Brightness Problem
+
+ASCII characters are small bright dots on a black background. Most pixels in any frame are background (black). This means:
+- Mean frame brightness is inherently low (often 5-30 out of 255)
+- Different effect combinations produce wildly different brightness levels
+- A spiral scene might be 50 mean, while a fire scene is 9 mean
+- Linear multipliers (e.g., `canvas * 2.0`) either leave dark scenes dark or blow out bright scenes
+
+### The `tonemap()` Function
+
+Replaces linear brightness multipliers with adaptive per-frame normalization + gamma correction:
+
+```python
+def tonemap(canvas, target_mean=90, gamma=0.75, black_point=2, white_point=253):
+ """Adaptive tone-mapping: normalizes + gamma-corrects so no frame is
+ fully dark or washed out.
+
+ 1. Compute 1st and 99.5th percentile on 4x subsample (16x fewer values,
+ negligible accuracy loss, major speedup at 1080p+)
+ 2. Stretch that range to [0, 1]
+ 3. Apply gamma curve (< 1 lifts shadows, > 1 darkens)
+ 4. Rescale to [black_point, white_point]
+ """
+ f = canvas.astype(np.float32)
+ sub = f[::4, ::4] # 4x subsample: ~390K values vs ~6.2M at 1080p
+ lo = np.percentile(sub, 1)
+ hi = np.percentile(sub, 99.5)
+ if hi - lo < 10:
+ hi = max(hi, lo + 10) # near-uniform frame fallback
+ f = np.clip((f - lo) / (hi - lo), 0.0, 1.0)
+ np.power(f, gamma, out=f) # in-place: avoids allocation
+ np.multiply(f, (white_point - black_point), out=f)
+ np.add(f, black_point, out=f)
+ return np.clip(f, 0, 255).astype(np.uint8)
+```
+
+### Why Gamma, Not Linear
+
+Linear multiplier `* 2.0`:
+```
+input 10 -> output 20 (still dark)
+input 100 -> output 200 (ok)
+input 200 -> output 255 (clipped, lost detail)
+```
+
+Gamma 0.75 after normalization:
+```
+input 0.04 -> output 0.08 (lifted from invisible to visible)
+input 0.39 -> output 0.50 (moderate lift)
+input 0.78 -> output 0.84 (gentle lift, no clipping)
+```
+
+Gamma < 1 compresses the highlights and expands the shadows. This is exactly what we need: lift dark ASCII content into visibility without blowing out the bright parts.
+
+### Pipeline Ordering
+
+The pipeline in `render_clip()` is:
+
+```
+scene_fn(r, f, t, S) -> canvas
+ |
+ tonemap(canvas, gamma=scene_gamma)
+ |
+ FeedbackBuffer.apply(canvas, ...)
+ |
+ ShaderChain.apply(canvas, f=f, t=t)
+ |
+ ffmpeg pipe
+```
+
+Tonemap runs BEFORE feedback and shaders. This means:
+- Feedback operates on normalized data (consistent behavior regardless of scene brightness)
+- Shaders like solarize, posterize, contrast operate on properly-ranged data
+- The brightness shader in the chain is no longer needed (tonemap handles it)
+
+### Per-Scene Gamma Tuning
+
+Default gamma is 0.75. Scenes that apply destructive post-processing need more aggressive lift because the destruction happens after tonemap:
+
+| Scene Type | Recommended Gamma | Why |
+|------------|-------------------|-----|
+| Standard effects | 0.75 | Default, works for most scenes |
+| Solarize post-process | 0.50-0.60 | Solarize inverts bright pixels, reducing overall brightness |
+| Posterize post-process | 0.50-0.55 | Posterize quantizes, often crushing mid-values to black |
+| Heavy difference blending | 0.60-0.70 | Difference mode creates many near-zero pixels |
+| Already bright scenes | 0.85-1.0 | Don't over-boost scenes that are naturally bright |
+
+Configure via the scene table:
+
+```python
+SCENES = [
+ {"start": 9.17, "end": 11.25, "name": "fire", "gamma": 0.55,
+ "fx": fx_fire, "shaders": [("solarize", {"threshold": 200}), ...]},
+ {"start": 25.96, "end": 27.29, "name": "diamond", "gamma": 0.5,
+ "fx": fx_diamond, "shaders": [("bloom", {"thr": 90}), ...]},
+]
+```
+
+### Brightness Verification
+
+After rendering, spot-check frame brightness:
+
+```python
+# In test-frame mode
+canvas = scene["fx"](r, feat, t, r.S)
+canvas = tonemap(canvas, gamma=scene.get("gamma", 0.75))
+chain = ShaderChain()
+for sn, kw in scene.get("shaders", []):
+ chain.add(sn, **kw)
+canvas = chain.apply(canvas, f=feat, t=t)
+print(f"Mean brightness: {canvas.astype(float).mean():.1f}, max: {canvas.max()}")
+```
+
+Target ranges after tonemap + shaders:
+- Quiet/ambient scenes: mean 30-60
+- Active scenes: mean 40-100
+- Climax/peak scenes: mean 60-150
+- If mean < 20: gamma is too high or a shader is destroying brightness
+- If mean > 180: gamma is too low or add is stacking too much
+
+---
+
+## FeedbackBuffer Spatial Transforms
+
+The feedback buffer stores the previous frame and blends it into the current frame with decay. Spatial transforms applied to the buffer before blending create the illusion of motion in the feedback trail.
+
+### Implementation
+
+```python
+class FeedbackBuffer:
+ def __init__(self):
+ self.buf = None
+
+ def apply(self, canvas, decay=0.85, blend="screen", opacity=0.5,
+ transform=None, transform_amt=0.02, hue_shift=0.0):
+ if self.buf is None:
+ self.buf = canvas.astype(np.float32) / 255.0
+ return canvas
+
+ # Decay old buffer
+ self.buf *= decay
+
+ # Spatial transform
+ if transform:
+ self.buf = self._transform(self.buf, transform, transform_amt)
+
+ # Hue shift the feedback for rainbow trails
+ if hue_shift > 0:
+ self.buf = self._hue_shift(self.buf, hue_shift)
+
+ # Blend feedback into current frame
+ result = blend_canvas(canvas,
+ np.clip(self.buf * 255, 0, 255).astype(np.uint8),
+ blend, opacity)
+
+ # Update buffer with current frame
+ self.buf = result.astype(np.float32) / 255.0
+ return result
+
+ def _transform(self, buf, transform, amt):
+ h, w = buf.shape[:2]
+ if transform == "zoom":
+ # Zoom in: sample from slightly inside (creates expanding tunnel)
+ m = int(h * amt); n = int(w * amt)
+ if m > 0 and n > 0:
+ cropped = buf[m:-m or None, n:-n or None]
+ # Resize back to full (nearest-neighbor for speed)
+ buf = np.array(Image.fromarray(
+ np.clip(cropped * 255, 0, 255).astype(np.uint8)
+ ).resize((w, h), Image.NEAREST)).astype(np.float32) / 255.0
+ elif transform == "shrink":
+ # Zoom out: pad edges, shrink center
+ m = int(h * amt); n = int(w * amt)
+ small = np.array(Image.fromarray(
+ np.clip(buf * 255, 0, 255).astype(np.uint8)
+ ).resize((w - 2*n, h - 2*m), Image.NEAREST))
+ new = np.zeros((h, w, 3), dtype=np.uint8)
+ new[m:m+small.shape[0], n:n+small.shape[1]] = small
+ buf = new.astype(np.float32) / 255.0
+ elif transform == "rotate_cw":
+ # Small clockwise rotation via affine
+ angle = amt * 10 # amt=0.005 -> 0.05 degrees per frame
+ cy, cx = h / 2, w / 2
+ Y = np.arange(h, dtype=np.float32)[:, None]
+ X = np.arange(w, dtype=np.float32)[None, :]
+ cos_a, sin_a = np.cos(angle), np.sin(angle)
+ sx = (X - cx) * cos_a + (Y - cy) * sin_a + cx
+ sy = -(X - cx) * sin_a + (Y - cy) * cos_a + cy
+ sx = np.clip(sx.astype(int), 0, w - 1)
+ sy = np.clip(sy.astype(int), 0, h - 1)
+ buf = buf[sy, sx]
+ elif transform == "rotate_ccw":
+ angle = -amt * 10
+ cy, cx = h / 2, w / 2
+ Y = np.arange(h, dtype=np.float32)[:, None]
+ X = np.arange(w, dtype=np.float32)[None, :]
+ cos_a, sin_a = np.cos(angle), np.sin(angle)
+ sx = (X - cx) * cos_a + (Y - cy) * sin_a + cx
+ sy = -(X - cx) * sin_a + (Y - cy) * cos_a + cy
+ sx = np.clip(sx.astype(int), 0, w - 1)
+ sy = np.clip(sy.astype(int), 0, h - 1)
+ buf = buf[sy, sx]
+ elif transform == "shift_up":
+ pixels = max(1, int(h * amt))
+ buf = np.roll(buf, -pixels, axis=0)
+ buf[-pixels:] = 0 # black fill at bottom
+ elif transform == "shift_down":
+ pixels = max(1, int(h * amt))
+ buf = np.roll(buf, pixels, axis=0)
+ buf[:pixels] = 0
+ elif transform == "mirror_h":
+ buf = buf[:, ::-1]
+ return buf
+
+ def _hue_shift(self, buf, amount):
+ """Rotate hues of the feedback buffer. Operates on float32 [0,1]."""
+ rgb = np.clip(buf * 255, 0, 255).astype(np.uint8)
+ hsv = np.zeros_like(buf)
+ # Simple approximate RGB->HSV->shift->RGB
+ r, g, b = buf[:,:,0], buf[:,:,1], buf[:,:,2]
+ mx = np.maximum(np.maximum(r, g), b)
+ mn = np.minimum(np.minimum(r, g), b)
+ delta = mx - mn + 1e-10
+ # Hue
+ h = np.where(mx == r, ((g - b) / delta) % 6,
+ np.where(mx == g, (b - r) / delta + 2, (r - g) / delta + 4))
+ h = (h / 6 + amount) % 1.0
+ # Reconstruct with shifted hue (simplified)
+ s = delta / (mx + 1e-10)
+ v = mx
+ c = v * s; x = c * (1 - np.abs((h * 6) % 2 - 1)); m = v - c
+ ro = np.zeros_like(h); go = np.zeros_like(h); bo = np.zeros_like(h)
+ for lo, hi, rv, gv, bv in [(0,1,c,x,0),(1,2,x,c,0),(2,3,0,c,x),
+ (3,4,0,x,c),(4,5,x,0,c),(5,6,c,0,x)]:
+ mask = ((h*6) >= lo) & ((h*6) < hi)
+ ro[mask] = rv[mask] if not isinstance(rv, (int,float)) else rv
+ go[mask] = gv[mask] if not isinstance(gv, (int,float)) else gv
+ bo[mask] = bv[mask] if not isinstance(bv, (int,float)) else bv
+ return np.stack([ro+m, go+m, bo+m], axis=2)
+```
+
+### Feedback Presets
+
+| Preset | Config | Visual Effect |
+|--------|--------|---------------|
+| Infinite zoom tunnel | `decay=0.8, blend="screen", transform="zoom", transform_amt=0.015` | Expanding ring patterns |
+| Rainbow trails | `decay=0.7, blend="screen", transform="zoom", transform_amt=0.01, hue_shift=0.02` | Psychedelic color trails |
+| Ghostly echo | `decay=0.9, blend="add", opacity=0.15, transform="shift_up", transform_amt=0.01` | Faint upward smearing |
+| Kaleidoscopic recursion | `decay=0.75, blend="screen", transform="rotate_cw", transform_amt=0.005, hue_shift=0.01` | Rotating mandala feedback |
+| Color evolution | `decay=0.8, blend="difference", opacity=0.4, hue_shift=0.03` | Frame-to-frame color XOR |
+| Rising heat haze | `decay=0.5, blend="add", opacity=0.2, transform="shift_up", transform_amt=0.02` | Hot air shimmer |
+
+---
+
+## Masking / Stencil System
+
+Masks are float32 arrays `(rows, cols)` or `(VH, VW)` in range [0, 1]. They control where effects are visible: 1.0 = fully visible, 0.0 = fully hidden. Use masks to create figure/ground relationships, focal points, and shaped reveals.
+
+### Shape Masks
+
+```python
+def mask_circle(g, cx_frac=0.5, cy_frac=0.5, radius=0.3, feather=0.05):
+ """Circular mask centered at (cx_frac, cy_frac) in normalized coords.
+ feather: width of soft edge (0 = hard cutoff)."""
+ asp = g.cw / g.ch if hasattr(g, 'cw') else 1.0
+ dx = (g.cc / g.cols - cx_frac)
+ dy = (g.rr / g.rows - cy_frac) * asp
+ d = np.sqrt(dx**2 + dy**2)
+ if feather > 0:
+ return np.clip(1.0 - (d - radius) / feather, 0, 1)
+ return (d <= radius).astype(np.float32)
+
+def mask_rect(g, x0=0.2, y0=0.2, x1=0.8, y1=0.8, feather=0.03):
+ """Rectangular mask. Coordinates in [0,1] normalized."""
+ dx = np.maximum(x0 - g.cc / g.cols, g.cc / g.cols - x1)
+ dy = np.maximum(y0 - g.rr / g.rows, g.rr / g.rows - y1)
+ d = np.maximum(dx, dy)
+ if feather > 0:
+ return np.clip(1.0 - d / feather, 0, 1)
+ return (d <= 0).astype(np.float32)
+
+def mask_ring(g, cx_frac=0.5, cy_frac=0.5, inner_r=0.15, outer_r=0.35,
+ feather=0.03):
+ """Ring / annulus mask."""
+ inner = mask_circle(g, cx_frac, cy_frac, inner_r, feather)
+ outer = mask_circle(g, cx_frac, cy_frac, outer_r, feather)
+ return outer - inner
+
+def mask_gradient_h(g, start=0.0, end=1.0):
+ """Left-to-right gradient mask."""
+ return np.clip((g.cc / g.cols - start) / (end - start + 1e-10), 0, 1).astype(np.float32)
+
+def mask_gradient_v(g, start=0.0, end=1.0):
+ """Top-to-bottom gradient mask."""
+ return np.clip((g.rr / g.rows - start) / (end - start + 1e-10), 0, 1).astype(np.float32)
+
+def mask_gradient_radial(g, cx_frac=0.5, cy_frac=0.5, inner=0.0, outer=0.5):
+ """Radial gradient mask — bright at center, dark at edges."""
+ d = np.sqrt((g.cc / g.cols - cx_frac)**2 + (g.rr / g.rows - cy_frac)**2)
+ return np.clip(1.0 - (d - inner) / (outer - inner + 1e-10), 0, 1)
+```
+
+### Value Field as Mask
+
+Use any `vf_*` function's output as a spatial mask:
+
+```python
+def mask_from_vf(vf_result, threshold=0.5, feather=0.1):
+ """Convert a value field to a mask by thresholding.
+ feather: smooth edge width around threshold."""
+ if feather > 0:
+ return np.clip((vf_result - threshold + feather) / (2 * feather), 0, 1)
+ return (vf_result > threshold).astype(np.float32)
+
+def mask_select(mask, vf_a, vf_b):
+ """Spatial conditional: show vf_a where mask is 1, vf_b where mask is 0.
+ mask: float32 [0,1] array. Intermediate values blend."""
+ return vf_a * mask + vf_b * (1 - mask)
+```
+
+### Text Stencil
+
+Render text to a mask. Effects are visible only through the letterforms:
+
+```python
+def mask_text(grid, text, row_frac=0.5, font=None, font_size=None):
+ """Render text string as a float32 mask [0,1] at grid resolution.
+ Characters = 1.0, background = 0.0.
+
+ row_frac: vertical position as fraction of grid height.
+ font: PIL ImageFont (defaults to grid's font if None).
+ font_size: override font size for the mask text (for larger stencil text).
+ """
+ from PIL import Image, ImageDraw, ImageFont
+
+ f = font or grid.font
+ if font_size and font != grid.font:
+ f = ImageFont.truetype(font.path, font_size)
+
+ # Render text to image at pixel resolution, then downsample to grid
+ img = Image.new("L", (grid.cols * grid.cw, grid.ch), 0)
+ draw = ImageDraw.Draw(img)
+ bbox = draw.textbbox((0, 0), text, font=f)
+ tw = bbox[2] - bbox[0]
+ x = (grid.cols * grid.cw - tw) // 2
+ draw.text((x, 0), text, fill=255, font=f)
+ row_mask = np.array(img, dtype=np.float32) / 255.0
+
+ # Place in full grid mask
+ mask = np.zeros((grid.rows, grid.cols), dtype=np.float32)
+ target_row = int(grid.rows * row_frac)
+ # Downsample rendered text to grid cells
+ for c in range(grid.cols):
+ px = c * grid.cw
+ if px + grid.cw <= row_mask.shape[1]:
+ cell = row_mask[:, px:px + grid.cw]
+ if cell.mean() > 0.1:
+ mask[target_row, c] = cell.mean()
+ return mask
+
+def mask_text_block(grid, lines, start_row_frac=0.3, font=None):
+ """Multi-line text stencil. Returns full grid mask."""
+ mask = np.zeros((grid.rows, grid.cols), dtype=np.float32)
+ for i, line in enumerate(lines):
+ row_frac = start_row_frac + i / grid.rows
+ line_mask = mask_text(grid, line, row_frac, font)
+ mask = np.maximum(mask, line_mask)
+ return mask
+```
+
+### Animated Masks
+
+Masks that change over time for reveals, wipes, and morphing:
+
+```python
+def mask_iris(g, t, t_start, t_end, cx_frac=0.5, cy_frac=0.5,
+ max_radius=0.7, ease_fn=None):
+ """Iris open/close: circle that grows from 0 to max_radius.
+ ease_fn: easing function (default: ease_in_out_cubic from effects.md)."""
+ if ease_fn is None:
+ ease_fn = lambda x: x * x * (3 - 2 * x) # smoothstep fallback
+ progress = np.clip((t - t_start) / (t_end - t_start), 0, 1)
+ radius = ease_fn(progress) * max_radius
+ return mask_circle(g, cx_frac, cy_frac, radius, feather=0.03)
+
+def mask_wipe_h(g, t, t_start, t_end, direction="right"):
+ """Horizontal wipe reveal."""
+ progress = np.clip((t - t_start) / (t_end - t_start), 0, 1)
+ if direction == "left":
+ progress = 1 - progress
+ return mask_gradient_h(g, start=progress - 0.05, end=progress + 0.05)
+
+def mask_wipe_v(g, t, t_start, t_end, direction="down"):
+ """Vertical wipe reveal."""
+ progress = np.clip((t - t_start) / (t_end - t_start), 0, 1)
+ if direction == "up":
+ progress = 1 - progress
+ return mask_gradient_v(g, start=progress - 0.05, end=progress + 0.05)
+
+def mask_dissolve(g, t, t_start, t_end, seed=42):
+ """Random pixel dissolve — noise threshold sweeps from 0 to 1."""
+ progress = np.clip((t - t_start) / (t_end - t_start), 0, 1)
+ rng = np.random.RandomState(seed)
+ noise = rng.random((g.rows, g.cols)).astype(np.float32)
+ return (noise < progress).astype(np.float32)
+```
+
+### Mask Boolean Operations
+
+```python
+def mask_union(a, b):
+ """OR — visible where either mask is active."""
+ return np.maximum(a, b)
+
+def mask_intersect(a, b):
+ """AND — visible only where both masks are active."""
+ return np.minimum(a, b)
+
+def mask_subtract(a, b):
+ """A minus B — visible where A is active but B is not."""
+ return np.clip(a - b, 0, 1)
+
+def mask_invert(m):
+ """NOT — flip mask."""
+ return 1.0 - m
+```
+
+### Applying Masks to Canvases
+
+```python
+def apply_mask_canvas(canvas, mask, bg_canvas=None):
+ """Apply a grid-resolution mask to a pixel canvas.
+ Expands mask from (rows, cols) to (VH, VW) via nearest-neighbor.
+
+ canvas: uint8 (VH, VW, 3)
+ mask: float32 (rows, cols) [0,1]
+ bg_canvas: what shows through where mask=0. None = black.
+ """
+ # Expand mask to pixel resolution
+ mask_px = np.repeat(np.repeat(mask, canvas.shape[0] // mask.shape[0] + 1, axis=0),
+ canvas.shape[1] // mask.shape[1] + 1, axis=1)
+ mask_px = mask_px[:canvas.shape[0], :canvas.shape[1]]
+
+ if bg_canvas is not None:
+ return np.clip(canvas * mask_px[:, :, None] +
+ bg_canvas * (1 - mask_px[:, :, None]), 0, 255).astype(np.uint8)
+ return np.clip(canvas * mask_px[:, :, None], 0, 255).astype(np.uint8)
+
+def apply_mask_vf(vf_a, vf_b, mask):
+ """Apply mask at value-field level — blend two value fields spatially.
+ All arrays are (rows, cols) float32."""
+ return vf_a * mask + vf_b * (1 - mask)
+```
+
+---
+
+## PixelBlendStack
+
+Higher-level wrapper for multi-layer compositing:
+
+```python
+class PixelBlendStack:
+ def __init__(self):
+ self.layers = []
+
+ def add(self, canvas, mode="normal", opacity=1.0):
+ self.layers.append((canvas, mode, opacity))
+ return self
+
+ def composite(self):
+ if not self.layers:
+ return np.zeros((VH, VW, 3), dtype=np.uint8)
+ result = self.layers[0][0]
+ for canvas, mode, opacity in self.layers[1:]:
+ result = blend_canvas(result, canvas, mode, opacity)
+ return result
+```
+
+## Text Backdrop (Readability Mask)
+
+When placing readable text over busy multi-grid ASCII backgrounds, the text will blend into the background and become illegible. **Always apply a dark backdrop behind text regions.**
+
+The technique: compute the bounding box of all text glyphs, create a gaussian-blurred dark mask covering that area with padding, and multiply the background by `(1 - mask * darkness)` before rendering text on top.
+
+```python
+from scipy.ndimage import gaussian_filter
+
+def apply_text_backdrop(canvas, glyphs, padding=80, darkness=0.75):
+ """Darken the background behind text for readability.
+
+ Call AFTER rendering background, BEFORE rendering text.
+
+ Args:
+ canvas: (VH, VW, 3) uint8 background
+ glyphs: list of {"x": float, "y": float, ...} glyph positions
+ padding: pixel padding around text bounding box
+ darkness: 0.0 = no darkening, 1.0 = fully black
+ Returns:
+ darkened canvas (uint8)
+ """
+ if not glyphs:
+ return canvas
+ xs = [g['x'] for g in glyphs]
+ ys = [g['y'] for g in glyphs]
+ x0 = max(0, int(min(xs)) - padding)
+ y0 = max(0, int(min(ys)) - padding)
+ x1 = min(VW, int(max(xs)) + padding + 50) # extra for char width
+ y1 = min(VH, int(max(ys)) + padding + 60) # extra for char height
+
+ # Soft dark mask with gaussian blur for feathered edges
+ mask = np.zeros((VH, VW), dtype=np.float32)
+ mask[y0:y1, x0:x1] = 1.0
+ mask = gaussian_filter(mask, sigma=padding * 0.6)
+
+ factor = 1.0 - mask * darkness
+ return (canvas.astype(np.float32) * factor[:, :, np.newaxis]).astype(np.uint8)
+```
+
+### Usage in render pipeline
+
+Insert between background rendering and text rendering:
+
+```python
+# 1. Render background (multi-grid ASCII effects)
+bg = render_background(cfg, t)
+
+# 2. Darken behind text region
+bg = apply_text_backdrop(bg, frame_glyphs, padding=80, darkness=0.75)
+
+# 3. Render text on top (now readable against dark backdrop)
+bg = text_renderer.render(bg, frame_glyphs, color=(255, 255, 255))
+```
+
+Combine with **reverse vignette** (see shaders.md) for scenes where text is always centered — the reverse vignette provides a persistent center-dark zone, while the backdrop handles per-frame glyph positions.
+
+## External Layout Oracle Pattern
+
+For text-heavy videos where text needs to dynamically reflow around obstacles (shapes, icons, other text), use an external layout engine to pre-compute glyph positions and feed them into the Python renderer via JSON.
+
+### Architecture
+
+```
+Layout Engine (browser/Node.js) → layouts.json → Python ASCII Renderer
+ ↑ ↑
+ Computes per-frame Reads glyph positions,
+ glyph (x,y) positions renders as ASCII chars
+ with obstacle-aware reflow with full effect pipeline
+```
+
+### JSON interchange format
+
+```json
+{
+ "meta": {
+ "canvas_width": 1080, "canvas_height": 1080,
+ "fps": 24, "total_frames": 1248,
+ "fonts": {
+ "body": {"charW": 12.04, "charH": 24, "fontSize": 20},
+ "hero": {"charW": 24.08, "charH": 48, "fontSize": 40}
+ }
+ },
+ "scenes": [
+ {
+ "id": "scene_name",
+ "start_frame": 0, "end_frame": 96,
+ "frames": {
+ "0": {
+ "glyphs": [
+ {"char": "H", "x": 287.1, "y": 400.0, "alpha": 1.0},
+ {"char": "e", "x": 311.2, "y": 400.0, "alpha": 1.0}
+ ],
+ "obstacles": [
+ {"type": "circle", "cx": 540, "cy": 540, "r": 80},
+ {"type": "rect", "x": 300, "y": 500, "w": 120, "h": 80}
+ ]
+ }
+ }
+ }
+ ]
+}
+```
+
+### When to use
+
+- Text that dynamically reflows around moving objects
+- Per-glyph animation (reveal, scatter, physics)
+- Variable typography that needs precise measurement
+- Any case where Python's Pillow text layout is insufficient
+
+### When NOT to use
+
+- Static centered text (just use PIL `draw.text()` directly)
+- Text that only fades in/out without spatial animation
+- Simple typewriter effects (handle in Python with a character counter)
+
+### Running the oracle
+
+Use Playwright to run the layout engine in a headless browser:
+
+```javascript
+// extract.mjs
+import { chromium } from 'playwright';
+const browser = await chromium.launch({ headless: true });
+const page = await browser.newPage();
+await page.goto(`file://${oraclePath}`);
+await page.waitForFunction(() => window.__ORACLE_DONE__ === true, null, { timeout: 60000 });
+const result = await page.evaluate(() => window.__ORACLE_RESULT__);
+writeFileSync('layouts.json', JSON.stringify(result));
+await browser.close();
+```
+
+### Consuming in Python
+
+```python
+# In the renderer, map pixel positions to the canvas:
+for glyph in frame_data['glyphs']:
+ char, px, py = glyph['char'], glyph['x'], glyph['y']
+ alpha = glyph.get('alpha', 1.0)
+ # Render using PIL draw.text() at exact pixel position
+ draw.text((px, py), char, fill=(int(255*alpha),)*3, font=font)
+```
+
+Obstacles from the JSON can also be rendered as glowing ASCII shapes (circles, rectangles) to visualize the reflow zones.
diff --git a/src/skills/bundled/ascii-video/references/effects.md b/src/skills/bundled/ascii-video/references/effects.md
new file mode 100644
index 0000000..4ac1441
--- /dev/null
+++ b/src/skills/bundled/ascii-video/references/effects.md
@@ -0,0 +1,1865 @@
+# Effect Catalog
+
+Effect building blocks that produce visual patterns. In v2, these are used **inside scene functions** that return a pixel canvas directly. The building blocks below operate on grid coordinate arrays and produce `(chars, colors)` or value/hue fields that the scene function renders to canvas via `_render_vf()`.
+
+> **See also:** architecture.md · composition.md · scenes.md · shaders.md · troubleshooting.md
+
+## Design Philosophy
+
+Effects are the creative core. Don't copy these verbatim for every project -- use them as **building blocks** and **combine, modify, and invent** new ones. Every project should feel distinct.
+
+Key principles:
+- **Layer multiple effects** rather than using a single monolithic function
+- **Parameterize everything** -- hue, speed, density, amplitude should all be arguments
+- **React to features** -- audio/video features should modulate at least 2-3 parameters per effect
+- **Vary per section** -- never use the same effect config for the entire video
+- **Invent project-specific effects** -- the catalog below is a starting vocabulary, not a fixed set
+
+---
+
+## Background Fills
+
+Every effect should start with a background. Never leave flat black.
+
+### Animated Sine Field (General Purpose)
+```python
+def bg_sinefield(g, f, t, hue=0.6, bri=0.5, pal=PAL_DEFAULT,
+ freq=(0.13, 0.17, 0.07, 0.09), speed=(0.5, -0.4, -0.3, 0.2)):
+ """Layered sine field. Adjust freq/speed tuples for different textures."""
+ v1 = np.sin(g.cc*freq[0] + t*speed[0]) * np.sin(g.rr*freq[1] - t*speed[1]) * 0.5 + 0.5
+ v2 = np.sin(g.cc*freq[2] - t*speed[2] + g.rr*freq[3]) * 0.4 + 0.5
+ v3 = np.sin(g.dist_n*5 + t*0.2) * 0.3 + 0.4
+ v4 = np.cos(g.angle*3 - t*0.6) * 0.15 + 0.5
+ val = np.clip((v1*0.3 + v2*0.25 + v3*0.25 + v4*0.2) * bri * (0.6 + f["rms"]*0.6), 0.06, 1)
+ mask = val > 0.03
+ ch = val2char(val, mask, pal)
+ h = np.full_like(val, hue) + f.get("cent", 0.5)*0.1 + val*0.08
+ R, G, B = hsv2rgb(h, np.clip(0.35+f.get("flat",0.4)*0.4, 0, 1) * np.ones_like(val), val)
+ return ch, mkc(R, G, B, g.rows, g.cols)
+```
+
+### Video-Source Background
+```python
+def bg_video(g, frame_rgb, pal=PAL_DEFAULT, brightness=0.5):
+ small = np.array(Image.fromarray(frame_rgb).resize((g.cols, g.rows)))
+ lum = np.mean(small, axis=2) / 255.0 * brightness
+ mask = lum > 0.02
+ ch = val2char(lum, mask, pal)
+ co = np.clip(small * np.clip(lum[:,:,None]*1.5+0.3, 0.3, 1), 0, 255).astype(np.uint8)
+ return ch, co
+```
+
+### Noise / Static Field
+```python
+def bg_noise(g, f, t, pal=PAL_BLOCKS, density=0.3, hue_drift=0.02):
+ val = np.random.random((g.rows, g.cols)).astype(np.float32) * density * (0.5 + f["rms"]*0.5)
+ val = np.clip(val, 0, 1); mask = val > 0.02
+ ch = val2char(val, mask, pal)
+ R, G, B = hsv2rgb(np.full_like(val, t*hue_drift % 1), np.full_like(val, 0.3), val)
+ return ch, mkc(R, G, B, g.rows, g.cols)
+```
+
+### Perlin-Like Smooth Noise
+```python
+def bg_smooth_noise(g, f, t, hue=0.5, bri=0.5, pal=PAL_DOTS, octaves=3):
+ """Layered sine approximation of Perlin noise. Cheap, smooth, organic."""
+ val = np.zeros((g.rows, g.cols), dtype=np.float32)
+ for i in range(octaves):
+ freq = 0.05 * (2 ** i)
+ amp = 0.5 / (i + 1)
+ phase = t * (0.3 + i * 0.2)
+ val += np.sin(g.cc * freq + phase) * np.cos(g.rr * freq * 0.7 - phase * 0.5) * amp
+ val = np.clip(val * 0.5 + 0.5, 0, 1) * bri
+ mask = val > 0.03
+ ch = val2char(val, mask, pal)
+ h = np.full_like(val, hue) + val * 0.1
+ R, G, B = hsv2rgb(h, np.full_like(val, 0.5), val)
+ return ch, mkc(R, G, B, g.rows, g.cols)
+```
+
+### Cellular / Voronoi Approximation
+```python
+def bg_cellular(g, f, t, n_centers=12, hue=0.5, bri=0.6, pal=PAL_BLOCKS):
+ """Voronoi-like cells using distance to nearest of N moving centers."""
+ rng = np.random.RandomState(42) # deterministic centers
+ cx = (rng.rand(n_centers) * g.cols).astype(np.float32)
+ cy = (rng.rand(n_centers) * g.rows).astype(np.float32)
+ # Animate centers
+ cx_t = cx + np.sin(t * 0.5 + np.arange(n_centers) * 0.7) * 5
+ cy_t = cy + np.cos(t * 0.4 + np.arange(n_centers) * 0.9) * 3
+ # Min distance to any center
+ min_d = np.full((g.rows, g.cols), 999.0, dtype=np.float32)
+ for i in range(n_centers):
+ d = np.sqrt((g.cc - cx_t[i])**2 + (g.rr - cy_t[i])**2)
+ min_d = np.minimum(min_d, d)
+ val = np.clip(1.0 - min_d / (g.cols * 0.3), 0, 1) * bri
+ # Cell edges (where distance is near-equal between two centers)
+ # ... second-nearest trick for edge highlighting
+ mask = val > 0.03
+ ch = val2char(val, mask, pal)
+ R, G, B = hsv2rgb(np.full_like(val, hue) + min_d * 0.005, np.full_like(val, 0.5), val)
+ return ch, mkc(R, G, B, g.rows, g.cols)
+```
+
+---
+
+> **Note:** The v1 `eff_rings`, `eff_rays`, `eff_spiral`, `eff_glow`, `eff_tunnel`, `eff_vortex`, `eff_freq_waves`, `eff_interference`, `eff_aurora`, and `eff_ripple` functions are superseded by the `vf_*` value field generators below (used via `_render_vf()`). The `vf_*` versions integrate with the multi-grid composition pipeline and are preferred for all new scenes.
+
+---
+
+## Particle Systems
+
+### General Pattern
+All particle systems use persistent state via the `S` dict parameter:
+```python
+# S is the persistent state dict (same as r.S, passed explicitly)
+if "px" not in S:
+ S["px"]=[]; S["py"]=[]; S["vx"]=[]; S["vy"]=[]; S["life"]=[]; S["char"]=[]
+
+# Emit new particles (on beat, continuously, or on trigger)
+# Update: position += velocity, apply forces, decay life
+# Draw: map to grid, set char/color based on life
+# Cull: remove dead, cap total count
+```
+
+### Particle Character Sets
+
+Don't hardcode particle chars. Choose per project/mood:
+
+```python
+# Energy / explosive
+PART_ENERGY = list("*+#@\u26a1\u2726\u2605\u2588\u2593")
+PART_SPARK = list("\u00b7\u2022\u25cf\u2605\u2736*+")
+# Organic / natural
+PART_LEAF = list("\u2740\u2741\u2742\u2743\u273f\u2618\u2022")
+PART_SNOW = list("\u2744\u2745\u2746\u00b7\u2022*\u25cb")
+PART_RAIN = list("|\u2502\u2503\u2551/\\")
+PART_BUBBLE = list("\u25cb\u25ce\u25c9\u25cf\u2218\u2219\u00b0")
+# Data / tech
+PART_DATA = list("01{}[]<>|/\\")
+PART_HEX = list("0123456789ABCDEF")
+PART_BINARY = list("01")
+# Mystical
+PART_RUNE = list("\u16a0\u16a2\u16a6\u16b1\u16b7\u16c1\u16c7\u16d2\u16d6\u16da\u16de\u16df\u2726\u2605")
+PART_ZODIAC = list("\u2648\u2649\u264a\u264b\u264c\u264d\u264e\u264f\u2650\u2651\u2652\u2653")
+# Minimal
+PART_DOT = list("\u00b7\u2022\u25cf")
+PART_DASH = list("-=~\u2500\u2550")
+```
+
+### Explosion (Beat-Triggered)
+```python
+def emit_explosion(S, f, center_r, center_c, char_set=PART_ENERGY, count_base=80):
+ if f.get("beat", 0) > 0:
+ for _ in range(int(count_base + f["rms"]*150)):
+ ang = random.uniform(0, 2*math.pi)
+ sp = random.uniform(1, 9) * (0.5 + f.get("sub_r", 0.3)*2)
+ S["px"].append(float(center_c))
+ S["py"].append(float(center_r))
+ S["vx"].append(math.cos(ang)*sp*2.5)
+ S["vy"].append(math.sin(ang)*sp)
+ S["life"].append(1.0)
+ S["char"].append(random.choice(char_set))
+# Update: gravity on vy += 0.03, life -= 0.015
+# Color: life * 255 for brightness, hue fade controlled by caller
+```
+
+### Rising Embers
+```python
+# Emit: sy = rows-1, vy = -random.uniform(1,5), vx = random.uniform(-1.5,1.5)
+# Update: vx += random jitter * 0.3, life -= 0.01
+# Cap at ~1500 particles
+```
+
+### Dissolving Cloud
+```python
+# Init: N=600 particles spread across screen
+# Update: slow upward drift, fade life progressively
+# life -= 0.002 * (1 + elapsed * 0.05) # accelerating fade
+```
+
+### Starfield (3D Projection)
+```python
+# N stars with (sx, sy, sz) in normalized coords
+# Move: sz -= speed (stars approach camera)
+# Project: px = cx + sx/sz * cx, py = cy + sy/sz * cy
+# Reset stars that pass camera (sz <= 0.01)
+# Brightness = (1 - sz), draw streaks behind bright stars
+```
+
+### Orbit (Circular/Elliptical Motion)
+```python
+def emit_orbit(S, n=20, radius=15, speed=1.0, char_set=PART_DOT):
+ """Particles orbiting a center point."""
+ for i in range(n):
+ angle = i * 2 * math.pi / n
+ S["px"].append(0.0); S["py"].append(0.0) # will be computed from angle
+ S["vx"].append(angle) # store angle as "vx" for orbit
+ S["vy"].append(radius + random.uniform(-2, 2)) # store radius
+ S["life"].append(1.0)
+ S["char"].append(random.choice(char_set))
+# Update: angle += speed * dt, px = cx + radius * cos(angle), py = cy + radius * sin(angle)
+```
+
+### Gravity Well
+```python
+# Particles attracted toward one or more gravity points
+# Update: compute force vector toward each well, apply as acceleration
+# Particles that reach well center respawn at edges
+```
+
+### Flocking / Boids
+
+Emergent swarm behavior from three simple rules: separation, alignment, cohesion.
+
+```python
+def update_boids(S, g, f, n_boids=200, perception=8.0, max_speed=2.0,
+ sep_weight=1.5, ali_weight=1.0, coh_weight=1.0,
+ char_set=None):
+ """Boids flocking simulation. Particles self-organize into organic groups.
+
+ perception: how far each boid can see (grid cells)
+ sep_weight: separation (avoid crowding) strength
+ ali_weight: alignment (match neighbor velocity) strength
+ coh_weight: cohesion (steer toward group center) strength
+ """
+ if char_set is None:
+ char_set = list("·•●◦∘⬤")
+ if "boid_x" not in S:
+ rng = np.random.RandomState(42)
+ S["boid_x"] = rng.uniform(0, g.cols, n_boids).astype(np.float32)
+ S["boid_y"] = rng.uniform(0, g.rows, n_boids).astype(np.float32)
+ S["boid_vx"] = (rng.random(n_boids).astype(np.float32) - 0.5) * max_speed
+ S["boid_vy"] = (rng.random(n_boids).astype(np.float32) - 0.5) * max_speed
+ S["boid_ch"] = [random.choice(char_set) for _ in range(n_boids)]
+
+ bx = S["boid_x"]; by = S["boid_y"]
+ bvx = S["boid_vx"]; bvy = S["boid_vy"]
+ n = len(bx)
+
+ # For each boid, compute steering forces
+ ax = np.zeros(n, dtype=np.float32)
+ ay = np.zeros(n, dtype=np.float32)
+
+ # Spatial hash for efficient neighbor lookup
+ cell_size = perception
+ cells = {}
+ for i in range(n):
+ cx_i = int(bx[i] / cell_size)
+ cy_i = int(by[i] / cell_size)
+ key = (cx_i, cy_i)
+ if key not in cells:
+ cells[key] = []
+ cells[key].append(i)
+
+ for i in range(n):
+ cx_i = int(bx[i] / cell_size)
+ cy_i = int(by[i] / cell_size)
+ sep_x, sep_y = 0.0, 0.0
+ ali_x, ali_y = 0.0, 0.0
+ coh_x, coh_y = 0.0, 0.0
+ count = 0
+
+ # Check neighboring cells
+ for dcx in range(-1, 2):
+ for dcy in range(-1, 2):
+ for j in cells.get((cx_i + dcx, cy_i + dcy), []):
+ if j == i:
+ continue
+ dx = bx[j] - bx[i]
+ dy = by[j] - by[i]
+ dist = np.sqrt(dx * dx + dy * dy)
+ if dist < perception and dist > 0.01:
+ count += 1
+ # Separation: steer away from close neighbors
+ if dist < perception * 0.4:
+ sep_x -= dx / (dist * dist)
+ sep_y -= dy / (dist * dist)
+ # Alignment: match velocity
+ ali_x += bvx[j]
+ ali_y += bvy[j]
+ # Cohesion: steer toward center of group
+ coh_x += bx[j]
+ coh_y += by[j]
+
+ if count > 0:
+ # Normalize and weight
+ ax[i] += sep_x * sep_weight
+ ay[i] += sep_y * sep_weight
+ ax[i] += (ali_x / count - bvx[i]) * ali_weight * 0.1
+ ay[i] += (ali_y / count - bvy[i]) * ali_weight * 0.1
+ ax[i] += (coh_x / count - bx[i]) * coh_weight * 0.01
+ ay[i] += (coh_y / count - by[i]) * coh_weight * 0.01
+
+ # Audio reactivity: bass pushes boids outward from center
+ if f.get("bass", 0) > 0.5:
+ cx_g, cy_g = g.cols / 2, g.rows / 2
+ dx = bx - cx_g; dy = by - cy_g
+ dist = np.sqrt(dx**2 + dy**2) + 1
+ ax += (dx / dist) * f["bass"] * 2
+ ay += (dy / dist) * f["bass"] * 2
+
+ # Update velocity and position
+ bvx += ax; bvy += ay
+ # Clamp speed
+ speed = np.sqrt(bvx**2 + bvy**2) + 1e-10
+ over = speed > max_speed
+ bvx[over] *= max_speed / speed[over]
+ bvy[over] *= max_speed / speed[over]
+ bx += bvx; by += bvy
+
+ # Wrap at edges
+ bx %= g.cols; by %= g.rows
+
+ S["boid_x"] = bx; S["boid_y"] = by
+ S["boid_vx"] = bvx; S["boid_vy"] = bvy
+
+ # Draw
+ ch = np.full((g.rows, g.cols), " ", dtype="U1")
+ co = np.zeros((g.rows, g.cols, 3), dtype=np.uint8)
+ for i in range(n):
+ r, c = int(by[i]) % g.rows, int(bx[i]) % g.cols
+ ch[r, c] = S["boid_ch"][i]
+ spd = min(1.0, speed[i] / max_speed)
+ R, G, B = hsv2rgb_scalar(spd * 0.3, 0.8, 0.5 + spd * 0.5)
+ co[r, c] = (R, G, B)
+ return ch, co
+```
+
+### Flow Field Particles
+
+Particles that follow the gradient of a value field. Any `vf_*` function becomes a "river" that carries particles:
+
+```python
+def update_flow_particles(S, g, f, flow_field, n=500, speed=1.0,
+ life_drain=0.005, emit_rate=10,
+ char_set=None):
+ """Particles steered by a value field gradient.
+
+ flow_field: float32 (rows, cols) — the field particles follow.
+ Particles flow from low to high values (uphill) or along
+ the gradient direction.
+ """
+ if char_set is None:
+ char_set = list("·•∘◦°⋅")
+ if "fp_x" not in S:
+ S["fp_x"] = []; S["fp_y"] = []; S["fp_vx"] = []; S["fp_vy"] = []
+ S["fp_life"] = []; S["fp_ch"] = []
+
+ # Emit new particles at random positions
+ for _ in range(emit_rate):
+ if len(S["fp_x"]) < n:
+ S["fp_x"].append(random.uniform(0, g.cols - 1))
+ S["fp_y"].append(random.uniform(0, g.rows - 1))
+ S["fp_vx"].append(0.0); S["fp_vy"].append(0.0)
+ S["fp_life"].append(1.0)
+ S["fp_ch"].append(random.choice(char_set))
+
+ # Compute gradient of flow field (central differences)
+ pad = np.pad(flow_field, 1, mode="wrap")
+ grad_x = (pad[1:-1, 2:] - pad[1:-1, :-2]) * 0.5
+ grad_y = (pad[2:, 1:-1] - pad[:-2, 1:-1]) * 0.5
+
+ # Update particles
+ i = 0
+ while i < len(S["fp_x"]):
+ px, py = S["fp_x"][i], S["fp_y"][i]
+ # Sample gradient at particle position
+ gc = int(px) % g.cols; gr = int(py) % g.rows
+ gx = grad_x[gr, gc]; gy = grad_y[gr, gc]
+ # Steer velocity toward gradient direction
+ S["fp_vx"][i] = S["fp_vx"][i] * 0.9 + gx * speed * 10
+ S["fp_vy"][i] = S["fp_vy"][i] * 0.9 + gy * speed * 10
+ S["fp_x"][i] += S["fp_vx"][i]
+ S["fp_y"][i] += S["fp_vy"][i]
+ S["fp_life"][i] -= life_drain
+
+ if S["fp_life"][i] <= 0:
+ for k in ("fp_x", "fp_y", "fp_vx", "fp_vy", "fp_life", "fp_ch"):
+ S[k].pop(i)
+ else:
+ i += 1
+
+ # Draw
+ ch = np.full((g.rows, g.cols), " ", dtype="U1")
+ co = np.zeros((g.rows, g.cols, 3), dtype=np.uint8)
+ for i in range(len(S["fp_x"])):
+ r = int(S["fp_y"][i]) % g.rows
+ c = int(S["fp_x"][i]) % g.cols
+ ch[r, c] = S["fp_ch"][i]
+ v = S["fp_life"][i]
+ co[r, c] = (int(v * 200), int(v * 180), int(v * 255))
+ return ch, co
+```
+
+### Particle Trails
+
+Draw fading lines between current and previous positions:
+
+```python
+def draw_particle_trails(S, g, trail_key="trails", max_trail=8, fade=0.7):
+ """Add trails to any particle system. Call after updating positions.
+ Stores previous positions in S[trail_key] and draws fading lines.
+
+ Expects S to have 'px', 'py' lists (standard particle keys).
+ max_trail: number of previous positions to remember
+ fade: brightness multiplier per trail step (0.7 = 70% each step back)
+ """
+ if trail_key not in S:
+ S[trail_key] = []
+
+ # Store current positions
+ current = list(zip(
+ [int(y) for y in S.get("py", [])],
+ [int(x) for x in S.get("px", [])]
+ ))
+ S[trail_key].append(current)
+ if len(S[trail_key]) > max_trail:
+ S[trail_key] = S[trail_key][-max_trail:]
+
+ # Draw trails onto char/color arrays
+ ch = np.full((g.rows, g.cols), " ", dtype="U1")
+ co = np.zeros((g.rows, g.cols, 3), dtype=np.uint8)
+ trail_chars = list("·∘◦°⋅.,'`")
+
+ for age, positions in enumerate(reversed(S[trail_key])):
+ bri = fade ** age
+ if bri < 0.05:
+ break
+ ci = min(age, len(trail_chars) - 1)
+ for r, c in positions:
+ if 0 <= r < g.rows and 0 <= c < g.cols and ch[r, c] == " ":
+ ch[r, c] = trail_chars[ci]
+ v = int(bri * 180)
+ co[r, c] = (v, v, int(v * 0.8))
+ return ch, co
+```
+
+---
+
+## Rain / Matrix Effects
+
+### Column Rain (Vectorized)
+```python
+def eff_matrix_rain(g, f, t, S, hue=0.33, bri=0.6, pal=PAL_KATA,
+ speed_base=0.5, speed_beat=3.0):
+ """Vectorized matrix rain. S dict persists column positions."""
+ if "ry" not in S or len(S["ry"]) != g.cols:
+ S["ry"] = np.random.uniform(-g.rows, g.rows, g.cols).astype(np.float32)
+ S["rsp"] = np.random.uniform(0.3, 2.0, g.cols).astype(np.float32)
+ S["rln"] = np.random.randint(8, 40, g.cols)
+ S["rch"] = np.random.randint(0, len(pal), (g.rows, g.cols)) # pre-assign chars
+
+ speed_mult = speed_base + f.get("bass", 0.3)*speed_beat + f.get("sub_r", 0.3)*3
+ if f.get("beat", 0) > 0: speed_mult *= 2.5
+ S["ry"] += S["rsp"] * speed_mult
+
+ # Reset columns that fall past bottom
+ rst = (S["ry"] - S["rln"]) > g.rows
+ S["ry"][rst] = np.random.uniform(-25, -2, rst.sum())
+
+ # Vectorized draw using fancy indexing
+ ch = np.full((g.rows, g.cols), " ", dtype="U1")
+ co = np.zeros((g.rows, g.cols, 3), dtype=np.uint8)
+ heads = S["ry"].astype(int)
+ for c in range(g.cols):
+ head = heads[c]
+ trail_len = S["rln"][c]
+ for i in range(trail_len):
+ row = head - i
+ if 0 <= row < g.rows:
+ fade = 1.0 - i / trail_len
+ ci = S["rch"][row, c] % len(pal)
+ ch[row, c] = pal[ci]
+ v = fade * bri * 255
+ if i == 0: # head is bright white-ish
+ co[row, c] = (int(v*0.9), int(min(255, v*1.1)), int(v*0.9))
+ else:
+ R, G, B = hsv2rgb_single(hue, 0.7, fade * bri)
+ co[row, c] = (R, G, B)
+ return ch, co, S
+```
+
+---
+
+## Glitch / Data Effects
+
+### Horizontal Band Displacement
+```python
+def eff_glitch_displace(ch, co, f, intensity=1.0):
+ n_bands = int(8 + f.get("flux", 0.3)*25 + f.get("bdecay", 0)*15) * intensity
+ for _ in range(int(n_bands)):
+ y = random.randint(0, ch.shape[0]-1)
+ h = random.randint(1, int(3 + f.get("sub", 0.3)*8))
+ shift = int((random.random()-0.5) * f.get("rms", 0.3)*40 + f.get("bdecay", 0)*20*(random.random()-0.5))
+ if shift != 0:
+ for row in range(h):
+ rr = y + row
+ if 0 <= rr < ch.shape[0]:
+ ch[rr] = np.roll(ch[rr], shift)
+ co[rr] = np.roll(co[rr], shift, axis=0)
+ return ch, co
+```
+
+### Block Corruption
+```python
+def eff_block_corrupt(ch, co, f, char_pool=None, count_base=20):
+ if char_pool is None:
+ char_pool = list(PAL_BLOCKS[4:] + PAL_KATA[2:8])
+ for _ in range(int(count_base + f.get("flux", 0.3)*60 + f.get("bdecay", 0)*40)):
+ bx = random.randint(0, max(1, ch.shape[1]-6))
+ by = random.randint(0, max(1, ch.shape[0]-4))
+ bw, bh = random.randint(2,6), random.randint(1,4)
+ block_char = random.choice(char_pool)
+ # Fill rectangle with single char and random color
+ for r in range(bh):
+ for c in range(bw):
+ rr, cc = by+r, bx+c
+ if 0 <= rr < ch.shape[0] and 0 <= cc < ch.shape[1]:
+ ch[rr, cc] = block_char
+ co[rr, cc] = (random.randint(100,255), random.randint(0,100), random.randint(0,80))
+ return ch, co
+```
+
+### Scan Bars (Vertical)
+```python
+def eff_scanbars(ch, co, f, t, n_base=4, chars="|\u2551|!1l"):
+ for bi in range(int(n_base + f.get("himid_r", 0.3)*12)):
+ sx = int((t*50*(1+bi*0.3) + bi*37) % ch.shape[1])
+ for rr in range(ch.shape[0]):
+ if random.random() < 0.7:
+ ch[rr, sx] = random.choice(chars)
+ return ch, co
+```
+
+### Error Messages
+```python
+# Parameterize the error vocabulary per project:
+ERRORS_TECH = ["SEGFAULT","0xDEADBEEF","BUFFER_OVERRUN","PANIC!","NULL_PTR",
+ "CORRUPT","SIGSEGV","ERR_OVERFLOW","STACK_SMASH","BAD_ALLOC"]
+ERRORS_COSMIC = ["VOID_BREACH","ENTROPY_MAX","SINGULARITY","DIMENSION_FAULT",
+ "REALITY_ERR","TIME_PARADOX","DARK_MATTER_LEAK","QUANTUM_DECOHERE"]
+ERRORS_ORGANIC = ["CELL_DIVISION_ERR","DNA_MISMATCH","MUTATION_OVERFLOW",
+ "NEURAL_DEADLOCK","SYNAPSE_TIMEOUT","MEMBRANE_BREACH"]
+```
+
+### Hex Data Stream
+```python
+hex_str = "".join(random.choice("0123456789ABCDEF") for _ in range(random.randint(8,20)))
+stamp(ch, co, hex_str, rand_row, rand_col, (0, 160, 80))
+```
+
+---
+
+## Spectrum / Visualization
+
+### Mirrored Spectrum Bars
+```python
+def eff_spectrum(g, f, t, n_bars=64, pal=PAL_BLOCKS, mirror=True):
+ bar_w = max(1, g.cols // n_bars); mid = g.rows // 2
+ band_vals = np.array([f.get("sub",0.3), f.get("bass",0.3), f.get("lomid",0.3),
+ f.get("mid",0.3), f.get("himid",0.3), f.get("hi",0.3)])
+ ch = np.full((g.rows, g.cols), " ", dtype="U1")
+ co = np.zeros((g.rows, g.cols, 3), dtype=np.uint8)
+ for b in range(n_bars):
+ frac = b / n_bars
+ fi = frac * 5; lo_i = int(fi); hi_i = min(lo_i+1, 5)
+ bval = min(1, (band_vals[lo_i]*(1-fi%1) + band_vals[hi_i]*(fi%1)) * 1.8)
+ height = int(bval * (g.rows//2 - 2))
+ for dy in range(height):
+ hue = (f.get("cent",0.5)*0.3 + frac*0.3 + dy/max(height,1)*0.15) % 1.0
+ ci = pal[min(int(dy/max(height,1)*len(pal)*0.7+len(pal)*0.2), len(pal)-1)]
+ for dc in range(bar_w - (1 if bar_w > 2 else 0)):
+ cc = b*bar_w + dc
+ if 0 <= cc < g.cols:
+ rows_to_draw = [mid - dy, mid + dy] if mirror else [g.rows - 1 - dy]
+ for row in rows_to_draw:
+ if 0 <= row < g.rows:
+ ch[row, cc] = ci
+ co[row, cc] = hsv_to_rgb_single(hue, 0.85, 0.5+dy/max(height,1)*0.5)
+ return ch, co
+```
+
+### Waveform
+```python
+def eff_waveform(g, f, t, row_offset=-5, hue=0.1):
+ ch = np.full((g.rows, g.cols), " ", dtype="U1")
+ co = np.zeros((g.rows, g.cols, 3), dtype=np.uint8)
+ for c in range(g.cols):
+ wv = (math.sin(c*0.15+t*5)*f.get("bass",0.3)*0.5
+ + math.sin(c*0.3+t*8)*f.get("mid",0.3)*0.3
+ + math.sin(c*0.6+t*12)*f.get("hi",0.3)*0.15)
+ wr = g.rows + row_offset + int(wv * 4)
+ if 0 <= wr < g.rows:
+ ch[wr, c] = "~"
+ v = int(120 + f.get("rms",0.3)*135)
+ co[wr, c] = [v, int(v*0.7), int(v*0.4)]
+ return ch, co
+```
+
+---
+
+## Fire / Lava
+
+### Fire Columns
+```python
+def eff_fire(g, f, t, n_base=20, hue_base=0.02, hue_range=0.12, pal=PAL_BLOCKS):
+ n_cols = int(n_base + f.get("bass",0.3)*30 + f.get("sub_r",0.3)*20)
+ ch = np.full((g.rows, g.cols), " ", dtype="U1")
+ co = np.zeros((g.rows, g.cols, 3), dtype=np.uint8)
+ for fi in range(n_cols):
+ fx_c = int((fi*g.cols/n_cols + np.sin(t*2+fi*0.7)*3) % g.cols)
+ height = int((f.get("bass",0.3)*0.4 + f.get("sub_r",0.3)*0.3 + f.get("rms",0.3)*0.3) * g.rows * 0.7)
+ for dy in range(min(height, g.rows)):
+ fr = g.rows - 1 - dy
+ frac = dy / max(height, 1)
+ bri = max(0.1, (1 - frac*0.6) * (0.5 + f.get("rms",0.3)*0.5))
+ hue = hue_base + frac * hue_range
+ ci = "\u2588" if frac<0.2 else ("\u2593" if frac<0.4 else ("\u2592" if frac<0.6 else "\u2591"))
+ ch[fr, fx_c] = ci
+ R, G, B = hsv2rgb_single(hue, 0.9, bri)
+ co[fr, fx_c] = (R, G, B)
+ return ch, co
+```
+
+### Ice / Cold Fire (same structure, different hue range)
+```python
+# hue_base=0.55, hue_range=0.15 -- blue to cyan
+# Lower intensity, slower movement
+```
+
+---
+
+## Text Overlays
+
+### Scrolling Ticker
+```python
+def eff_ticker(ch, co, t, text, row, speed=15, color=(80, 100, 140)):
+ off = int(t * speed) % max(len(text), 1)
+ doubled = text + " " + text
+ stamp(ch, co, doubled[off:off+ch.shape[1]], row, 0, color)
+```
+
+### Beat-Triggered Words
+```python
+def eff_beat_words(ch, co, f, words, row_center=None, color=(255,240,220)):
+ if f.get("beat", 0) > 0:
+ w = random.choice(words)
+ r = (row_center or ch.shape[0]//2) + random.randint(-5,5)
+ stamp(ch, co, w, r, (ch.shape[1]-len(w))//2, color)
+```
+
+### Fading Message Sequence
+```python
+def eff_fading_messages(ch, co, t, elapsed, messages, period=4.0, color_base=(220,220,220)):
+ msg_idx = int(elapsed / period) % len(messages)
+ phase = elapsed % period
+ fade = max(0, min(1.0, phase) * min(1.0, period - phase))
+ if fade > 0.05:
+ v = fade
+ msg = messages[msg_idx]
+ cr, cg, cb = [int(c * v) for c in color_base]
+ stamp(ch, co, msg, ch.shape[0]//2, (ch.shape[1]-len(msg))//2, (cr, cg, cb))
+```
+
+---
+
+## Screen Shake
+Shift entire char/color arrays on beat:
+```python
+def eff_shake(ch, co, f, x_amp=6, y_amp=3):
+ shake_x = int(f.get("sub",0.3)*x_amp*(random.random()-0.5)*2 + f.get("bdecay",0)*4*(random.random()-0.5)*2)
+ shake_y = int(f.get("bass",0.3)*y_amp*(random.random()-0.5)*2)
+ if abs(shake_x) > 0:
+ ch = np.roll(ch, shake_x, axis=1)
+ co = np.roll(co, shake_x, axis=1)
+ if abs(shake_y) > 0:
+ ch = np.roll(ch, shake_y, axis=0)
+ co = np.roll(co, shake_y, axis=0)
+ return ch, co
+```
+
+---
+
+## Composable Effect System
+
+The real creative power comes from **composition**. There are three levels:
+
+### Level 1: Character-Level Layering
+
+Stack multiple effects as `(chars, colors)` layers:
+
+```python
+class LayerStack(EffectNode):
+ """Render effects bottom-to-top with character-level compositing."""
+ def add(self, effect, alpha=1.0):
+ """alpha < 1.0 = probabilistic override (sparse overlay)."""
+ self.layers.append((effect, alpha))
+
+# Usage:
+stack = LayerStack()
+stack.add(bg_effect) # base — fills screen
+stack.add(main_effect) # overlay on top (space chars = transparent)
+stack.add(particle_effect) # sparse overlay on top of that
+ch, co = stack.render(g, f, t, S)
+```
+
+### Level 2: Pixel-Level Blending
+
+After rendering to canvases, blend with Photoshop-style modes:
+
+```python
+class PixelBlendStack:
+ """Stack canvases with blend modes for complex compositing."""
+ def add(self, canvas, mode="normal", opacity=1.0)
+ def composite(self) -> canvas
+
+# Usage:
+pbs = PixelBlendStack()
+pbs.add(canvas_a) # base
+pbs.add(canvas_b, "screen", 0.7) # additive glow
+pbs.add(canvas_c, "difference", 0.5) # psychedelic interference
+result = pbs.composite()
+```
+
+### Level 3: Temporal Feedback
+
+Feed previous frame back into current frame for recursive effects:
+
+```python
+fb = FeedbackBuffer()
+for each frame:
+ canvas = render_current()
+ canvas = fb.apply(canvas, decay=0.8, blend="screen",
+ transform="zoom", transform_amt=0.015, hue_shift=0.02)
+```
+
+### Effect Nodes — Uniform Interface
+
+In the v2 protocol, effect nodes are used **inside** scene functions. The scene function itself returns a canvas. Effect nodes produce intermediate `(chars, colors)` that are rendered to canvas via the grid's `.render()` method or `_render_vf()`.
+
+```python
+class EffectNode:
+ def render(self, g, f, t, S) -> (chars, colors)
+
+# Concrete implementations:
+class ValueFieldEffect(EffectNode):
+ """Wraps a value field function + hue field function + palette."""
+ def __init__(self, val_fn, hue_fn, pal=PAL_DEFAULT, sat=0.7)
+
+class LambdaEffect(EffectNode):
+ """Wrap any (g,f,t,S) -> (ch,co) function."""
+ def __init__(self, fn)
+
+class ConditionalEffect(EffectNode):
+ """Switch effects based on audio features."""
+ def __init__(self, condition, if_true, if_false=None)
+```
+
+### Value Field Generators (Atomic Building Blocks)
+
+These produce float32 arrays `(rows, cols)` in range [0,1]. They are the raw visual patterns. All have signature `(g, f, t, S, **params) -> float32 array`.
+
+#### Trigonometric Fields (sine/cosine-based)
+
+```python
+def vf_sinefield(g, f, t, S, bri=0.5,
+ freq=(0.13, 0.17, 0.07, 0.09), speed=(0.5, -0.4, -0.3, 0.2)):
+ """Layered sine field. General purpose background/texture."""
+ v1 = np.sin(g.cc*freq[0] + t*speed[0]) * np.sin(g.rr*freq[1] - t*speed[1]) * 0.5 + 0.5
+ v2 = np.sin(g.cc*freq[2] - t*speed[2] + g.rr*freq[3]) * 0.4 + 0.5
+ v3 = np.sin(g.dist_n*5 + t*0.2) * 0.3 + 0.4
+ return np.clip((v1*0.35 + v2*0.35 + v3*0.3) * bri * (0.6 + f.get("rms",0.3)*0.6), 0, 1)
+
+def vf_smooth_noise(g, f, t, S, octaves=3, bri=0.5):
+ """Multi-octave sine approximation of Perlin noise."""
+ val = np.zeros((g.rows, g.cols), dtype=np.float32)
+ for i in range(octaves):
+ freq = 0.05 * (2 ** i); amp = 0.5 / (i + 1)
+ phase = t * (0.3 + i * 0.2)
+ val = val + np.sin(g.cc*freq + phase) * np.cos(g.rr*freq*0.7 - phase*0.5) * amp
+ return np.clip(val * 0.5 + 0.5, 0, 1) * bri
+
+def vf_rings(g, f, t, S, n_base=6, spacing_base=4):
+ """Concentric rings, bass-driven count and wobble."""
+ n = int(n_base + f.get("sub_r",0.3)*25 + f.get("bass",0.3)*10)
+ sp = spacing_base + f.get("bass_r",0.3)*7 + f.get("rms",0.3)*3
+ val = np.zeros((g.rows, g.cols), dtype=np.float32)
+ for ri in range(n):
+ rad = (ri+1)*sp + f.get("bdecay",0)*15
+ wobble = f.get("mid_r",0.3)*5*np.sin(g.angle*3+t*4)
+ rd = np.abs(g.dist - rad - wobble)
+ th = 1 + f.get("sub",0.3)*3
+ val = np.maximum(val, np.clip((1 - rd/th) * (0.4 + f.get("bass",0.3)*0.8), 0, 1))
+ return val
+
+def vf_spiral(g, f, t, S, n_arms=3, tightness=2.5):
+ """Logarithmic spiral arms."""
+ val = np.zeros((g.rows, g.cols), dtype=np.float32)
+ for ai in range(n_arms):
+ offset = ai * 2*np.pi / n_arms
+ log_r = np.log(g.dist + 1) * tightness
+ arm_phase = g.angle + offset - log_r + t * 0.8
+ arm_val = np.clip(np.cos(arm_phase * n_arms) * 0.6 + 0.2, 0, 1)
+ arm_val *= (0.4 + f.get("rms",0.3)*0.6) * np.clip(1 - g.dist_n*0.5, 0.2, 1)
+ val = np.maximum(val, arm_val)
+ return val
+
+def vf_tunnel(g, f, t, S, speed=3.0, complexity=6):
+ """Tunnel depth effect — infinite zoom feeling."""
+ tunnel_d = 1.0 / (g.dist_n + 0.1)
+ v1 = np.sin(tunnel_d*2 - t*speed) * 0.45 + 0.55
+ v2 = np.sin(g.angle*complexity + tunnel_d*1.5 - t*2) * 0.35 + 0.55
+ return np.clip(v1*0.5 + v2*0.5, 0, 1)
+
+def vf_vortex(g, f, t, S, twist=3.0):
+ """Twisting radial pattern — distance modulates angle."""
+ twisted = g.angle + g.dist_n * twist * np.sin(t * 0.5)
+ val = np.sin(twisted * 4 - t * 2) * 0.5 + 0.5
+ return np.clip(val * (0.5 + f.get("bass",0.3)*0.8), 0, 1)
+
+def vf_interference(g, f, t, S, n_waves=6):
+ """Overlapping sine waves creating moire patterns."""
+ drivers = ["mid_r", "himid_r", "bass_r", "lomid_r", "hi_r", "sub_r"]
+ vals = np.zeros((g.rows, g.cols), dtype=np.float32)
+ for i in range(min(n_waves, len(drivers))):
+ angle = i * np.pi / n_waves
+ freq = 0.06 + i * 0.03; sp = 0.5 + i * 0.3
+ proj = g.cc * np.cos(angle) + g.rr * np.sin(angle)
+ vals = vals + np.sin(proj*freq + t*sp) * f.get(drivers[i], 0.3) * 2.5
+ return np.clip(vals * 0.12 + 0.45, 0.1, 1)
+
+def vf_aurora(g, f, t, S, n_bands=3):
+ """Horizontal aurora bands."""
+ val = np.zeros((g.rows, g.cols), dtype=np.float32)
+ for i in range(n_bands):
+ fr = 0.08 + i*0.04; fc = 0.012 + i*0.008
+ sr = 0.7 + i*0.3; sc = 0.18 + i*0.12
+ val = val + np.sin(g.rr*fr + t*sr) * np.sin(g.cc*fc + t*sc) * (0.6/n_bands)
+ return np.clip(val * (f.get("lomid_r",0.3)*3 + 0.2), 0, 0.7)
+
+def vf_ripple(g, f, t, S, sources=None, freq=0.3, damping=0.02):
+ """Concentric ripples from point sources."""
+ if sources is None: sources = [(0.5, 0.5)]
+ val = np.zeros((g.rows, g.cols), dtype=np.float32)
+ for ry, rx in sources:
+ dy = g.rr - g.rows*ry; dx = g.cc - g.cols*rx
+ d = np.sqrt(dy**2 + dx**2)
+ val = val + np.sin(d*freq - t*4) * np.exp(-d*damping) * 0.5
+ return np.clip(val + 0.5, 0, 1)
+
+def vf_plasma(g, f, t, S):
+ """Classic plasma: sum of sines at different orientations and speeds."""
+ v = np.sin(g.cc * 0.03 + t * 0.7) * 0.5
+ v = v + np.sin(g.rr * 0.04 - t * 0.5) * 0.4
+ v = v + np.sin((g.cc * 0.02 + g.rr * 0.03) + t * 0.3) * 0.3
+ v = v + np.sin(g.dist_n * 4 - t * 0.8) * 0.3
+ return np.clip(v * 0.5 + 0.5, 0, 1)
+
+def vf_diamond(g, f, t, S, freq=0.15):
+ """Diamond/checkerboard pattern."""
+ val = np.abs(np.sin(g.cc * freq + t * 0.5)) * np.abs(np.sin(g.rr * freq * 1.2 - t * 0.3))
+ return np.clip(val * (0.6 + f.get("rms",0.3)*0.8), 0, 1)
+
+def vf_noise_static(g, f, t, S, density=0.4):
+ """Random noise — different each frame. Non-deterministic."""
+ return np.random.random((g.rows, g.cols)).astype(np.float32) * density * (0.5 + f.get("rms",0.3)*0.5)
+```
+
+#### Noise-Based Fields (organic, non-periodic)
+
+These produce qualitatively different textures from sine-based fields — organic, non-repeating, without visible axis alignment. They're the foundation of high-end generative art.
+
+```python
+def _hash2d(ix, iy):
+ """Integer-coordinate hash for gradient noise. Returns float32 in [0,1]."""
+ # Good-quality hash via large prime mixing
+ n = ix * 374761393 + iy * 668265263
+ n = (n ^ (n >> 13)) * 1274126177
+ return ((n ^ (n >> 16)) & 0x7fffffff).astype(np.float32) / 0x7fffffff
+
+def _smoothstep(t):
+ """Hermite smoothstep: 3t^2 - 2t^3. Smooth interpolation in [0,1]."""
+ t = np.clip(t, 0, 1)
+ return t * t * (3 - 2 * t)
+
+def _smootherstep(t):
+ """Perlin's improved smoothstep: 6t^5 - 15t^4 + 10t^3. C2-continuous."""
+ t = np.clip(t, 0, 1)
+ return t * t * t * (t * (t * 6 - 15) + 10)
+
+def _value_noise_2d(x, y):
+ """2D value noise at arbitrary float coordinates. Returns float32 in [0,1].
+ x, y: float32 arrays of same shape."""
+ ix = np.floor(x).astype(np.int64)
+ iy = np.floor(y).astype(np.int64)
+ fx = _smootherstep(x - ix)
+ fy = _smootherstep(y - iy)
+ # 4-corner hashes
+ n00 = _hash2d(ix, iy)
+ n10 = _hash2d(ix + 1, iy)
+ n01 = _hash2d(ix, iy + 1)
+ n11 = _hash2d(ix + 1, iy + 1)
+ # Bilinear interpolation
+ nx0 = n00 * (1 - fx) + n10 * fx
+ nx1 = n01 * (1 - fx) + n11 * fx
+ return nx0 * (1 - fy) + nx1 * fy
+
+def vf_noise(g, f, t, S, freq=0.08, speed=0.3, bri=0.7):
+ """Value noise. Smooth, organic, no axis alignment artifacts.
+ freq: spatial frequency (higher = finer detail).
+ speed: temporal scroll rate."""
+ x = g.cc * freq + t * speed
+ y = g.rr * freq * 0.8 - t * speed * 0.4
+ return np.clip(_value_noise_2d(x, y) * bri, 0, 1)
+
+def vf_fbm(g, f, t, S, octaves=5, freq=0.06, lacunarity=2.0, gain=0.5,
+ speed=0.2, bri=0.8):
+ """Fractal Brownian Motion — octaved noise with lacunarity/gain control.
+ The standard building block for clouds, terrain, smoke, organic textures.
+
+ octaves: number of noise layers (more = finer detail, more cost)
+ freq: base spatial frequency
+ lacunarity: frequency multiplier per octave (2.0 = standard)
+ gain: amplitude multiplier per octave (0.5 = standard, <0.5 = smoother)
+ speed: temporal evolution rate
+ """
+ val = np.zeros((g.rows, g.cols), dtype=np.float32)
+ amplitude = 1.0
+ f_x = freq
+ f_y = freq * 0.85 # slight anisotropy avoids grid artifacts
+ for i in range(octaves):
+ phase = t * speed * (1 + i * 0.3)
+ x = g.cc * f_x + phase + i * 17.3 # offset per octave
+ y = g.rr * f_y - phase * 0.6 + i * 31.7
+ val = val + _value_noise_2d(x, y) * amplitude
+ amplitude *= gain
+ f_x *= lacunarity
+ f_y *= lacunarity
+ # Normalize to [0,1]
+ max_amp = (1 - gain ** octaves) / (1 - gain) if gain != 1 else octaves
+ return np.clip(val / max_amp * bri * (0.6 + f.get("rms", 0.3) * 0.6), 0, 1)
+
+def vf_domain_warp(g, f, t, S, base_fn=None, warp_fn=None,
+ warp_strength=15.0, freq=0.06, speed=0.2):
+ """Domain warping — feed one noise field's output as coordinate offsets
+ into another noise field. Produces flowing, melting organic distortion.
+ Signature technique of high-end generative art (Inigo Quilez).
+
+ base_fn: value field to distort (default: fbm)
+ warp_fn: value field for displacement (default: noise at different freq)
+ warp_strength: how many grid cells to displace (higher = more warped)
+ """
+ # Warp field: displacement in x and y
+ wx = _value_noise_2d(g.cc * freq * 1.3 + t * speed, g.rr * freq + 7.1)
+ wy = _value_noise_2d(g.cc * freq + t * speed * 0.7 + 3.2, g.rr * freq * 1.1 - 11.8)
+ # Center warp around 0 (noise returns [0,1], shift to [-0.5, 0.5])
+ wx = (wx - 0.5) * warp_strength * (0.5 + f.get("rms", 0.3) * 1.0)
+ wy = (wy - 0.5) * warp_strength * (0.5 + f.get("bass", 0.3) * 0.8)
+ # Sample base field at warped coordinates
+ warped_cc = g.cc + wx
+ warped_rr = g.rr + wy
+ if base_fn is not None:
+ # Create a temporary grid-like object with warped coords
+ # Simplification: evaluate base_fn with modified coordinates
+ val = _value_noise_2d(warped_cc * freq * 0.8 + t * speed * 0.5,
+ warped_rr * freq * 0.7 - t * speed * 0.3)
+ else:
+ # Default: fbm at warped coordinates
+ val = np.zeros((g.rows, g.cols), dtype=np.float32)
+ amp = 1.0
+ fx, fy = freq * 0.8, freq * 0.7
+ for i in range(4):
+ val = val + _value_noise_2d(warped_cc * fx + t * speed * 0.5 + i * 13.7,
+ warped_rr * fy - t * speed * 0.3 + i * 27.3) * amp
+ amp *= 0.5; fx *= 2.0; fy *= 2.0
+ val = val / 1.875 # normalize 4-octave sum
+ return np.clip(val * 0.8, 0, 1)
+
+def vf_voronoi(g, f, t, S, n_cells=20, speed=0.3, edge_width=1.5,
+ mode="distance", seed=42):
+ """Voronoi diagram as value field. Proper implementation with
+ nearest/second-nearest distance for cell interiors and edges.
+
+ mode: "distance" (bright at center, dark at edges),
+ "edge" (bright at cell boundaries),
+ "cell_id" (flat color per cell — use with discrete palette)
+ edge_width: thickness of edge highlight (for "edge" mode)
+ """
+ rng = np.random.RandomState(seed)
+ # Animated cell centers
+ cx = rng.rand(n_cells).astype(np.float32) * g.cols
+ cy = rng.rand(n_cells).astype(np.float32) * g.rows
+ vx = (rng.rand(n_cells).astype(np.float32) - 0.5) * speed * 10
+ vy = (rng.rand(n_cells).astype(np.float32) - 0.5) * speed * 10
+ cx_t = (cx + vx * np.sin(t * 0.5 + np.arange(n_cells) * 0.8)) % g.cols
+ cy_t = (cy + vy * np.cos(t * 0.4 + np.arange(n_cells) * 1.1)) % g.rows
+
+ # Compute nearest and second-nearest distance
+ d1 = np.full((g.rows, g.cols), 1e9, dtype=np.float32)
+ d2 = np.full((g.rows, g.cols), 1e9, dtype=np.float32)
+ id1 = np.zeros((g.rows, g.cols), dtype=np.int32)
+ for i in range(n_cells):
+ d = np.sqrt((g.cc - cx_t[i]) ** 2 + (g.rr - cy_t[i]) ** 2)
+ mask = d < d1
+ d2 = np.where(mask, d1, np.minimum(d2, d))
+ id1 = np.where(mask, i, id1)
+ d1 = np.minimum(d1, d)
+
+ if mode == "edge":
+ # Edges: where d2 - d1 is small
+ edge_val = np.clip(1.0 - (d2 - d1) / edge_width, 0, 1)
+ return edge_val * (0.5 + f.get("rms", 0.3) * 0.8)
+ elif mode == "cell_id":
+ # Flat per-cell value
+ return (id1.astype(np.float32) / n_cells) % 1.0
+ else:
+ # Distance: bright near center, dark at edges
+ max_d = g.cols * 0.15
+ return np.clip(1.0 - d1 / max_d, 0, 1) * (0.5 + f.get("rms", 0.3) * 0.7)
+```
+
+#### Simulation-Based Fields (emergent, evolving)
+
+These use persistent state `S` to evolve patterns frame-by-frame. They produce complexity that can't be achieved with stateless math.
+
+```python
+def vf_reaction_diffusion(g, f, t, S, feed=0.055, kill=0.062,
+ da=1.0, db=0.5, dt=1.0, steps_per_frame=8,
+ init_mode="spots"):
+ """Gray-Scott reaction-diffusion model. Produces coral, leopard spots,
+ mitosis, worm-like, and labyrinthine patterns depending on feed/kill.
+
+ The two chemicals A and B interact:
+ A + 2B → 3B (autocatalytic)
+ B → P (decay)
+ feed: rate A is replenished, kill: rate B decays
+ Different feed/kill ratios produce radically different patterns.
+
+ Presets (feed, kill):
+ Spots/dots: (0.055, 0.062)
+ Worms/stripes: (0.046, 0.063)
+ Coral/branching: (0.037, 0.060)
+ Mitosis/splitting: (0.028, 0.062)
+ Labyrinth/maze: (0.029, 0.057)
+ Holes/negative: (0.039, 0.058)
+ Chaos/unstable: (0.026, 0.051)
+
+ steps_per_frame: simulation steps per video frame (more = faster evolution)
+ """
+ key = "rd_" + str(id(g)) # unique per grid
+ if key + "_a" not in S:
+ # Initialize chemical fields
+ A = np.ones((g.rows, g.cols), dtype=np.float32)
+ B = np.zeros((g.rows, g.cols), dtype=np.float32)
+ if init_mode == "spots":
+ # Random seed spots
+ rng = np.random.RandomState(42)
+ for _ in range(max(3, g.rows * g.cols // 200)):
+ r, c = rng.randint(2, g.rows - 2), rng.randint(2, g.cols - 2)
+ B[r - 1:r + 2, c - 1:c + 2] = 1.0
+ elif init_mode == "center":
+ cr, cc = g.rows // 2, g.cols // 2
+ B[cr - 3:cr + 3, cc - 3:cc + 3] = 1.0
+ elif init_mode == "ring":
+ mask = (g.dist_n > 0.2) & (g.dist_n < 0.3)
+ B[mask] = 1.0
+ S[key + "_a"] = A
+ S[key + "_b"] = B
+
+ A = S[key + "_a"]
+ B = S[key + "_b"]
+
+ # Audio modulation: feed/kill shift subtly with audio
+ f_mod = feed + f.get("bass", 0.3) * 0.003
+ k_mod = kill + f.get("hi_r", 0.3) * 0.002
+
+ for _ in range(steps_per_frame):
+ # Laplacian via 3x3 convolution kernel
+ # [0.05, 0.2, 0.05]
+ # [0.2, -1.0, 0.2]
+ # [0.05, 0.2, 0.05]
+ pA = np.pad(A, 1, mode="wrap")
+ pB = np.pad(B, 1, mode="wrap")
+ lapA = (pA[:-2, 1:-1] + pA[2:, 1:-1] + pA[1:-1, :-2] + pA[1:-1, 2:]) * 0.2 \
+ + (pA[:-2, :-2] + pA[:-2, 2:] + pA[2:, :-2] + pA[2:, 2:]) * 0.05 \
+ - A * 1.0
+ lapB = (pB[:-2, 1:-1] + pB[2:, 1:-1] + pB[1:-1, :-2] + pB[1:-1, 2:]) * 0.2 \
+ + (pB[:-2, :-2] + pB[:-2, 2:] + pB[2:, :-2] + pB[2:, 2:]) * 0.05 \
+ - B * 1.0
+ ABB = A * B * B
+ A = A + (da * lapA - ABB + f_mod * (1 - A)) * dt
+ B = B + (db * lapB + ABB - (f_mod + k_mod) * B) * dt
+ A = np.clip(A, 0, 1)
+ B = np.clip(B, 0, 1)
+
+ S[key + "_a"] = A
+ S[key + "_b"] = B
+ # Output B chemical as value (the visible pattern)
+ return np.clip(B * 2.0, 0, 1)
+
+def vf_game_of_life(g, f, t, S, rule="life", birth=None, survive=None,
+ steps_per_frame=1, density=0.3, fade=0.92, seed=42):
+ """Cellular automaton as value field with analog fade trails.
+ Grid cells are born/die by neighbor count rules. Dead cells fade
+ gradually instead of snapping to black, producing ghost trails.
+
+ rule presets:
+ "life": B3/S23 (Conway's Game of Life)
+ "coral": B3/S45678 (slow crystalline growth)
+ "maze": B3/S12345 (fills to labyrinth)
+ "anneal": B4678/S35678 (smooth blobs)
+ "day_night": B3678/S34678 (balanced growth/decay)
+ Or specify birth/survive directly as sets: birth={3}, survive={2,3}
+
+ fade: how fast dead cells dim (0.9 = slow trails, 0.5 = fast)
+ """
+ presets = {
+ "life": ({3}, {2, 3}),
+ "coral": ({3}, {4, 5, 6, 7, 8}),
+ "maze": ({3}, {1, 2, 3, 4, 5}),
+ "anneal": ({4, 6, 7, 8}, {3, 5, 6, 7, 8}),
+ "day_night": ({3, 6, 7, 8}, {3, 4, 6, 7, 8}),
+ }
+ if birth is None or survive is None:
+ birth, survive = presets.get(rule, presets["life"])
+
+ key = "gol_" + str(id(g))
+ if key + "_grid" not in S:
+ rng = np.random.RandomState(seed)
+ S[key + "_grid"] = (rng.random((g.rows, g.cols)) < density).astype(np.float32)
+ S[key + "_display"] = S[key + "_grid"].copy()
+
+ grid = S[key + "_grid"]
+ display = S[key + "_display"]
+
+ # Beat can inject random noise
+ if f.get("beat", 0) > 0.5:
+ inject = np.random.random((g.rows, g.cols)) < 0.02
+ grid = np.clip(grid + inject.astype(np.float32), 0, 1)
+
+ for _ in range(steps_per_frame):
+ # Count neighbors (toroidal wrap)
+ padded = np.pad(grid > 0.5, 1, mode="wrap").astype(np.int8)
+ neighbors = (padded[:-2, :-2] + padded[:-2, 1:-1] + padded[:-2, 2:] +
+ padded[1:-1, :-2] + padded[1:-1, 2:] +
+ padded[2:, :-2] + padded[2:, 1:-1] + padded[2:, 2:])
+ alive = grid > 0.5
+ new_alive = np.zeros_like(grid, dtype=bool)
+ for b in birth:
+ new_alive |= (~alive) & (neighbors == b)
+ for s in survive:
+ new_alive |= alive & (neighbors == s)
+ grid = new_alive.astype(np.float32)
+
+ # Analog display: alive cells = 1.0, dead cells fade
+ display = np.where(grid > 0.5, 1.0, display * fade)
+ S[key + "_grid"] = grid
+ S[key + "_display"] = display
+ return np.clip(display, 0, 1)
+
+def vf_strange_attractor(g, f, t, S, attractor="clifford",
+ n_points=50000, warmup=500, bri=0.8, seed=42,
+ params=None):
+ """Strange attractor projected to 2D density field.
+ Iterates N points through attractor equations, bins to grid,
+ produces a density map. Elegant, non-repeating curves.
+
+ attractor presets:
+ "clifford": sin(a*y) + c*cos(a*x), sin(b*x) + d*cos(b*y)
+ "de_jong": sin(a*y) - cos(b*x), sin(c*x) - cos(d*y)
+ "bedhead": sin(x*y/b) + cos(a*x - y), x*sin(a*y) + cos(b*x - y)
+
+ params: (a, b, c, d) floats — each attractor has different sweet spots.
+ If None, uses time-varying defaults for animation.
+ """
+ key = "attr_" + attractor
+ if params is None:
+ # Time-varying parameters for slow morphing
+ a = -1.4 + np.sin(t * 0.05) * 0.3
+ b = 1.6 + np.cos(t * 0.07) * 0.2
+ c = 1.0 + np.sin(t * 0.03 + 1) * 0.3
+ d = 0.7 + np.cos(t * 0.04 + 2) * 0.2
+ else:
+ a, b, c, d = params
+
+ # Iterate attractor
+ rng = np.random.RandomState(seed)
+ x = rng.uniform(-0.1, 0.1, n_points).astype(np.float64)
+ y = rng.uniform(-0.1, 0.1, n_points).astype(np.float64)
+
+ # Warmup iterations (reach the attractor)
+ for _ in range(warmup):
+ if attractor == "clifford":
+ xn = np.sin(a * y) + c * np.cos(a * x)
+ yn = np.sin(b * x) + d * np.cos(b * y)
+ elif attractor == "de_jong":
+ xn = np.sin(a * y) - np.cos(b * x)
+ yn = np.sin(c * x) - np.cos(d * y)
+ elif attractor == "bedhead":
+ xn = np.sin(x * y / b) + np.cos(a * x - y)
+ yn = x * np.sin(a * y) + np.cos(b * x - y)
+ else:
+ xn = np.sin(a * y) + c * np.cos(a * x)
+ yn = np.sin(b * x) + d * np.cos(b * y)
+ x, y = xn, yn
+
+ # Bin to grid
+ # Find bounds
+ margin = 0.1
+ x_min, x_max = x.min() - margin, x.max() + margin
+ y_min, y_max = y.min() - margin, y.max() + margin
+
+ # Map to grid coordinates
+ gx = ((x - x_min) / (x_max - x_min) * (g.cols - 1)).astype(np.int32)
+ gy = ((y - y_min) / (y_max - y_min) * (g.rows - 1)).astype(np.int32)
+ valid = (gx >= 0) & (gx < g.cols) & (gy >= 0) & (gy < g.rows)
+ gx, gy = gx[valid], gy[valid]
+
+ # Accumulate density
+ density = np.zeros((g.rows, g.cols), dtype=np.float32)
+ np.add.at(density, (gy, gx), 1.0)
+
+ # Log-scale density for visibility (most bins have few hits)
+ density = np.log1p(density)
+ mx = density.max()
+ if mx > 0:
+ density = density / mx
+ return np.clip(density * bri * (0.5 + f.get("rms", 0.3) * 0.8), 0, 1)
+```
+
+#### SDF-Based Fields (geometric precision)
+
+Signed Distance Fields produce mathematically precise shapes. Unlike sine fields (organic, blurry), SDFs give hard geometric boundaries with controllable edge softness. Combined with domain warping, they create "melting geometry" effects.
+
+All SDF primitives return a **signed distance** (negative inside, positive outside). Convert to a value field with `sdf_render()`.
+
+```python
+def sdf_render(dist, edge_width=1.5, invert=False):
+ """Convert signed distance to value field [0,1].
+ edge_width: controls anti-aliasing / softness of the boundary.
+ invert: True = bright inside shape, False = bright outside."""
+ val = 1.0 - np.clip(dist / edge_width, 0, 1) if not invert else np.clip(dist / edge_width, 0, 1)
+ return np.clip(val, 0, 1)
+
+def sdf_glow(dist, falloff=0.05):
+ """Render SDF as glowing outline — bright at boundary, fading both directions."""
+ return np.clip(np.exp(-np.abs(dist) * falloff), 0, 1)
+
+# --- Primitives ---
+
+def sdf_circle(g, cx_frac=0.5, cy_frac=0.5, radius=0.3):
+ """Circle SDF. cx/cy/radius in normalized [0,1] coordinates."""
+ dx = (g.cc / g.cols - cx_frac) * (g.cols / g.rows) # aspect correction
+ dy = g.rr / g.rows - cy_frac
+ return np.sqrt(dx**2 + dy**2) - radius
+
+def sdf_box(g, cx_frac=0.5, cy_frac=0.5, w=0.3, h=0.2, round_r=0.0):
+ """Rounded rectangle SDF."""
+ dx = np.abs(g.cc / g.cols - cx_frac) * (g.cols / g.rows) - w + round_r
+ dy = np.abs(g.rr / g.rows - cy_frac) - h + round_r
+ outside = np.sqrt(np.maximum(dx, 0)**2 + np.maximum(dy, 0)**2)
+ inside = np.minimum(np.maximum(dx, dy), 0)
+ return outside + inside - round_r
+
+def sdf_ring(g, cx_frac=0.5, cy_frac=0.5, radius=0.3, thickness=0.03):
+ """Ring (annulus) SDF."""
+ d = sdf_circle(g, cx_frac, cy_frac, radius)
+ return np.abs(d) - thickness
+
+def sdf_line(g, x0=0.2, y0=0.5, x1=0.8, y1=0.5, thickness=0.01):
+ """Line segment SDF between two points (normalized coords)."""
+ ax = g.cc / g.cols * (g.cols / g.rows) - x0 * (g.cols / g.rows)
+ ay = g.rr / g.rows - y0
+ bx = (x1 - x0) * (g.cols / g.rows)
+ by = y1 - y0
+ h = np.clip((ax * bx + ay * by) / (bx * bx + by * by + 1e-10), 0, 1)
+ dx = ax - bx * h
+ dy = ay - by * h
+ return np.sqrt(dx**2 + dy**2) - thickness
+
+def sdf_triangle(g, cx=0.5, cy=0.5, size=0.25):
+ """Equilateral triangle SDF centered at (cx, cy)."""
+ px = (g.cc / g.cols - cx) * (g.cols / g.rows) / size
+ py = (g.rr / g.rows - cy) / size
+ # Equilateral triangle math
+ k = np.sqrt(3.0)
+ px = np.abs(px) - 1.0
+ py = py + 1.0 / k
+ cond = px + k * py > 0
+ px2 = np.where(cond, (px - k * py) / 2.0, px)
+ py2 = np.where(cond, (-k * px - py) / 2.0, py)
+ px2 = np.clip(px2, -2.0, 0.0)
+ return -np.sqrt(px2**2 + py2**2) * np.sign(py2) * size
+
+def sdf_star(g, cx=0.5, cy=0.5, n_points=5, outer_r=0.25, inner_r=0.12):
+ """Star polygon SDF — n-pointed star."""
+ px = (g.cc / g.cols - cx) * (g.cols / g.rows)
+ py = g.rr / g.rows - cy
+ angle = np.arctan2(py, px)
+ dist = np.sqrt(px**2 + py**2)
+ # Modular angle for star symmetry
+ wedge = 2 * np.pi / n_points
+ a = np.abs((angle % wedge) - wedge / 2)
+ # Interpolate radius between inner and outer
+ r_at_angle = inner_r + (outer_r - inner_r) * np.clip(np.cos(a * n_points) * 0.5 + 0.5, 0, 1)
+ return dist - r_at_angle
+
+def sdf_heart(g, cx=0.5, cy=0.45, size=0.25):
+ """Heart shape SDF."""
+ px = (g.cc / g.cols - cx) * (g.cols / g.rows) / size
+ py = -(g.rr / g.rows - cy) / size + 0.3 # flip y, offset
+ px = np.abs(px)
+ cond = (px + py) > 1.0
+ d1 = np.sqrt((px - 0.25)**2 + (py - 0.75)**2) - np.sqrt(2.0) / 4.0
+ d2 = np.sqrt((px + py - 1.0)**2) / np.sqrt(2.0)
+ return np.where(cond, d1, d2) * size
+
+# --- Combinators ---
+
+def sdf_union(d1, d2):
+ """Boolean union — shape is wherever either SDF is inside."""
+ return np.minimum(d1, d2)
+
+def sdf_intersect(d1, d2):
+ """Boolean intersection — shape is where both SDFs overlap."""
+ return np.maximum(d1, d2)
+
+def sdf_subtract(d1, d2):
+ """Boolean subtraction — d1 minus d2."""
+ return np.maximum(d1, -d2)
+
+def sdf_smooth_union(d1, d2, k=0.1):
+ """Smooth minimum (polynomial) — blends shapes with rounded join.
+ k: smoothing radius. Higher = more rounding."""
+ h = np.clip(0.5 + 0.5 * (d2 - d1) / k, 0, 1)
+ return d2 * (1 - h) + d1 * h - k * h * (1 - h)
+
+def sdf_smooth_subtract(d1, d2, k=0.1):
+ """Smooth subtraction — d1 minus d2 with rounded edge."""
+ return sdf_smooth_union(d1, -d2, k)
+
+def sdf_repeat(g, sdf_fn, spacing_x=0.25, spacing_y=0.25, **sdf_kwargs):
+ """Tile an SDF primitive infinitely. spacing in normalized coords."""
+ # Modular coordinates
+ mod_cc = (g.cc / g.cols) % spacing_x - spacing_x / 2
+ mod_rr = (g.rr / g.rows) % spacing_y - spacing_y / 2
+ # Create modified grid-like arrays for the SDF
+ # This is a simplified approach — build a temporary namespace
+ class ModGrid:
+ pass
+ mg = ModGrid()
+ mg.cc = mod_cc * g.cols; mg.rr = mod_rr * g.rows
+ mg.cols = g.cols; mg.rows = g.rows
+ return sdf_fn(mg, **sdf_kwargs)
+
+# --- SDF as Value Field ---
+
+def vf_sdf(g, f, t, S, sdf_fn=sdf_circle, edge_width=1.5, glow=False,
+ glow_falloff=0.03, animate=True, **sdf_kwargs):
+ """Wrap any SDF primitive as a standard vf_* value field.
+ If animate=True, applies slow rotation and breathing to the shape."""
+ if animate:
+ sdf_kwargs.setdefault("cx_frac", 0.5)
+ sdf_kwargs.setdefault("cy_frac", 0.5)
+ d = sdf_fn(g, **sdf_kwargs)
+ if glow:
+ return sdf_glow(d, glow_falloff) * (0.5 + f.get("rms", 0.3) * 0.8)
+ return sdf_render(d, edge_width) * (0.5 + f.get("rms", 0.3) * 0.8)
+```
+
+### Hue Field Generators (Color Mapping)
+
+These produce float32 hue arrays [0,1]. Independently combinable with any value field. Each is a factory returning a closure with signature `(g, f, t, S) -> float32 array`. Can also be a plain float for fixed hue.
+
+```python
+def hf_fixed(hue):
+ """Single hue everywhere."""
+ def fn(g, f, t, S):
+ return np.full((g.rows, g.cols), hue, dtype=np.float32)
+ return fn
+
+def hf_angle(offset=0.0):
+ """Hue mapped to angle from center — rainbow wheel."""
+ def fn(g, f, t, S):
+ return (g.angle / (2 * np.pi) + offset + t * 0.05) % 1.0
+ return fn
+
+def hf_distance(base=0.5, scale=0.02):
+ """Hue mapped to distance from center."""
+ def fn(g, f, t, S):
+ return (base + g.dist * scale + t * 0.03) % 1.0
+ return fn
+
+def hf_time_cycle(speed=0.1):
+ """Hue cycles uniformly over time."""
+ def fn(g, f, t, S):
+ return np.full((g.rows, g.cols), (t * speed) % 1.0, dtype=np.float32)
+ return fn
+
+def hf_audio_cent():
+ """Hue follows spectral centroid — timbral color shifting."""
+ def fn(g, f, t, S):
+ return np.full((g.rows, g.cols), f.get("cent", 0.5) * 0.3, dtype=np.float32)
+ return fn
+
+def hf_gradient_h(start=0.0, end=1.0):
+ """Left-to-right hue gradient."""
+ def fn(g, f, t, S):
+ h = np.broadcast_to(
+ start + (g.cc / g.cols) * (end - start),
+ (g.rows, g.cols)
+ ).copy() # .copy() is CRITICAL — see troubleshooting.md
+ return h % 1.0
+ return fn
+
+def hf_gradient_v(start=0.0, end=1.0):
+ """Top-to-bottom hue gradient."""
+ def fn(g, f, t, S):
+ h = np.broadcast_to(
+ start + (g.rr / g.rows) * (end - start),
+ (g.rows, g.cols)
+ ).copy()
+ return h % 1.0
+ return fn
+
+def hf_plasma(speed=0.3):
+ """Plasma-style hue field — organic color variation."""
+ def fn(g, f, t, S):
+ return (np.sin(g.cc*0.02 + t*speed)*0.5 + np.sin(g.rr*0.015 + t*speed*0.7)*0.5) % 1.0
+ return fn
+```
+
+---
+
+## Coordinate Transforms
+
+UV-space transforms applied **before** effect evaluation. Any `vf_*` function can be rotated, zoomed, tiled, or distorted by transforming the grid coordinates it sees.
+
+### Transform Helpers
+
+```python
+def uv_rotate(g, angle):
+ """Rotate UV coordinates around grid center.
+ Returns (rotated_cc, rotated_rr) arrays — use in place of g.cc, g.rr."""
+ cx, cy = g.cols / 2.0, g.rows / 2.0
+ cos_a, sin_a = np.cos(angle), np.sin(angle)
+ dx = g.cc - cx
+ dy = g.rr - cy
+ return cx + dx * cos_a - dy * sin_a, cy + dx * sin_a + dy * cos_a
+
+def uv_scale(g, sx=1.0, sy=1.0, cx_frac=0.5, cy_frac=0.5):
+ """Scale UV coordinates around a center point.
+ sx, sy > 1 = zoom in (fewer repeats), < 1 = zoom out (more repeats)."""
+ cx = g.cols * cx_frac; cy = g.rows * cy_frac
+ return cx + (g.cc - cx) / sx, cy + (g.rr - cy) / sy
+
+def uv_skew(g, kx=0.0, ky=0.0):
+ """Skew UV coordinates. kx shears horizontally, ky vertically."""
+ return g.cc + g.rr * kx, g.rr + g.cc * ky
+
+def uv_tile(g, nx=3.0, ny=3.0, mirror=False):
+ """Tile UV coordinates. nx, ny = number of repeats.
+ mirror=True: alternating tiles are flipped (seamless)."""
+ u = (g.cc / g.cols * nx) % 1.0
+ v = (g.rr / g.rows * ny) % 1.0
+ if mirror:
+ flip_u = ((g.cc / g.cols * nx).astype(int) % 2) == 1
+ flip_v = ((g.rr / g.rows * ny).astype(int) % 2) == 1
+ u = np.where(flip_u, 1.0 - u, u)
+ v = np.where(flip_v, 1.0 - v, v)
+ return u * g.cols, v * g.rows
+
+def uv_polar(g):
+ """Convert Cartesian to polar UV. Returns (angle_as_cc, dist_as_rr).
+ Use to make any linear effect radial."""
+ # Angle wraps [0, cols), distance wraps [0, rows)
+ return g.angle / (2 * np.pi) * g.cols, g.dist_n * g.rows
+
+def uv_cartesian_from_polar(g):
+ """Convert polar-addressed effects back to Cartesian.
+ Treats g.cc as angle and g.rr as radius."""
+ angle = g.cc / g.cols * 2 * np.pi
+ radius = g.rr / g.rows
+ cx, cy = g.cols / 2.0, g.rows / 2.0
+ return cx + radius * np.cos(angle) * cx, cy + radius * np.sin(angle) * cy
+
+def uv_twist(g, amount=2.0):
+ """Twist: rotation increases with distance from center. Creates spiral distortion."""
+ twist_angle = g.dist_n * amount
+ return uv_rotate_raw(g.cc, g.rr, g.cols / 2, g.rows / 2, twist_angle)
+
+def uv_rotate_raw(cc, rr, cx, cy, angle):
+ """Raw rotation on arbitrary coordinate arrays."""
+ cos_a, sin_a = np.cos(angle), np.sin(angle)
+ dx = cc - cx; dy = rr - cy
+ return cx + dx * cos_a - dy * sin_a, cy + dx * sin_a + dy * cos_a
+
+def uv_fisheye(g, strength=1.5):
+ """Fisheye / barrel distortion on UV coordinates."""
+ cx, cy = g.cols / 2.0, g.rows / 2.0
+ dx = (g.cc - cx) / cx
+ dy = (g.rr - cy) / cy
+ r = np.sqrt(dx**2 + dy**2)
+ r_distort = np.power(r, strength)
+ scale = np.where(r > 0, r_distort / (r + 1e-10), 1.0)
+ return cx + dx * scale * cx, cy + dy * scale * cy
+
+def uv_wave(g, t, freq=0.1, amp=3.0, axis="x"):
+ """Sinusoidal coordinate displacement. Wobbles the UV space."""
+ if axis == "x":
+ return g.cc + np.sin(g.rr * freq + t * 3) * amp, g.rr
+ else:
+ return g.cc, g.rr + np.sin(g.cc * freq + t * 3) * amp
+
+def uv_mobius(g, a=1.0, b=0.0, c=0.0, d=1.0):
+ """Möbius transformation (conformal map): f(z) = (az + b) / (cz + d).
+ Operates on complex plane. Produces mathematically precise, visually
+ striking inversions and circular transforms."""
+ cx, cy = g.cols / 2.0, g.rows / 2.0
+ # Map grid to complex plane [-1, 1]
+ zr = (g.cc - cx) / cx
+ zi = (g.rr - cy) / cy
+ # Complex division: (a*z + b) / (c*z + d)
+ num_r = a * zr - 0 * zi + b # imaginary parts of a,b,c,d = 0 for real params
+ num_i = a * zi + 0 * zr + 0
+ den_r = c * zr - 0 * zi + d
+ den_i = c * zi + 0 * zr + 0
+ denom = den_r**2 + den_i**2 + 1e-10
+ wr = (num_r * den_r + num_i * den_i) / denom
+ wi = (num_i * den_r - num_r * den_i) / denom
+ return cx + wr * cx, cy + wi * cy
+```
+
+### Using Transforms with Value Fields
+
+Transforms modify what coordinates a value field sees. Wrap the transform around the `vf_*` call:
+
+```python
+# Rotate a plasma field 45 degrees
+def vf_rotated_plasma(g, f, t, S):
+ rc, rr = uv_rotate(g, np.pi / 4 + t * 0.1)
+ class TG: # transformed grid
+ pass
+ tg = TG(); tg.cc = rc; tg.rr = rr
+ tg.rows = g.rows; tg.cols = g.cols
+ tg.dist_n = g.dist_n; tg.angle = g.angle; tg.dist = g.dist
+ return vf_plasma(tg, f, t, S)
+
+# Tile a vortex 3x3 with mirror
+def vf_tiled_vortex(g, f, t, S):
+ tc, tr = uv_tile(g, 3, 3, mirror=True)
+ class TG:
+ pass
+ tg = TG(); tg.cc = tc; tg.rr = tr
+ tg.rows = g.rows; tg.cols = g.cols
+ tg.dist = np.sqrt((tc - g.cols/2)**2 + (tr - g.rows/2)**2)
+ tg.dist_n = tg.dist / (tg.dist.max() + 1e-10)
+ tg.angle = np.arctan2(tr - g.rows/2, tc - g.cols/2)
+ return vf_vortex(tg, f, t, S)
+
+# Helper: create transformed grid from coordinate arrays
+def make_tgrid(g, new_cc, new_rr):
+ """Build a grid-like object with transformed coordinates.
+ Preserves rows/cols for sizing, recomputes polar coords."""
+ class TG:
+ pass
+ tg = TG()
+ tg.cc = new_cc; tg.rr = new_rr
+ tg.rows = g.rows; tg.cols = g.cols
+ cx, cy = g.cols / 2.0, g.rows / 2.0
+ dx = new_cc - cx; dy = new_rr - cy
+ tg.dist = np.sqrt(dx**2 + dy**2)
+ tg.dist_n = tg.dist / (max(cx, cy) + 1e-10)
+ tg.angle = np.arctan2(dy, dx)
+ tg.dx = dx; tg.dy = dy
+ tg.dx_n = dx / max(g.cols, 1)
+ tg.dy_n = dy / max(g.rows, 1)
+ return tg
+```
+
+---
+
+## Temporal Coherence
+
+Tools for smooth, intentional parameter evolution over time. Replaces the default pattern of either static parameters or raw audio reactivity.
+
+### Easing Functions
+
+Standard animation easing curves. All take `t` in [0,1] and return [0,1]:
+
+```python
+def ease_linear(t): return t
+def ease_in_quad(t): return t * t
+def ease_out_quad(t): return t * (2 - t)
+def ease_in_out_quad(t): return np.where(t < 0.5, 2*t*t, -1 + (4-2*t)*t)
+def ease_in_cubic(t): return t**3
+def ease_out_cubic(t): return (t - 1)**3 + 1
+def ease_in_out_cubic(t):
+ return np.where(t < 0.5, 4*t**3, 1 - (-2*t + 2)**3 / 2)
+def ease_in_expo(t): return np.where(t == 0, 0, 2**(10*(t-1)))
+def ease_out_expo(t): return np.where(t == 1, 1, 1 - 2**(-10*t))
+def ease_elastic(t):
+ """Elastic ease-out — overshoots then settles."""
+ return np.where(t == 0, 0, np.where(t == 1, 1,
+ 2**(-10*t) * np.sin((t*10 - 0.75) * (2*np.pi) / 3) + 1))
+def ease_bounce(t):
+ """Bounce ease-out — bounces at the end."""
+ t = np.asarray(t, dtype=np.float64)
+ result = np.empty_like(t)
+ m1 = t < 1/2.75
+ m2 = (~m1) & (t < 2/2.75)
+ m3 = (~m1) & (~m2) & (t < 2.5/2.75)
+ m4 = ~(m1 | m2 | m3)
+ result[m1] = 7.5625 * t[m1]**2
+ t2 = t[m2] - 1.5/2.75; result[m2] = 7.5625 * t2**2 + 0.75
+ t3 = t[m3] - 2.25/2.75; result[m3] = 7.5625 * t3**2 + 0.9375
+ t4 = t[m4] - 2.625/2.75; result[m4] = 7.5625 * t4**2 + 0.984375
+ return result
+```
+
+### Keyframe Interpolation
+
+Define parameter values at specific times. Interpolates between them with easing:
+
+```python
+def keyframe(t, points, ease_fn=ease_in_out_cubic, loop=False):
+ """Interpolate between keyframed values.
+
+ Args:
+ t: current time (float, seconds)
+ points: list of (time, value) tuples, sorted by time
+ ease_fn: easing function for interpolation
+ loop: if True, wraps around after last keyframe
+
+ Returns:
+ interpolated value at time t
+
+ Example:
+ twist = keyframe(t, [(0, 1.0), (5, 6.0), (10, 2.0)], ease_out_cubic)
+ """
+ if not points:
+ return 0.0
+ if loop:
+ period = points[-1][0] - points[0][0]
+ if period > 0:
+ t = points[0][0] + (t - points[0][0]) % period
+
+ # Clamp to range
+ if t <= points[0][0]:
+ return points[0][1]
+ if t >= points[-1][0]:
+ return points[-1][1]
+
+ # Find surrounding keyframes
+ for i in range(len(points) - 1):
+ t0, v0 = points[i]
+ t1, v1 = points[i + 1]
+ if t0 <= t <= t1:
+ progress = (t - t0) / (t1 - t0)
+ eased = ease_fn(progress)
+ return v0 + (v1 - v0) * eased
+
+ return points[-1][1]
+
+def keyframe_array(t, points, ease_fn=ease_in_out_cubic):
+ """Keyframe interpolation that works with numpy arrays as values.
+ points: list of (time, np.array) tuples."""
+ if t <= points[0][0]: return points[0][1].copy()
+ if t >= points[-1][0]: return points[-1][1].copy()
+ for i in range(len(points) - 1):
+ t0, v0 = points[i]
+ t1, v1 = points[i + 1]
+ if t0 <= t <= t1:
+ progress = ease_fn((t - t0) / (t1 - t0))
+ return v0 * (1 - progress) + v1 * progress
+ return points[-1][1].copy()
+```
+
+### Value Field Morphing
+
+Smooth transition between two different value fields:
+
+```python
+def vf_morph(g, f, t, S, vf_a, vf_b, t_start, t_end,
+ ease_fn=ease_in_out_cubic):
+ """Morph between two value fields over a time range.
+
+ Usage:
+ val = vf_morph(g, f, t, S,
+ lambda g,f,t,S: vf_plasma(g,f,t,S),
+ lambda g,f,t,S: vf_vortex(g,f,t,S, twist=5),
+ t_start=10.0, t_end=15.0)
+ """
+ if t <= t_start:
+ return vf_a(g, f, t, S)
+ if t >= t_end:
+ return vf_b(g, f, t, S)
+ progress = ease_fn((t - t_start) / (t_end - t_start))
+ a = vf_a(g, f, t, S)
+ b = vf_b(g, f, t, S)
+ return a * (1 - progress) + b * progress
+
+def vf_sequence(g, f, t, S, fields, durations, crossfade=1.0,
+ ease_fn=ease_in_out_cubic):
+ """Cycle through a sequence of value fields with crossfades.
+
+ fields: list of vf_* callables
+ durations: list of float seconds per field
+ crossfade: seconds of overlap between adjacent fields
+ """
+ total = sum(durations)
+ t_local = t % total # loop
+ elapsed = 0
+ for i, dur in enumerate(durations):
+ if t_local < elapsed + dur:
+ # Current field
+ base = fields[i](g, f, t, S)
+ # Check if we're in a crossfade zone
+ time_in = t_local - elapsed
+ time_left = dur - time_in
+ if time_in < crossfade and i > 0:
+ # Fading in from previous
+ prev = fields[(i - 1) % len(fields)](g, f, t, S)
+ blend = ease_fn(time_in / crossfade)
+ return prev * (1 - blend) + base * blend
+ if time_left < crossfade and i < len(fields) - 1:
+ # Fading out to next
+ nxt = fields[(i + 1) % len(fields)](g, f, t, S)
+ blend = ease_fn(1 - time_left / crossfade)
+ return base * (1 - blend) + nxt * blend
+ return base
+ elapsed += dur
+ return fields[-1](g, f, t, S)
+```
+
+### Temporal Noise
+
+3D noise sampled at `(x, y, t)` — patterns evolve smoothly in time without per-frame discontinuities:
+
+```python
+def vf_temporal_noise(g, f, t, S, freq=0.06, t_freq=0.3, octaves=4,
+ bri=0.8):
+ """Noise field that evolves smoothly in time. Uses 3D noise via
+ two 2D noise lookups combined with temporal interpolation.
+
+ Unlike vf_fbm which scrolls noise (creating directional motion),
+ this morphs the pattern in-place — cells brighten and dim without
+ the field moving in any direction."""
+ # Two noise samples at floor/ceil of temporal coordinate
+ t_scaled = t * t_freq
+ t_lo = np.floor(t_scaled)
+ t_frac = _smootherstep(np.full((g.rows, g.cols), t_scaled - t_lo, dtype=np.float32))
+
+ val_lo = np.zeros((g.rows, g.cols), dtype=np.float32)
+ val_hi = np.zeros((g.rows, g.cols), dtype=np.float32)
+ amp = 1.0; fx = freq
+ for i in range(octaves):
+ val_lo = val_lo + _value_noise_2d(
+ g.cc * fx + t_lo * 7.3 + i * 13, g.rr * fx + t_lo * 3.1 + i * 29) * amp
+ val_hi = val_hi + _value_noise_2d(
+ g.cc * fx + (t_lo + 1) * 7.3 + i * 13, g.rr * fx + (t_lo + 1) * 3.1 + i * 29) * amp
+ amp *= 0.5; fx *= 2.0
+ max_amp = (1 - 0.5 ** octaves) / 0.5
+ val = (val_lo * (1 - t_frac) + val_hi * t_frac) / max_amp
+ return np.clip(val * bri * (0.6 + f.get("rms", 0.3) * 0.6), 0, 1)
+```
+
+---
+
+### Combining Value Fields
+
+The combinatorial explosion comes from mixing value fields with math:
+
+```python
+# Multiplication = intersection (only shows where both have brightness)
+combined = vf_plasma(g,f,t,S) * vf_vortex(g,f,t,S)
+
+# Addition = union (shows both, clips at 1.0)
+combined = np.clip(vf_rings(g,f,t,S) + vf_spiral(g,f,t,S), 0, 1)
+
+# Interference = beat pattern (shows XOR-like patterns)
+combined = np.abs(vf_plasma(g,f,t,S) - vf_tunnel(g,f,t,S))
+
+# Modulation = one effect shapes the other
+combined = vf_rings(g,f,t,S) * (0.3 + 0.7 * vf_plasma(g,f,t,S))
+
+# Maximum = shows the brightest of two effects
+combined = np.maximum(vf_spiral(g,f,t,S), vf_aurora(g,f,t,S))
+```
+
+### Full Scene Example (v2 — Canvas Return)
+
+A v2 scene function composes effects internally and returns a pixel canvas:
+
+```python
+def scene_complex(r, f, t, S):
+ """v2 scene function: returns canvas (uint8 H,W,3).
+ r = Renderer, f = audio features, t = time, S = persistent state dict."""
+ g = r.grids["md"]
+ rows, cols = g.rows, g.cols
+
+ # 1. Value field composition
+ plasma = vf_plasma(g, f, t, S)
+ vortex = vf_vortex(g, f, t, S, twist=4.0)
+ combined = np.clip(plasma * 0.6 + vortex * 0.5 + plasma * vortex * 0.4, 0, 1)
+
+ # 2. Color from hue field
+ h = (hf_angle(0.3)(g,f,t,S) * 0.5 + hf_time_cycle(0.08)(g,f,t,S) * 0.5) % 1.0
+
+ # 3. Render to canvas via _render_vf helper
+ canvas = _render_vf(g, combined, h, sat=0.75, pal=PAL_DENSE)
+
+ # 4. Optional: blend a second layer
+ overlay = _render_vf(r.grids["sm"], vf_rings(r.grids["sm"],f,t,S),
+ hf_fixed(0.6)(r.grids["sm"],f,t,S), pal=PAL_BLOCK)
+ canvas = blend_canvas(canvas, overlay, "screen", 0.4)
+
+ return canvas
+
+# In the render_clip() loop (handled by the framework):
+# canvas = scene_fn(r, f, t, S)
+# canvas = tonemap(canvas, gamma=scene_gamma)
+# canvas = feedback.apply(canvas, ...)
+# canvas = shader_chain.apply(canvas, f=f, t=t)
+# pipe.stdin.write(canvas.tobytes())
+```
+
+Vary the **value field combo**, **hue field**, **palette**, **blend modes**, **feedback config**, and **shader chain** per section for maximum visual variety. With 12 value fields × 8 hue fields × 14 palettes × 20 blend modes × 7 feedback transforms × 38 shaders, the combinations are effectively infinite.
+
+---
+
+## Combining Effects — Creative Guide
+
+The catalog above is vocabulary. Here's how to compose it into something that looks intentional.
+
+### Layering for Depth
+Every scene should have at least two layers at different grid densities:
+- **Background** (sm or xs): dense, dim texture that prevents flat black. fBM, smooth noise, or domain warp at low brightness (bri=0.15-0.25).
+- **Content** (md): the main visual — rings, voronoi, spirals, tunnel. Full brightness.
+- **Accent** (lg or xl): sparse highlights — particles, text stencil, glow pulse. Screen-blended on top.
+
+### Interesting Effect Pairs
+| Pair | Blend | Why it works |
+|------|-------|-------------|
+| fBM + voronoi edges | `screen` | Organic fills the cells, edges add structure |
+| Domain warp + plasma | `difference` | Psychedelic organic interference |
+| Tunnel + vortex | `screen` | Depth perspective + rotational energy |
+| Spiral + interference | `exclusion` | Moire patterns from different spatial frequencies |
+| Reaction-diffusion + fire | `add` | Living organic base + dynamic foreground |
+| SDF geometry + domain warp | `screen` | Clean shapes floating in organic texture |
+
+### Effects as Masks
+Any value field can be used as a mask for another effect via `mask_from_vf()`:
+- Voronoi cells masking fire (fire visible only inside cells)
+- fBM masking a solid color layer (organic color clouds)
+- SDF shapes masking a reaction-diffusion field
+- Animated iris/wipe revealing one effect over another
+
+### Inventing New Effects
+For every project, create at least one effect that isn't in the catalog:
+- **Combine two vf_* functions** with math: `np.clip(vf_fbm(...) * vf_rings(...), 0, 1)`
+- **Apply coordinate transforms** before evaluation: `vf_plasma(twisted_grid, ...)`
+- **Use one field to modulate another's parameters**: `vf_spiral(..., tightness=2 + vf_fbm(...) * 5)`
+- **Stack time offsets**: render the same field at `t` and `t - 0.5`, difference-blend for motion trails
+- **Mirror a value field** through an SDF boundary for kaleidoscopic geometry
diff --git a/src/skills/bundled/ascii-video/references/inputs.md b/src/skills/bundled/ascii-video/references/inputs.md
new file mode 100644
index 0000000..045b64a
--- /dev/null
+++ b/src/skills/bundled/ascii-video/references/inputs.md
@@ -0,0 +1,685 @@
+# Input Sources
+
+> **See also:** architecture.md · effects.md · scenes.md · shaders.md · optimization.md · troubleshooting.md
+
+## Audio Analysis
+
+### Loading
+
+```python
+tmp = tempfile.mktemp(suffix=".wav")
+subprocess.run(["ffmpeg", "-y", "-i", input_path, "-ac", "1", "-ar", "22050",
+ "-sample_fmt", "s16", tmp], capture_output=True, check=True)
+with wave.open(tmp) as wf:
+ sr = wf.getframerate()
+ raw = wf.readframes(wf.getnframes())
+samples = np.frombuffer(raw, dtype=np.int16).astype(np.float32) / 32768.0
+```
+
+### Per-Frame FFT
+
+```python
+hop = sr // fps # samples per frame
+win = hop * 2 # analysis window (2x hop for overlap)
+window = np.hanning(win)
+freqs = rfftfreq(win, 1.0 / sr)
+
+bands = {
+ "sub": (freqs >= 20) & (freqs < 80),
+ "bass": (freqs >= 80) & (freqs < 250),
+ "lomid": (freqs >= 250) & (freqs < 500),
+ "mid": (freqs >= 500) & (freqs < 2000),
+ "himid": (freqs >= 2000)& (freqs < 6000),
+ "hi": (freqs >= 6000),
+}
+```
+
+For each frame: extract chunk, apply window, FFT, compute band energies.
+
+### Feature Set
+
+| Feature | Formula | Controls |
+|---------|---------|----------|
+| `rms` | `sqrt(mean(chunk²))` | Overall loudness/energy |
+| `sub`..`hi` | `sqrt(mean(band_magnitudes²))` | Per-band energy |
+| `centroid` | `sum(freq*mag) / sum(mag)` | Brightness/timbre |
+| `flatness` | `geomean(mag) / mean(mag)` | Noise vs tone |
+| `flux` | `sum(max(0, mag - prev_mag))` | Transient strength |
+| `sub_r`..`hi_r` | `band / sum(all_bands)` | Spectral shape (volume-independent) |
+| `cent_d` | `abs(gradient(centroid))` | Timbral change rate |
+| `beat` | Flux peak detection | Binary beat onset |
+| `bdecay` | Exponential decay from beats | Smooth beat pulse (0→1→0) |
+
+**Band ratios are critical** — they decouple spectral shape from volume, so a quiet bass section and a loud bass section both read as "bassy" rather than just "loud" vs "quiet".
+
+### Smoothing
+
+EMA prevents visual jitter:
+
+```python
+def ema(arr, alpha):
+ out = np.empty_like(arr); out[0] = arr[0]
+ for i in range(1, len(arr)):
+ out[i] = alpha * arr[i] + (1 - alpha) * out[i-1]
+ return out
+
+# Slow-moving features (alpha=0.12): centroid, flatness, band ratios, cent_d
+# Fast-moving features (alpha=0.3): rms, flux, raw bands
+```
+
+### Beat Detection
+
+```python
+flux_smooth = np.convolve(flux, np.ones(5)/5, mode="same")
+peaks, _ = signal.find_peaks(flux_smooth, height=0.15, distance=fps//5, prominence=0.05)
+
+beat = np.zeros(n_frames)
+bdecay = np.zeros(n_frames, dtype=np.float32)
+for p in peaks:
+ beat[p] = 1.0
+ for d in range(fps // 2):
+ if p + d < n_frames:
+ bdecay[p + d] = max(bdecay[p + d], math.exp(-d * 2.5 / (fps // 2)))
+```
+
+`bdecay` gives smooth 0→1→0 pulse per beat, decaying over ~0.5s. Use for flash/glitch/mirror triggers.
+
+### Normalization
+
+After computing all frames, normalize each feature to 0-1:
+
+```python
+for k in features:
+ a = features[k]
+ lo, hi = a.min(), a.max()
+ features[k] = (a - lo) / (hi - lo + 1e-10)
+```
+
+## Video Sampling
+
+### Frame Extraction
+
+```python
+# Method 1: ffmpeg pipe (memory efficient)
+cmd = ["ffmpeg", "-i", input_video, "-f", "rawvideo", "-pix_fmt", "rgb24",
+ "-s", f"{target_w}x{target_h}", "-r", str(fps), "-"]
+pipe = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.DEVNULL)
+frame_size = target_w * target_h * 3
+for fi in range(n_frames):
+ raw = pipe.stdout.read(frame_size)
+ if len(raw) < frame_size: break
+ frame = np.frombuffer(raw, dtype=np.uint8).reshape(target_h, target_w, 3)
+ # process frame...
+
+# Method 2: OpenCV (if available)
+cap = cv2.VideoCapture(input_video)
+```
+
+### Luminance-to-Character Mapping
+
+Convert video pixels to ASCII characters based on brightness:
+
+```python
+def frame_to_ascii(frame_rgb, grid, pal=PAL_DEFAULT):
+ """Convert video frame to character + color arrays."""
+ rows, cols = grid.rows, grid.cols
+ # Resize frame to grid dimensions
+ small = np.array(Image.fromarray(frame_rgb).resize((cols, rows), Image.LANCZOS))
+ # Luminance
+ lum = (0.299 * small[:,:,0] + 0.587 * small[:,:,1] + 0.114 * small[:,:,2]) / 255.0
+ # Map to chars
+ chars = val2char(lum, lum > 0.02, pal)
+ # Colors: use source pixel colors, scaled by luminance for visibility
+ colors = np.clip(small * np.clip(lum[:,:,None] * 1.5 + 0.3, 0.3, 1), 0, 255).astype(np.uint8)
+ return chars, colors
+```
+
+### Edge-Weighted Character Mapping
+
+Use edge detection for more detail in contour regions:
+
+```python
+def frame_to_ascii_edges(frame_rgb, grid, pal=PAL_DEFAULT, edge_pal=PAL_BOX):
+ gray = np.mean(frame_rgb, axis=2)
+ small_gray = resize(gray, (grid.rows, grid.cols))
+ lum = small_gray / 255.0
+
+ # Sobel edge detection
+ gx = np.abs(small_gray[:, 2:] - small_gray[:, :-2])
+ gy = np.abs(small_gray[2:, :] - small_gray[:-2, :])
+ edge = np.zeros_like(small_gray)
+ edge[:, 1:-1] += gx; edge[1:-1, :] += gy
+ edge = np.clip(edge / edge.max(), 0, 1)
+
+ # Edge regions get box drawing chars, flat regions get brightness chars
+ is_edge = edge > 0.15
+ chars = val2char(lum, lum > 0.02, pal)
+ edge_chars = val2char(edge, is_edge, edge_pal)
+ chars[is_edge] = edge_chars[is_edge]
+
+ return chars, colors
+```
+
+### Motion Detection
+
+Detect pixel changes between frames for motion-reactive effects:
+
+```python
+prev_frame = None
+def compute_motion(frame):
+ global prev_frame
+ if prev_frame is None:
+ prev_frame = frame.astype(np.float32)
+ return np.zeros(frame.shape[:2])
+ diff = np.abs(frame.astype(np.float32) - prev_frame).mean(axis=2)
+ prev_frame = frame.astype(np.float32) * 0.7 + prev_frame * 0.3 # smoothed
+ return np.clip(diff / 30.0, 0, 1) # normalized motion map
+```
+
+Use motion map to drive particle emission, glitch intensity, or character density.
+
+### Video Feature Extraction
+
+Per-frame features analogous to audio features, for driving effects:
+
+```python
+def analyze_video_frame(frame_rgb):
+ gray = np.mean(frame_rgb, axis=2)
+ return {
+ "brightness": gray.mean() / 255.0,
+ "contrast": gray.std() / 128.0,
+ "edge_density": compute_edge_density(gray),
+ "motion": compute_motion(frame_rgb).mean(),
+ "dominant_hue": compute_dominant_hue(frame_rgb),
+ "color_variance": compute_color_variance(frame_rgb),
+ }
+```
+
+## Image Sequence
+
+### Static Image to ASCII
+
+Same as single video frame conversion. For animated sequences:
+
+```python
+import glob
+frames = sorted(glob.glob("frames/*.png"))
+for fi, path in enumerate(frames):
+ img = np.array(Image.open(path).resize((VW, VH)))
+ chars, colors = frame_to_ascii(img, grid, pal)
+```
+
+### Image as Texture Source
+
+Use an image as a background texture that effects modulate:
+
+```python
+def load_texture(path, grid):
+ img = np.array(Image.open(path).resize((grid.cols, grid.rows)))
+ lum = np.mean(img, axis=2) / 255.0
+ return lum, img # luminance for char mapping, RGB for colors
+```
+
+## Text / Lyrics
+
+### SRT Parsing
+
+```python
+import re
+def parse_srt(path):
+ """Returns [(start_sec, end_sec, text), ...]"""
+ entries = []
+ with open(path) as f:
+ content = f.read()
+ blocks = content.strip().split("\n\n")
+ for block in blocks:
+ lines = block.strip().split("\n")
+ if len(lines) >= 3:
+ times = lines[1]
+ m = re.match(r"(\d+):(\d+):(\d+),(\d+) --> (\d+):(\d+):(\d+),(\d+)", times)
+ if m:
+ g = [int(x) for x in m.groups()]
+ start = g[0]*3600 + g[1]*60 + g[2] + g[3]/1000
+ end = g[4]*3600 + g[5]*60 + g[6] + g[7]/1000
+ text = " ".join(lines[2:])
+ entries.append((start, end, text))
+ return entries
+```
+
+### Lyrics Display Modes
+
+- **Typewriter**: characters appear left-to-right over the time window
+- **Fade-in**: whole line fades from dark to bright
+- **Flash**: appear instantly on beat, fade out
+- **Scatter**: characters start at random positions, converge to final position
+- **Wave**: text follows a sine wave path
+
+```python
+def lyrics_typewriter(ch, co, text, row, col, t, t_start, t_end, color):
+ """Reveal characters progressively over time window."""
+ progress = np.clip((t - t_start) / (t_end - t_start), 0, 1)
+ n_visible = int(len(text) * progress)
+ stamp(ch, co, text[:n_visible], row, col, color)
+```
+
+## Generative (No Input)
+
+For pure generative ASCII art, the "features" dict is synthesized from time:
+
+```python
+def synthetic_features(t, bpm=120):
+ """Generate audio-like features from time alone."""
+ beat_period = 60.0 / bpm
+ beat_phase = (t % beat_period) / beat_period
+ return {
+ "rms": 0.5 + 0.3 * math.sin(t * 0.5),
+ "bass": 0.5 + 0.4 * math.sin(t * 2 * math.pi / beat_period),
+ "sub": 0.3 + 0.3 * math.sin(t * 0.8),
+ "mid": 0.4 + 0.3 * math.sin(t * 1.3),
+ "hi": 0.3 + 0.2 * math.sin(t * 2.1),
+ "cent": 0.5 + 0.2 * math.sin(t * 0.3),
+ "flat": 0.4,
+ "flux": 0.3 + 0.2 * math.sin(t * 3),
+ "beat": 1.0 if beat_phase < 0.05 else 0.0,
+ "bdecay": max(0, 1.0 - beat_phase * 4),
+ # ratios
+ "sub_r": 0.2, "bass_r": 0.25, "lomid_r": 0.15,
+ "mid_r": 0.2, "himid_r": 0.12, "hi_r": 0.08,
+ "cent_d": 0.1,
+ }
+```
+
+## TTS Integration
+
+For narrated videos (testimonials, quotes, storytelling), generate speech audio per segment and mix with background music.
+
+### ElevenLabs Voice Generation
+
+```python
+import requests, time, os
+
+def generate_tts(text, voice_id, api_key, output_path, model="eleven_multilingual_v2"):
+ """Generate TTS audio via ElevenLabs API. Streams response to disk."""
+ # Skip if already generated (idempotent re-runs)
+ if os.path.exists(output_path) and os.path.getsize(output_path) > 1000:
+ return
+
+ url = f"https://api.elevenlabs.io/v1/text-to-speech/{voice_id}"
+ headers = {"xi-api-key": api_key, "Content-Type": "application/json"}
+ data = {
+ "text": text,
+ "model_id": model,
+ "voice_settings": {
+ "stability": 0.65,
+ "similarity_boost": 0.80,
+ "style": 0.15,
+ "use_speaker_boost": True,
+ },
+ }
+ resp = requests.post(url, json=data, headers=headers, stream=True)
+ resp.raise_for_status()
+ with open(output_path, "wb") as f:
+ for chunk in resp.iter_content(chunk_size=4096):
+ f.write(chunk)
+ time.sleep(0.3) # rate limit: avoid 429s on batch generation
+```
+
+Voice settings notes:
+- `stability` 0.65 gives natural variation without drift. Lower (0.3-0.5) for more expressive reads, higher (0.7-0.9) for monotone/narration.
+- `similarity_boost` 0.80 keeps it close to the voice profile. Lower for more generic sound.
+- `style` 0.15 adds slight stylistic variation. Keep low (0-0.2) for straightforward reads.
+- `use_speaker_boost` True improves clarity at the cost of slightly more processing time.
+
+### Voice Pool
+
+ElevenLabs has ~20 built-in voices. Use multiple voices for variety across quotes. Reference pool:
+
+```python
+VOICE_POOL = [
+ ("JBFqnCBsd6RMkjVDRZzb", "George"),
+ ("nPczCjzI2devNBz1zQrb", "Brian"),
+ ("pqHfZKP75CvOlQylNhV4", "Bill"),
+ ("CwhRBWXzGAHq8TQ4Fs17", "Roger"),
+ ("cjVigY5qzO86Huf0OWal", "Eric"),
+ ("onwK4e9ZLuTAKqWW03F9", "Daniel"),
+ ("IKne3meq5aSn9XLyUdCD", "Charlie"),
+ ("iP95p4xoKVk53GoZ742B", "Chris"),
+ ("bIHbv24MWmeRgasZH58o", "Will"),
+ ("TX3LPaxmHKxFdv7VOQHJ", "Liam"),
+ ("SAz9YHcvj6GT2YYXdXww", "River"),
+ ("EXAVITQu4vr4xnSDxMaL", "Sarah"),
+ ("Xb7hH8MSUJpSbSDYk0k2", "Alice"),
+ ("pFZP5JQG7iQjIQuC4Bku", "Lily"),
+ ("XrExE9yKIg1WjnnlVkGX", "Matilda"),
+ ("FGY2WhTYpPnrIDTdsKH5", "Laura"),
+ ("SOYHLrjzK2X1ezoPC6cr", "Harry"),
+ ("hpp4J3VqNfWAUOO0d1Us", "Bella"),
+ ("N2lVS1w4EtoT3dr4eOWO", "Callum"),
+ ("cgSgspJ2msm6clMCkdW9", "Jessica"),
+ ("pNInz6obpgDQGcFmaJgB", "Adam"),
+]
+```
+
+### Voice Assignment
+
+Shuffle deterministically so re-runs produce the same voice mapping:
+
+```python
+import random as _rng
+
+def assign_voices(n_quotes, voice_pool, seed=42):
+ """Assign a different voice to each quote, cycling if needed."""
+ r = _rng.Random(seed)
+ ids = [v[0] for v in voice_pool]
+ r.shuffle(ids)
+ return [ids[i % len(ids)] for i in range(n_quotes)]
+```
+
+### Pronunciation Control
+
+TTS text must be separate from display text. The display text has line breaks for visual layout; the TTS text is a flat sentence with phonetic fixes.
+
+Common fixes:
+- Brand names: spell phonetically ("Nous" -> "Noose", "nginx" -> "engine-x")
+- Abbreviations: expand ("API" -> "A P I", "CLI" -> "C L I")
+- Technical terms: add phonetic hints
+- Punctuation for pacing: periods create pauses, commas create slight pauses
+
+```python
+# Display text: line breaks control visual layout
+QUOTES = [
+ ("It can do far more than the Claws,\nand you don't need to buy a Mac Mini.\nNous Research has a winner here.", "Brian Roemmele"),
+]
+
+# TTS text: flat, phonetically corrected for speech
+QUOTES_TTS = [
+ "It can do far more than the Claws, and you don't need to buy a Mac Mini. Noose Research has a winner here.",
+]
+# Keep both arrays in sync -- same indices
+```
+
+### Audio Pipeline
+
+1. Generate individual TTS clips (MP3 per quote, skipping existing)
+2. Convert each to WAV (mono, 22050 Hz) for duration measurement and concatenation
+3. Calculate timing: intro pad + speech + gaps + outro pad = target duration
+4. Concatenate into single TTS track with silence padding
+5. Mix with background music
+
+```python
+def build_tts_track(tts_clips, target_duration, intro_pad=5.0, outro_pad=4.0):
+ """Concatenate TTS clips with calculated gaps, pad to target duration.
+
+ Returns:
+ timing: list of (start_time, end_time, quote_index) tuples
+ """
+ sr = 22050
+
+ # Convert MP3s to WAV for duration and sample-level concatenation
+ durations = []
+ for clip in tts_clips:
+ wav = clip.replace(".mp3", ".wav")
+ subprocess.run(
+ ["ffmpeg", "-y", "-i", clip, "-ac", "1", "-ar", str(sr),
+ "-sample_fmt", "s16", wav],
+ capture_output=True, check=True)
+ result = subprocess.run(
+ ["ffprobe", "-v", "error", "-show_entries", "format=duration",
+ "-of", "csv=p=0", wav],
+ capture_output=True, text=True)
+ durations.append(float(result.stdout.strip()))
+
+ # Calculate gap to fill target duration
+ total_speech = sum(durations)
+ n_gaps = len(tts_clips) - 1
+ remaining = target_duration - total_speech - intro_pad - outro_pad
+ gap = max(1.0, remaining / max(1, n_gaps))
+
+ # Build timing and concatenate samples
+ timing = []
+ t = intro_pad
+ all_audio = [np.zeros(int(sr * intro_pad), dtype=np.int16)]
+
+ for i, dur in enumerate(durations):
+ wav = tts_clips[i].replace(".mp3", ".wav")
+ with wave.open(wav) as wf:
+ samples = np.frombuffer(wf.readframes(wf.getnframes()), dtype=np.int16)
+ timing.append((t, t + dur, i))
+ all_audio.append(samples)
+ t += dur
+ if i < len(tts_clips) - 1:
+ all_audio.append(np.zeros(int(sr * gap), dtype=np.int16))
+ t += gap
+
+ all_audio.append(np.zeros(int(sr * outro_pad), dtype=np.int16))
+
+ # Pad or trim to exactly target_duration
+ full = np.concatenate(all_audio)
+ target_samples = int(sr * target_duration)
+ if len(full) < target_samples:
+ full = np.pad(full, (0, target_samples - len(full)))
+ else:
+ full = full[:target_samples]
+
+ # Write concatenated TTS track
+ with wave.open("tts_full.wav", "w") as wf:
+ wf.setnchannels(1)
+ wf.setsampwidth(2)
+ wf.setframerate(sr)
+ wf.writeframes(full.tobytes())
+
+ return timing
+```
+
+### Audio Mixing
+
+Mix TTS (center) with background music (wide stereo, low volume). The filter chain:
+1. TTS mono duplicated to both channels (centered)
+2. BGM loudness-normalized, volume reduced to 15%, stereo widened with `extrastereo`
+3. Mixed together with dropout transition for smooth endings
+
+```python
+def mix_audio(tts_path, bgm_path, output_path, bgm_volume=0.15):
+ """Mix TTS centered with BGM panned wide stereo."""
+ filter_complex = (
+ # TTS: mono -> stereo center
+ "[0:a]aformat=sample_fmts=fltp:sample_rates=44100:channel_layouts=mono,"
+ "pan=stereo|c0=c0|c1=c0[tts];"
+ # BGM: normalize loudness, reduce volume, widen stereo
+ f"[1:a]aformat=sample_fmts=fltp:sample_rates=44100:channel_layouts=stereo,"
+ f"loudnorm=I=-16:TP=-1.5:LRA=11,"
+ f"volume={bgm_volume},"
+ f"extrastereo=m=2.5[bgm];"
+ # Mix with smooth dropout at end
+ "[tts][bgm]amix=inputs=2:duration=longest:dropout_transition=3,"
+ "aformat=sample_fmts=s16:sample_rates=44100:channel_layouts=stereo[out]"
+ )
+ cmd = [
+ "ffmpeg", "-y",
+ "-i", tts_path,
+ "-i", bgm_path,
+ "-filter_complex", filter_complex,
+ "-map", "[out]", output_path,
+ ]
+ subprocess.run(cmd, capture_output=True, check=True)
+```
+
+### Per-Quote Visual Style
+
+Cycle through visual presets per quote for variety. Each preset defines a background effect, color scheme, and text color:
+
+```python
+QUOTE_STYLES = [
+ {"hue": 0.08, "accent": 0.7, "bg": "spiral", "text_rgb": (255, 220, 140)}, # warm gold
+ {"hue": 0.55, "accent": 0.6, "bg": "rings", "text_rgb": (180, 220, 255)}, # cool blue
+ {"hue": 0.75, "accent": 0.7, "bg": "wave", "text_rgb": (220, 180, 255)}, # purple
+ {"hue": 0.35, "accent": 0.6, "bg": "matrix", "text_rgb": (140, 255, 180)}, # green
+ {"hue": 0.95, "accent": 0.8, "bg": "fire", "text_rgb": (255, 180, 160)}, # red/coral
+ {"hue": 0.12, "accent": 0.5, "bg": "interference", "text_rgb": (255, 240, 200)}, # amber
+ {"hue": 0.60, "accent": 0.7, "bg": "tunnel", "text_rgb": (160, 210, 255)}, # cyan
+ {"hue": 0.45, "accent": 0.6, "bg": "aurora", "text_rgb": (180, 255, 220)}, # teal
+]
+
+style = QUOTE_STYLES[quote_index % len(QUOTE_STYLES)]
+```
+
+This guarantees no two adjacent quotes share the same look, even without randomness.
+
+### Typewriter Text Rendering
+
+Display quote text character-by-character synced to speech progress. Recently revealed characters are brighter, creating a "just typed" glow:
+
+```python
+def render_typewriter(ch, co, lines, block_start, cols, progress, total_chars, text_rgb, t):
+ """Overlay typewriter text onto character/color grids.
+ progress: 0.0 (nothing visible) to 1.0 (all text visible)."""
+ chars_visible = int(total_chars * min(1.0, progress * 1.2)) # slight overshoot for snappy feel
+ tr, tg, tb = text_rgb
+ char_count = 0
+ for li, line in enumerate(lines):
+ row = block_start + li
+ col = (cols - len(line)) // 2
+ for ci, c in enumerate(line):
+ if char_count < chars_visible:
+ age = chars_visible - char_count
+ bri_factor = min(1.0, 0.5 + 0.5 / (1 + age * 0.015)) # newer = brighter
+ hue_shift = math.sin(char_count * 0.3 + t * 2) * 0.05
+ stamp(ch, co, c, row, col + ci,
+ (int(min(255, tr * bri_factor * (1.0 + hue_shift))),
+ int(min(255, tg * bri_factor)),
+ int(min(255, tb * bri_factor * (1.0 - hue_shift)))))
+ char_count += 1
+
+ # Blinking cursor at insertion point
+ if progress < 1.0 and int(t * 3) % 2 == 0:
+ # Find cursor position (char_count == chars_visible)
+ cc = 0
+ for li, line in enumerate(lines):
+ for ci, c in enumerate(line):
+ if cc == chars_visible:
+ stamp(ch, co, "\u258c", block_start + li,
+ (cols - len(line)) // 2 + ci, (255, 220, 100))
+ return
+ cc += 1
+```
+
+### Feature Analysis on Mixed Audio
+
+Run the standard audio analysis (FFT, beat detection) on the final mixed track so visual effects react to both TTS and music:
+
+```python
+# Analyze mixed_final.wav (not individual tracks)
+features = analyze_audio("mixed_final.wav", fps=24)
+```
+
+Visuals pulse with both the music beats and the speech energy.
+
+---
+
+## Audio-Video Sync Verification
+
+After rendering, verify that visual beat markers align with actual audio beats. Drift accumulates from frame timing errors, ffmpeg concat boundaries, and rounding in `fi / fps`.
+
+### Beat Timestamp Extraction
+
+```python
+def extract_beat_timestamps(features, fps, threshold=0.5):
+ """Extract timestamps where beat feature exceeds threshold."""
+ beat = features["beat"]
+ timestamps = []
+ for fi in range(len(beat)):
+ if beat[fi] > threshold:
+ timestamps.append(fi / fps)
+ return timestamps
+
+def extract_visual_beat_timestamps(video_path, fps, brightness_jump=30):
+ """Detect visual beats by brightness jumps between consecutive frames.
+ Returns timestamps where mean brightness increases by more than threshold."""
+ import subprocess
+ cmd = ["ffmpeg", "-i", video_path, "-f", "rawvideo", "-pix_fmt", "gray", "-"]
+ proc = subprocess.run(cmd, capture_output=True)
+ frames = np.frombuffer(proc.stdout, dtype=np.uint8)
+ # Infer frame dimensions from total byte count
+ n_pixels = len(frames)
+ # For 1080p: 1920*1080 pixels per frame
+ # Auto-detect from video metadata is more robust:
+ probe = subprocess.run(
+ ["ffprobe", "-v", "error", "-select_streams", "v:0",
+ "-show_entries", "stream=width,height",
+ "-of", "csv=p=0", video_path],
+ capture_output=True, text=True)
+ w, h = map(int, probe.stdout.strip().split(","))
+ ppf = w * h # pixels per frame
+ n_frames = n_pixels // ppf
+ frames = frames[:n_frames * ppf].reshape(n_frames, ppf)
+ means = frames.mean(axis=1)
+
+ timestamps = []
+ for i in range(1, len(means)):
+ if means[i] - means[i-1] > brightness_jump:
+ timestamps.append(i / fps)
+ return timestamps
+```
+
+### Sync Report
+
+```python
+def sync_report(audio_beats, visual_beats, tolerance_ms=50):
+ """Compare audio beat timestamps to visual beat timestamps.
+
+ Args:
+ audio_beats: list of timestamps (seconds) from audio analysis
+ visual_beats: list of timestamps (seconds) from video brightness analysis
+ tolerance_ms: max acceptable drift in milliseconds
+
+ Returns:
+ dict with matched/unmatched/drift statistics
+ """
+ tolerance = tolerance_ms / 1000.0
+ matched = []
+ unmatched_audio = []
+ unmatched_visual = list(visual_beats)
+
+ for at in audio_beats:
+ best_match = None
+ best_delta = float("inf")
+ for vt in unmatched_visual:
+ delta = abs(at - vt)
+ if delta < best_delta:
+ best_delta = delta
+ best_match = vt
+ if best_match is not None and best_delta < tolerance:
+ matched.append({"audio": at, "visual": best_match, "drift_ms": best_delta * 1000})
+ unmatched_visual.remove(best_match)
+ else:
+ unmatched_audio.append(at)
+
+ drifts = [m["drift_ms"] for m in matched]
+ return {
+ "matched": len(matched),
+ "unmatched_audio": len(unmatched_audio),
+ "unmatched_visual": len(unmatched_visual),
+ "total_audio_beats": len(audio_beats),
+ "total_visual_beats": len(visual_beats),
+ "mean_drift_ms": np.mean(drifts) if drifts else 0,
+ "max_drift_ms": np.max(drifts) if drifts else 0,
+ "p95_drift_ms": np.percentile(drifts, 95) if len(drifts) > 1 else 0,
+ }
+
+# Usage:
+audio_beats = extract_beat_timestamps(features, fps=24)
+visual_beats = extract_visual_beat_timestamps("output.mp4", fps=24)
+report = sync_report(audio_beats, visual_beats)
+print(f"Matched: {report['matched']}/{report['total_audio_beats']} beats")
+print(f"Mean drift: {report['mean_drift_ms']:.1f}ms, Max: {report['max_drift_ms']:.1f}ms")
+# Target: mean drift < 20ms, max drift < 42ms (1 frame at 24fps)
+```
+
+### Common Sync Issues
+
+| Symptom | Cause | Fix |
+|---------|-------|-----|
+| Consistent late visual beats | ffmpeg concat adds frames at boundaries | Use `-vsync cfr` flag; pad segments to exact frame count |
+| Drift increases over time | Floating-point accumulation in `t = fi / fps` | Use integer frame counter, compute `t` fresh each frame |
+| Random missed beats | Beat threshold too high / feature smoothing too aggressive | Lower threshold; reduce EMA alpha for beat feature |
+| Beats land on wrong frame | Off-by-one in frame indexing | Verify: frame 0 = t=0, frame 1 = t=1/fps (not t=0) |
diff --git a/src/skills/bundled/ascii-video/references/optimization.md b/src/skills/bundled/ascii-video/references/optimization.md
new file mode 100644
index 0000000..8813080
--- /dev/null
+++ b/src/skills/bundled/ascii-video/references/optimization.md
@@ -0,0 +1,688 @@
+# Optimization Reference
+
+> **See also:** architecture.md · composition.md · scenes.md · shaders.md · inputs.md · troubleshooting.md
+
+## Hardware Detection
+
+Detect the user's hardware at script startup and adapt rendering parameters automatically. Never hardcode worker counts or resolution.
+
+### CPU and Memory Detection
+
+```python
+import multiprocessing
+import platform
+import shutil
+import os
+
+def detect_hardware():
+ """Detect hardware capabilities and return render config."""
+ cpu_count = multiprocessing.cpu_count()
+
+ # Leave 1-2 cores free for OS + ffmpeg encoding
+ if cpu_count >= 16:
+ workers = cpu_count - 2
+ elif cpu_count >= 8:
+ workers = cpu_count - 1
+ elif cpu_count >= 4:
+ workers = cpu_count - 1
+ else:
+ workers = max(1, cpu_count)
+
+ # Memory detection (platform-specific)
+ try:
+ if platform.system() == "Darwin":
+ import subprocess
+ mem_bytes = int(subprocess.check_output(["sysctl", "-n", "hw.memsize"]).strip())
+ elif platform.system() == "Linux":
+ with open("/proc/meminfo") as f:
+ for line in f:
+ if line.startswith("MemTotal"):
+ mem_bytes = int(line.split()[1]) * 1024
+ break
+ else:
+ mem_bytes = 8 * 1024**3 # assume 8GB on unknown
+ except Exception:
+ mem_bytes = 8 * 1024**3
+
+ mem_gb = mem_bytes / (1024**3)
+
+ # Each worker uses ~50-150MB depending on grid sizes
+ # Cap workers if memory is tight
+ mem_per_worker_mb = 150
+ max_workers_by_mem = int(mem_gb * 1024 * 0.6 / mem_per_worker_mb) # use 60% of RAM
+ workers = min(workers, max_workers_by_mem)
+
+ # ffmpeg availability and codec support
+ has_ffmpeg = shutil.which("ffmpeg") is not None
+
+ return {
+ "cpu_count": cpu_count,
+ "workers": workers,
+ "mem_gb": mem_gb,
+ "platform": platform.system(),
+ "arch": platform.machine(),
+ "has_ffmpeg": has_ffmpeg,
+ }
+```
+
+### Adaptive Quality Profiles
+
+Scale resolution, FPS, CRF, and grid density based on hardware:
+
+```python
+def quality_profile(hw, target_duration_s, user_preference="auto"):
+ """
+ Returns render settings adapted to hardware.
+ user_preference: "auto", "draft", "preview", "production", "max"
+ """
+ if user_preference == "draft":
+ return {"vw": 960, "vh": 540, "fps": 12, "crf": 28, "workers": min(4, hw["workers"]),
+ "grid_scale": 0.5, "shaders": "minimal", "particles_max": 200}
+
+ if user_preference == "preview":
+ return {"vw": 1280, "vh": 720, "fps": 15, "crf": 25, "workers": hw["workers"],
+ "grid_scale": 0.75, "shaders": "standard", "particles_max": 500}
+
+ if user_preference == "max":
+ return {"vw": 3840, "vh": 2160, "fps": 30, "crf": 15, "workers": hw["workers"],
+ "grid_scale": 2.0, "shaders": "full", "particles_max": 3000}
+
+ # "production" or "auto"
+ # Auto-detect: estimate render time, downgrade if it would take too long
+ n_frames = int(target_duration_s * 24)
+ est_seconds_per_frame = 0.18 # ~180ms at 1080p
+ est_total_s = n_frames * est_seconds_per_frame / max(1, hw["workers"])
+
+ if hw["mem_gb"] < 4 or hw["cpu_count"] <= 2:
+ # Low-end: 720p, 15fps
+ return {"vw": 1280, "vh": 720, "fps": 15, "crf": 23, "workers": hw["workers"],
+ "grid_scale": 0.75, "shaders": "standard", "particles_max": 500}
+
+ if est_total_s > 3600: # would take over an hour
+ # Downgrade to 720p to speed up
+ return {"vw": 1280, "vh": 720, "fps": 24, "crf": 20, "workers": hw["workers"],
+ "grid_scale": 0.75, "shaders": "standard", "particles_max": 800}
+
+ # Standard production: 1080p 24fps
+ return {"vw": 1920, "vh": 1080, "fps": 24, "crf": 20, "workers": hw["workers"],
+ "grid_scale": 1.0, "shaders": "full", "particles_max": 1200}
+
+
+def apply_quality_profile(profile):
+ """Set globals from quality profile."""
+ global VW, VH, FPS, N_WORKERS
+ VW = profile["vw"]
+ VH = profile["vh"]
+ FPS = profile["fps"]
+ N_WORKERS = profile["workers"]
+ # Grid sizes scale with resolution
+ # CRF passed to ffmpeg encoder
+ # Shader set determines which post-processing is active
+```
+
+### CLI Integration
+
+```python
+parser = argparse.ArgumentParser()
+parser.add_argument("--quality", choices=["draft", "preview", "production", "max", "auto"],
+ default="auto", help="Render quality preset")
+parser.add_argument("--aspect", choices=["landscape", "portrait", "square"],
+ default="landscape", help="Aspect ratio preset")
+parser.add_argument("--workers", type=int, default=0, help="Override worker count (0=auto)")
+parser.add_argument("--resolution", type=str, default="", help="Override resolution e.g. 1280x720")
+args = parser.parse_args()
+
+hw = detect_hardware()
+if args.workers > 0:
+ hw["workers"] = args.workers
+profile = quality_profile(hw, target_duration, args.quality)
+
+# Apply aspect ratio preset (before manual resolution override)
+ASPECT_PRESETS = {
+ "landscape": (1920, 1080),
+ "portrait": (1080, 1920),
+ "square": (1080, 1080),
+}
+if args.aspect != "landscape" and not args.resolution:
+ profile["vw"], profile["vh"] = ASPECT_PRESETS[args.aspect]
+
+if args.resolution:
+ w, h = args.resolution.split("x")
+ profile["vw"], profile["vh"] = int(w), int(h)
+apply_quality_profile(profile)
+
+log(f"Hardware: {hw['cpu_count']} cores, {hw['mem_gb']:.1f}GB RAM, {hw['platform']}")
+log(f"Render: {profile['vw']}x{profile['vh']} @{profile['fps']}fps, "
+ f"CRF {profile['crf']}, {profile['workers']} workers")
+```
+
+### Portrait Mode Considerations
+
+Portrait (1080x1920) has the same pixel count as landscape 1080p, so performance is equivalent. But composition patterns differ:
+
+| Concern | Landscape | Portrait |
+|---------|-----------|----------|
+| Grid cols at `lg` | 160 | 90 |
+| Grid rows at `lg` | 45 | 80 |
+| Max text line chars | ~50 centered | ~25-30 centered |
+| Vertical rain | Short travel | Long, dramatic travel |
+| Horizontal spectrum | Full width | Needs rotation or compression |
+| Radial effects | Natural circles | Tall ellipses (aspect correction handles this) |
+| Particle explosions | Wide spread | Tall spread |
+| Text stacking | 3-4 lines comfortable | 8-10 lines comfortable |
+| Quote layout | 2-3 wide lines | 5-6 short lines |
+
+**Portrait-optimized patterns:**
+- Vertical rain/matrix effects are naturally enhanced — longer column travel
+- Fire columns rise through more screen space
+- Rising embers/particles have more vertical runway
+- Text can be stacked more aggressively with more lines
+- Radial effects work if aspect correction is applied (GridLayer handles this automatically)
+- Spectrum bars can be rotated 90 degrees (vertical bars from bottom)
+
+**Portrait text layout:**
+```python
+def layout_text_portrait(text, max_chars_per_line=25, grid=None):
+ """Break text into short lines for portrait display."""
+ words = text.split()
+ lines = []; current = ""
+ for w in words:
+ if len(current) + len(w) + 1 > max_chars_per_line:
+ lines.append(current.strip())
+ current = w + " "
+ else:
+ current += w + " "
+ if current.strip():
+ lines.append(current.strip())
+ return lines
+```
+
+## Performance Budget
+
+Target: 100-200ms per frame (5-10 fps single-threaded, 40-80 fps across 8 workers).
+
+| Component | Time | Notes |
+|-----------|------|-------|
+| Feature extraction | 1-5ms | Pre-computed for all frames before render |
+| Effect function | 2-15ms | Vectorized numpy, avoid Python loops |
+| Character render | 80-150ms | **Bottleneck** -- per-cell Python loop |
+| Shader pipeline | 5-25ms | Depends on active shaders |
+| ffmpeg encode | ~5ms | Amortized by pipe buffering |
+
+## Bitmap Pre-Rasterization
+
+Rasterize every character at init, not per-frame:
+
+```python
+# At init time -- done once
+for c in all_characters:
+ img = Image.new("L", (cell_w, cell_h), 0)
+ ImageDraw.Draw(img).text((0, 0), c, fill=255, font=font)
+ bitmaps[c] = np.array(img, dtype=np.float32) / 255.0 # float32 for fast multiply
+
+# At render time -- fast lookup
+bitmap = bitmaps[char]
+canvas[y:y+ch, x:x+cw] = np.maximum(canvas[y:y+ch, x:x+cw],
+ (bitmap[:,:,None] * color).astype(np.uint8))
+```
+
+Collect all characters from all palettes + overlay text into the init set. Lazy-init for any missed characters.
+
+## Pre-Rendered Background Textures
+
+Alternative to `_render_vf()` for backgrounds where characters don't need to change every frame. Pre-bake a static ASCII texture once at init, then multiply by a per-cell color field each frame. One matrix multiply vs thousands of bitmap blits.
+
+Use when: background layer uses a fixed character palette and only color/brightness varies per frame. NOT suitable for layers where character selection depends on a changing value field.
+
+### Init: Bake the Texture
+
+```python
+# In GridLayer.__init__:
+self._bg_row_idx = np.clip(
+ (np.arange(VH) - self.oy) // self.ch, 0, self.rows - 1
+)
+self._bg_col_idx = np.clip(
+ (np.arange(VW) - self.ox) // self.cw, 0, self.cols - 1
+)
+self._bg_textures = {}
+
+def make_bg_texture(self, palette):
+ """Pre-render a static ASCII texture (grayscale float32) once."""
+ if palette not in self._bg_textures:
+ texture = np.zeros((VH, VW), dtype=np.float32)
+ rng = random.Random(12345)
+ ch_list = [c for c in palette if c != " " and c in self.bm]
+ if not ch_list:
+ ch_list = list(self.bm.keys())[:5]
+ for row in range(self.rows):
+ y = self.oy + row * self.ch
+ if y + self.ch > VH:
+ break
+ for col in range(self.cols):
+ x = self.ox + col * self.cw
+ if x + self.cw > VW:
+ break
+ bm = self.bm[rng.choice(ch_list)]
+ texture[y:y+self.ch, x:x+self.cw] = bm
+ self._bg_textures[palette] = texture
+ return self._bg_textures[palette]
+```
+
+### Render: Color Field x Cached Texture
+
+```python
+def render_bg(self, color_field, palette=PAL_CIRCUIT):
+ """Fast background: pre-rendered ASCII texture * per-cell color field.
+ color_field: (rows, cols, 3) uint8. Returns (VH, VW, 3) uint8."""
+ texture = self.make_bg_texture(palette)
+ # Expand cell colors to pixel coords via pre-computed index maps
+ color_px = color_field[
+ self._bg_row_idx[:, None], self._bg_col_idx[None, :]
+ ].astype(np.float32)
+ return (texture[:, :, None] * color_px).astype(np.uint8)
+```
+
+### Usage in a Scene
+
+```python
+# Build per-cell color from effect fields (cheap — rows*cols, not VH*VW)
+hue = ((t * 0.05 + val * 0.2) % 1.0).astype(np.float32)
+R, G, B = hsv2rgb(hue, np.full_like(val, 0.5), val)
+color_field = mkc(R, G, B, g.rows, g.cols) # (rows, cols, 3) uint8
+
+# Render background — single matrix multiply, no per-cell loop
+canvas_bg = g.render_bg(color_field, PAL_DENSE)
+```
+
+The texture init loop runs once and is cached per palette. Per-frame cost is one fancy-index lookup + one broadcast multiply — orders of magnitude faster than the per-cell bitmap blit loop in `render()` for dense backgrounds.
+
+## Coordinate Array Caching
+
+Pre-compute all grid-relative coordinate arrays at init, not per-frame:
+
+```python
+# These are O(rows*cols) and used in every effect
+self.rr = np.arange(rows)[:, None] # row indices
+self.cc = np.arange(cols)[None, :] # col indices
+self.dist = np.sqrt(dx**2 + dy**2) # distance from center
+self.angle = np.arctan2(dy, dx) # angle from center
+self.dist_n = ... # normalized distance
+```
+
+## Vectorized Effect Patterns
+
+### Avoid Per-Cell Python Loops in Effects
+
+The render loop (compositing bitmaps) is unavoidably per-cell. But effect functions must be fully vectorized numpy -- never iterate over rows/cols in Python.
+
+Bad (O(rows*cols) Python loop):
+```python
+for r in range(rows):
+ for c in range(cols):
+ val[r, c] = math.sin(c * 0.1 + t) * math.cos(r * 0.1 - t)
+```
+
+Good (vectorized):
+```python
+val = np.sin(g.cc * 0.1 + t) * np.cos(g.rr * 0.1 - t)
+```
+
+### Vectorized Matrix Rain
+
+The naive per-column per-trail-pixel loop is the second biggest bottleneck after the render loop. Use numpy fancy indexing:
+
+```python
+# Instead of nested Python loops over columns and trail pixels:
+# Build row index arrays for all active trail pixels at once
+all_rows = []
+all_cols = []
+all_fades = []
+for c in range(cols):
+ head = int(S["ry"][c])
+ trail_len = S["rln"][c]
+ for i in range(trail_len):
+ row = head - i
+ if 0 <= row < rows:
+ all_rows.append(row)
+ all_cols.append(c)
+ all_fades.append(1.0 - i / trail_len)
+
+# Vectorized assignment
+ar = np.array(all_rows)
+ac = np.array(all_cols)
+af = np.array(all_fades, dtype=np.float32)
+# Assign chars and colors in bulk using fancy indexing
+ch[ar, ac] = ... # vectorized char assignment
+co[ar, ac, 1] = (af * bri * 255).astype(np.uint8) # green channel
+```
+
+### Vectorized Fire Columns
+
+Same pattern -- accumulate index arrays, assign in bulk:
+
+```python
+fire_val = np.zeros((rows, cols), dtype=np.float32)
+for fi in range(n_cols):
+ fx_c = int((fi * cols / n_cols + np.sin(t * 2 + fi * 0.7) * 3) % cols)
+ height = int(energy * rows * 0.7)
+ dy = np.arange(min(height, rows))
+ fr = rows - 1 - dy
+ frac = dy / max(height, 1)
+ # Width spread: base columns wider at bottom
+ for dx in range(-1, 2): # 3-wide columns
+ c = fx_c + dx
+ if 0 <= c < cols:
+ fire_val[fr, c] = np.maximum(fire_val[fr, c],
+ (1 - frac * 0.6) * (0.5 + rms * 0.5))
+# Now map fire_val to chars and colors in one vectorized pass
+```
+
+## PIL String Rendering for Text-Heavy Scenes
+
+Alternative to per-cell bitmap blitting when rendering many long text strings (scrolling tickers, typewriter sequences, idea floods). Uses PIL's native `ImageDraw.text()` which renders an entire string in one C call, vs one Python-loop bitmap blit per character.
+
+Typical win: a scene with 56 ticker rows renders 56 PIL `text()` calls instead of ~10K individual bitmap blits.
+
+Use when: scene renders many rows of readable text strings. NOT suitable for sparse or spatially-scattered single characters (use normal `render()` for those).
+
+```python
+from PIL import Image, ImageDraw
+
+def render_text_layer(grid, rows_data, font):
+ """Render dense text rows via PIL instead of per-cell bitmap blitting.
+
+ Args:
+ grid: GridLayer instance (for oy, ch, ox, font metrics)
+ rows_data: list of (row_index, text_string, rgb_tuple) — one per row
+ font: PIL ImageFont instance (grid.font)
+
+ Returns:
+ uint8 array (VH, VW, 3) — canvas with rendered text
+ """
+ img = Image.new("RGB", (VW, VH), (0, 0, 0))
+ draw = ImageDraw.Draw(img)
+ for row_idx, text, color in rows_data:
+ y = grid.oy + row_idx * grid.ch
+ if y + grid.ch > VH:
+ break
+ draw.text((grid.ox, y), text, fill=color, font=font)
+ return np.array(img)
+```
+
+### Usage in a Ticker Scene
+
+```python
+# Build ticker data (text + color per row)
+rows_data = []
+for row in range(n_tickers):
+ text = build_ticker_text(row, t) # scrolling substring
+ color = hsv2rgb_scalar(hue, 0.85, bri) # (R, G, B) tuple
+ rows_data.append((row, text, color))
+
+# One PIL pass instead of thousands of bitmap blits
+canvas_tickers = render_text_layer(g_md, rows_data, g_md.font)
+
+# Blend with other layers normally
+result = blend_canvas(canvas_bg, canvas_tickers, "screen", 0.9)
+```
+
+This is purely a rendering optimization — same visual output, fewer draw calls. The grid's `render()` method is still needed for sparse character fields where characters are placed individually based on value fields.
+
+## Bloom Optimization
+
+**Do NOT use `scipy.ndimage.uniform_filter`** -- measured at 424ms/frame.
+
+Use 4x downsample + manual box blur instead -- 84ms/frame (5x faster):
+
+```python
+sm = canvas[::4, ::4].astype(np.float32) # 4x downsample
+br = np.where(sm > threshold, sm, 0)
+for _ in range(3): # 3-pass manual box blur
+ p = np.pad(br, ((1,1),(1,1),(0,0)), mode='edge')
+ br = (p[:-2,:-2] + p[:-2,1:-1] + p[:-2,2:] +
+ p[1:-1,:-2] + p[1:-1,1:-1] + p[1:-1,2:] +
+ p[2:,:-2] + p[2:,1:-1] + p[2:,2:]) / 9.0
+bl = np.repeat(np.repeat(br, 4, axis=0), 4, axis=1)[:H, :W]
+```
+
+## Vignette Caching
+
+Distance field is resolution- and strength-dependent, never changes per frame:
+
+```python
+_vig_cache = {}
+def sh_vignette(canvas, strength):
+ key = (canvas.shape[0], canvas.shape[1], round(strength, 2))
+ if key not in _vig_cache:
+ Y = np.linspace(-1, 1, H)[:, None]
+ X = np.linspace(-1, 1, W)[None, :]
+ _vig_cache[key] = np.clip(1.0 - np.sqrt(X**2+Y**2) * strength, 0.15, 1).astype(np.float32)
+ return np.clip(canvas * _vig_cache[key][:,:,None], 0, 255).astype(np.uint8)
+```
+
+Same pattern for CRT barrel distortion (cache remap coordinates).
+
+## Film Grain Optimization
+
+Generate noise at half resolution, tile up:
+
+```python
+noise = np.random.randint(-amt, amt+1, (H//2, W//2, 1), dtype=np.int16)
+noise = np.repeat(np.repeat(noise, 2, axis=0), 2, axis=1)[:H, :W]
+```
+
+2x blocky grain looks like film grain and costs 1/4 the random generation.
+
+## Parallel Rendering
+
+### Worker Architecture
+
+```python
+hw = detect_hardware()
+N_WORKERS = hw["workers"]
+
+# Batch splitting (for non-clip architectures)
+batch_size = (n_frames + N_WORKERS - 1) // N_WORKERS
+batches = [(i, i*batch_size, min((i+1)*batch_size, n_frames), features, seg_path) ...]
+
+with multiprocessing.Pool(N_WORKERS) as pool:
+ segments = pool.starmap(render_batch, batches)
+```
+
+### Per-Clip Parallelism (Preferred for Segmented Videos)
+
+```python
+from concurrent.futures import ProcessPoolExecutor, as_completed
+
+with ProcessPoolExecutor(max_workers=N_WORKERS) as pool:
+ futures = {pool.submit(render_clip, seg, features, path): seg["id"]
+ for seg, path in clip_args}
+ for fut in as_completed(futures):
+ clip_id = futures[fut]
+ try:
+ fut.result()
+ log(f" {clip_id} done")
+ except Exception as e:
+ log(f" {clip_id} FAILED: {e}")
+```
+
+### Worker Isolation
+
+Each worker:
+- Creates its own `Renderer` instance (with full grid + bitmap init)
+- Opens its own ffmpeg subprocess
+- Has independent random seed (`random.seed(batch_id * 10000)`)
+- Writes to its own segment file and stderr log
+
+### ffmpeg Pipe Safety
+
+**CRITICAL**: Never `stderr=subprocess.PIPE` with long-running ffmpeg. The stderr buffer fills at ~64KB and deadlocks:
+
+```python
+# WRONG -- will deadlock
+pipe = subprocess.Popen(cmd, stdin=subprocess.PIPE, stderr=subprocess.PIPE)
+
+# RIGHT -- stderr to file
+stderr_fh = open(err_path, "w")
+pipe = subprocess.Popen(cmd, stdin=subprocess.PIPE, stdout=subprocess.DEVNULL, stderr=stderr_fh)
+# ... write all frames ...
+pipe.stdin.close()
+pipe.wait()
+stderr_fh.close()
+```
+
+### Concatenation
+
+```python
+with open(concat_file, "w") as cf:
+ for seg in segments:
+ cf.write(f"file '{seg}'\n")
+
+cmd = ["ffmpeg", "-y", "-f", "concat", "-safe", "0", "-i", concat_file]
+if audio_path:
+ cmd += ["-i", audio_path, "-c:v", "copy", "-c:a", "aac", "-b:a", "192k", "-shortest"]
+else:
+ cmd += ["-c:v", "copy"]
+cmd.append(output_path)
+subprocess.run(cmd, capture_output=True, check=True)
+```
+
+## Particle System Performance
+
+Cap particle counts based on quality profile:
+
+| System | Low | Standard | High |
+|--------|-----|----------|------|
+| Explosion | 300 | 1000 | 2500 |
+| Embers | 500 | 1500 | 3000 |
+| Starfield | 300 | 800 | 1500 |
+| Dissolve | 200 | 600 | 1200 |
+
+Cull by truncating lists:
+```python
+MAX_PARTICLES = profile.get("particles_max", 1200)
+if len(S["px"]) > MAX_PARTICLES:
+ for k in ("px", "py", "vx", "vy", "life", "char"):
+ S[k] = S[k][-MAX_PARTICLES:] # keep newest
+```
+
+## Memory Management
+
+- Feature arrays: pre-computed for all frames, shared across workers via fork semantics (COW)
+- Canvas: allocated once per worker, reused (`np.zeros(...)`)
+- Character arrays: allocated per frame (cheap -- rows*cols U1 strings)
+- Bitmap cache: ~500KB per grid size, initialized once per worker
+
+Total memory per worker: ~50-150MB. Total: ~400-800MB for 8 workers.
+
+For low-memory systems (< 4GB), reduce worker count and use smaller grids.
+
+## Brightness Verification
+
+After render, spot-check brightness at sample timestamps:
+
+```python
+for t in [2, 30, 60, 120, 180]:
+ cmd = ["ffmpeg", "-ss", str(t), "-i", output_path,
+ "-frames:v", "1", "-f", "rawvideo", "-pix_fmt", "rgb24", "-"]
+ r = subprocess.run(cmd, capture_output=True)
+ arr = np.frombuffer(r.stdout, dtype=np.uint8)
+ print(f"t={t}s mean={arr.mean():.1f} max={arr.max()}")
+```
+
+Target: mean > 5 for quiet sections, mean > 15 for active sections. If consistently below, increase brightness floor in effects and/or global boost multiplier.
+
+## Render Time Estimates
+
+Scale with hardware. Baseline: 1080p, 24fps, ~180ms/frame/worker.
+
+| Duration | Frames | 4 workers | 8 workers | 16 workers |
+|----------|--------|-----------|-----------|------------|
+| 30s | 720 | ~3 min | ~2 min | ~1 min |
+| 2 min | 2,880 | ~13 min | ~7 min | ~4 min |
+| 3.5 min | 5,040 | ~23 min | ~12 min | ~6 min |
+| 5 min | 7,200 | ~33 min | ~17 min | ~9 min |
+| 10 min | 14,400 | ~65 min | ~33 min | ~17 min |
+
+At 720p: multiply times by ~0.5. At 4K: multiply by ~4.
+
+Heavier effects (many particles, dense grids, extra shader passes) add ~20-50%.
+
+---
+
+## Temp File Cleanup
+
+Rendering generates intermediate files that accumulate across runs. Clean up after the final concat/mux step.
+
+### Files to Clean
+
+| File type | Source | Location |
+|-----------|--------|----------|
+| WAV extracts | `ffmpeg -i input.mp3 ... tmp.wav` | `tempfile.mktemp()` or project dir |
+| Segment clips | `render_clip()` output | `segments/seg_00.mp4` etc. |
+| Concat list | ffmpeg concat demuxer input | `segments/concat.txt` |
+| ffmpeg stderr logs | piped to file for debugging | `*.log` in project dir |
+| Feature cache | pickled numpy arrays | `*.pkl` or `*.npz` |
+
+### Cleanup Function
+
+```python
+import glob
+import tempfile
+import shutil
+
+def cleanup_render_artifacts(segments_dir="segments", keep_final=True):
+ """Remove intermediate files after successful render.
+
+ Call this AFTER verifying the final output exists and plays correctly.
+
+ Args:
+ segments_dir: directory containing segment clips and concat list
+ keep_final: if True, only delete intermediates (not the final output)
+ """
+ removed = []
+
+ # 1. Segment clips
+ if os.path.isdir(segments_dir):
+ shutil.rmtree(segments_dir)
+ removed.append(f"directory: {segments_dir}")
+
+ # 2. Temporary WAV files
+ for wav in glob.glob("*.wav"):
+ if wav.startswith("tmp") or wav.startswith("extracted_"):
+ os.remove(wav)
+ removed.append(wav)
+
+ # 3. ffmpeg stderr logs
+ for log in glob.glob("ffmpeg_*.log"):
+ os.remove(log)
+ removed.append(log)
+
+ # 4. Feature cache (optional — useful to keep for re-renders)
+ # for cache in glob.glob("features_*.npz"):
+ # os.remove(cache)
+ # removed.append(cache)
+
+ print(f"Cleaned {len(removed)} artifacts: {removed}")
+ return removed
+```
+
+### Integration with Render Pipeline
+
+Call cleanup at the end of the main render script, after the final output is verified:
+
+```python
+# At end of main()
+if os.path.exists(output_path) and os.path.getsize(output_path) > 1000:
+ cleanup_render_artifacts(segments_dir="segments")
+ print(f"Done. Output: {output_path}")
+else:
+ print("WARNING: final output missing or empty — skipping cleanup")
+```
+
+### Temp File Best Practices
+
+- Use `tempfile.mkdtemp()` for segment directories — avoids polluting the project dir
+- Name WAV extracts with `tempfile.mktemp(suffix=".wav")` so they're in the OS temp dir
+- For debugging, set `KEEP_INTERMEDIATES=1` env var to skip cleanup
+- Feature caches (`.npz`) are cheap to store and expensive to recompute — default to keeping them
diff --git a/src/skills/bundled/ascii-video/references/scenes.md b/src/skills/bundled/ascii-video/references/scenes.md
new file mode 100644
index 0000000..818281a
--- /dev/null
+++ b/src/skills/bundled/ascii-video/references/scenes.md
@@ -0,0 +1,1011 @@
+# Scene System & Creative Composition
+
+> **See also:** architecture.md · composition.md · effects.md · shaders.md
+
+## Scene Design Philosophy
+
+Scenes are storytelling units, not effect demos. Every scene needs:
+- A **concept** — what is happening visually? Not "plasma + rings" but "emergence from void" or "crystallization"
+- An **arc** — how does it change over its duration? Build, decay, transform, reveal?
+- A **role** — how does it serve the larger video narrative? Opening tension, peak energy, resolution?
+
+The design patterns below provide compositional techniques. The scene examples show them in practice at increasing complexity. The protocol section covers the technical contract.
+
+Good scene design starts with the concept, then selects effects and parameters that serve it. The design patterns section shows *how* to compose layers intentionally. The examples section shows complete working scenes at every complexity level. The protocol section covers the technical contract that all scenes must follow.
+
+---
+
+## Scene Design Patterns
+
+Higher-order patterns for composing scenes that feel intentional rather than random. These patterns use the existing building blocks (value fields, blend modes, shaders, feedback) but organize them with compositional intent.
+
+## Layer Hierarchy
+
+Every scene should have clear visual layers with distinct roles:
+
+| Layer | Grid | Brightness | Purpose |
+|-------|------|-----------|---------|
+| **Background** | xs or sm (dense) | 0.1–0.25 | Atmosphere, texture. Never competes with content. |
+| **Content** | md (balanced) | 0.4–0.8 | The main visual idea. Carries the scene's concept. |
+| **Accent** | lg or sm (sparse) | 0.5–1.0 (sparse coverage) | Highlights, punctuation, sparse bright points. |
+
+The background sets mood. The content layer is what the scene *is about*. The accent adds visual interest without overwhelming.
+
+```python
+def fx_example(r, f, t, S):
+ local = t
+ progress = min(local / 5.0, 1.0)
+
+ g_bg = r.get_grid("sm")
+ g_main = r.get_grid("md")
+ g_accent = r.get_grid("lg")
+
+ # --- Background: dim atmosphere ---
+ bg_val = vf_smooth_noise(g_bg, f, t * 0.3, S, octaves=2, bri=0.15)
+ # ... render bg to canvas
+
+ # --- Content: the main visual idea ---
+ content_val = vf_spiral(g_main, f, t, S, n_arms=n_arms, tightness=tightness)
+ # ... render content on top of canvas
+
+ # --- Accent: sparse highlights ---
+ accent_val = vf_noise_static(g_accent, f, t, S, density=0.05)
+ # ... render accent on top
+
+ return canvas
+```
+
+## Directional Parameter Arcs
+
+Parameters should *go somewhere* over the scene's duration — not oscillate aimlessly with `sin(t * N)`.
+
+**Bad:** `twist = 3.0 + 2.0 * math.sin(t * 0.6)` — wobbles back and forth, feels aimless.
+
+**Good:** `twist = 2.0 + progress * 5.0` — starts gentle, ends intense. The scene *builds*.
+
+Use `progress = min(local / duration, 1.0)` (0→1 over the scene) to drive directional change:
+
+| Pattern | Formula | Feel |
+|---------|---------|------|
+| Linear ramp | `progress * range` | Steady buildup |
+| Ease-out | `1 - (1 - progress) ** 2` | Fast start, gentle finish |
+| Ease-in | `progress ** 2` | Slow start, accelerating |
+| Step reveal | `np.clip((progress - 0.5) / 0.25, 0, 1)` | Nothing until 50%, then fades in |
+| Build + plateau | `min(1.0, progress * 1.5)` | Reaches full at 67%, holds |
+
+Oscillation is fine for *secondary* parameters (saturation shimmer, hue drift). But the *defining* parameter of the scene should have a direction.
+
+### Examples of Directional Arcs
+
+| Scene concept | Parameter | Arc |
+|--------------|-----------|-----|
+| Emergence | Ring radius | 0 → max (ease-out) |
+| Shatter | Voronoi cell count | 8 → 38 (linear) |
+| Descent | Tunnel speed | 2.0 → 10.0 (linear) |
+| Mandala | Shape complexity | ring → +polygon → +star → +rosette (step reveals) |
+| Crescendo | Layer count | 1 → 7 (staggered entry) |
+| Entropy | Geometry visibility | 1.0 → 0.0 (consumed) |
+
+## Scene Concepts
+
+Each scene should be built around a *visual idea*, not an effect name.
+
+**Bad:** "fx_plasma_cascade" — named after the effect. No concept.
+**Good:** "fx_emergence" — a point of light expands into a field. The name tells you *what happens*.
+
+Good scene concepts have:
+1. A **visual metaphor** (emergence, descent, collision, entropy)
+2. A **directional arc** (things change from A to B, not oscillate)
+3. **Motivated layer choices** (each layer serves the concept)
+4. **Motivated feedback** (transform direction matches the metaphor)
+
+| Concept | Metaphor | Feedback transform | Why |
+|---------|----------|-------------------|-----|
+| Emergence | Birth, expansion | zoom-out | Past frames expand outward |
+| Descent | Falling, acceleration | zoom-in | Past frames rush toward center |
+| Inferno | Rising fire | shift-up | Past frames rise with the flames |
+| Entropy | Decay, dissolution | none | Clean, no persistence — things disappear |
+| Crescendo | Accumulation | zoom + hue_shift | Everything compounds and shifts |
+
+## Compositional Techniques
+
+### Counter-Rotating Dual Systems
+
+Two instances of the same effect rotating in opposite directions create visual interference:
+
+```python
+# Primary spiral (clockwise)
+s1_val = vf_spiral(g_main, f, t * 1.5, S, n_arms=n_arms_1, tightness=tightness_1)
+
+# Counter-rotating spiral (counter-clockwise via negative time)
+s2_val = vf_spiral(g_accent, f, -t * 1.2, S, n_arms=n_arms_2, tightness=tightness_2)
+
+# Screen blend creates bright interference at crossing points
+canvas = blend_canvas(canvas_with_s1, c2, "screen", 0.7)
+```
+
+Works with spirals, vortexes, rings. The counter-rotation creates constantly shifting interference patterns.
+
+### Wave Collision
+
+Two wave fronts converging from opposite sides, meeting at a collision point:
+
+```python
+collision_phase = abs(progress - 0.5) * 2 # 1→0→1 (0 at collision)
+
+# Wave A approaches from left
+offset_a = (1 - progress) * g.cols * 0.4
+wave_a = np.sin((g.cc + offset_a) * 0.08 + t * 2) * 0.5 + 0.5
+
+# Wave B approaches from right
+offset_b = -(1 - progress) * g.cols * 0.4
+wave_b = np.sin((g.cc + offset_b) * 0.08 - t * 2) * 0.5 + 0.5
+
+# Interference peaks at collision
+combined = wave_a * 0.5 + wave_b * 0.5 + np.abs(wave_a - wave_b) * (1 - collision_phase) * 0.5
+```
+
+### Progressive Fragmentation
+
+Voronoi with cell count increasing over time — visual shattering:
+
+```python
+n_pts = int(8 + progress * 30) # 8 cells → 38 cells
+# Pre-generate enough points, slice to n_pts
+px = base_x[:n_pts] + np.sin(t * 0.3 + np.arange(n_pts) * 0.7) * (3 + progress * 3)
+```
+
+The edge glow width can also increase with progress to emphasize the cracks.
+
+### Entropy / Consumption
+
+A clean geometric pattern being overtaken by an organic process:
+
+```python
+# Geometry fades out
+geo_val = clean_pattern * max(0.05, 1.0 - progress * 0.9)
+
+# Organic process grows in
+rd_val = vf_reaction_diffusion(g, f, t, S) * min(1.0, progress * 1.5)
+
+# Render geometry first, organic on top — organic consumes geometry
+```
+
+### Staggered Layer Entry (Crescendo)
+
+Layers enter one at a time, building to overwhelming density:
+
+```python
+def layer_strength(enter_t, ramp=1.5):
+ """0.0 until enter_t, ramps to 1.0 over ramp seconds."""
+ return max(0.0, min(1.0, (local - enter_t) / ramp))
+
+# Layer 1: always present
+s1 = layer_strength(0.0)
+# Layer 2: enters at 2s
+s2 = layer_strength(2.0)
+# Layer 3: enters at 4s
+s3 = layer_strength(4.0)
+# ... etc
+
+# Each layer uses a different effect, grid, palette, and blend mode
+# Screen blend between layers so they accumulate light
+```
+
+For a 15-second crescendo, 7 layers entering every 2 seconds works well. Use different blend modes (screen for most, add for energy, colordodge for the final wash).
+
+## Scene Ordering
+
+For a multi-scene reel or video:
+- **Vary mood between adjacent scenes** — don't put two calm scenes next to each other
+- **Randomize order** rather than grouping by type — prevents "effect demo" feel
+- **End on the strongest scene** — crescendo or something with a clear payoff
+- **Open with energy** — grab attention in the first 2 seconds
+
+---
+
+## Scene Protocol
+
+Scenes are the top-level creative unit. Each scene is a time-bounded segment with its own effect function, shader chain, feedback configuration, and tone-mapping gamma.
+
+### Scene Protocol (v2)
+
+### Function Signature
+
+```python
+def fx_scene_name(r, f, t, S) -> canvas:
+ """
+ Args:
+ r: Renderer instance — access multiple grids via r.get_grid("sm")
+ f: dict of audio/video features, all values normalized to [0, 1]
+ t: time in seconds — local to scene (0.0 at scene start)
+ S: dict for persistent state (particles, rain columns, etc.)
+
+ Returns:
+ canvas: numpy uint8 array, shape (VH, VW, 3) — full pixel frame
+ """
+```
+
+**Local time convention:** Scene functions receive `t` starting at 0.0 for the first frame of the scene, regardless of where the scene appears in the timeline. The render loop subtracts the scene's start time before calling the function:
+
+```python
+# In render_clip:
+t_local = fi / FPS - scene_start
+canvas = fx_fn(r, feat, t_local, S)
+```
+
+This makes scenes reorderable without modifying their code. Compute scene progress as:
+
+```python
+progress = min(t / scene_duration, 1.0) # 0→1 over the scene
+```
+
+This replaces the v1 protocol where scenes returned `(chars, colors)` tuples. The v2 protocol gives scenes full control over multi-grid rendering and pixel-level composition internally.
+
+### The Renderer Class
+
+```python
+class Renderer:
+ def __init__(self):
+ self.grids = {} # lazy-initialized grid cache
+ self.g = None # "active" grid (for backward compat)
+ self.S = {} # persistent state dict
+
+ def get_grid(self, key):
+ """Get or create a GridLayer by size key."""
+ if key not in self.grids:
+ sizes = {"xs": 8, "sm": 10, "md": 16, "lg": 20, "xl": 24, "xxl": 40}
+ self.grids[key] = GridLayer(FONT_PATH, sizes[key])
+ return self.grids[key]
+
+ def set_grid(self, key):
+ """Set active grid (legacy). Prefer get_grid() for multi-grid scenes."""
+ self.g = self.get_grid(key)
+ return self.g
+```
+
+**Key difference from v1**: scenes call `r.get_grid("sm")`, `r.get_grid("lg")`, etc. to access multiple grids. Each grid is lazy-initialized and cached. The `set_grid()` method still works for single-grid scenes.
+
+### Minimal Scene (Single Grid)
+
+```python
+def fx_simple_rings(r, f, t, S):
+ """Single-grid scene: rings with distance-mapped hue."""
+ canvas = _render_vf(r, "md",
+ lambda g, f, t, S: vf_rings(g, f, t, S, n_base=8, spacing_base=3),
+ hf_distance(0.3, 0.02), PAL_STARS, f, t, S, sat=0.85)
+ return canvas
+```
+
+### Standard Scene (Two Grids + Blend)
+
+```python
+def fx_tunnel_ripple(r, f, t, S):
+ """Two-grid scene: tunnel depth exclusion-blended with ripple."""
+ canvas_a = _render_vf(r, "md",
+ lambda g, f, t, S: vf_tunnel(g, f, t, S, speed=5.0, complexity=10) * 1.3,
+ hf_distance(0.55, 0.02), PAL_GREEK, f, t, S, sat=0.7)
+
+ canvas_b = _render_vf(r, "sm",
+ lambda g, f, t, S: vf_ripple(g, f, t, S,
+ sources=[(0.3,0.3), (0.7,0.7), (0.5,0.2)], freq=0.5, damping=0.012) * 1.4,
+ hf_angle(0.1), PAL_STARS, f, t, S, sat=0.8)
+
+ return blend_canvas(canvas_a, canvas_b, "exclusion", 0.8)
+```
+
+### Complex Scene (Three Grids + Conditional + Custom Rendering)
+
+```python
+def fx_rings_explosion(r, f, t, S):
+ """Three-grid scene with particles and conditional kaleidoscope."""
+ # Layer 1: rings
+ canvas_a = _render_vf(r, "sm",
+ lambda g, f, t, S: vf_rings(g, f, t, S, n_base=10, spacing_base=2) * 1.4,
+ lambda g, f, t, S: (g.angle / (2*np.pi) + t * 0.15) % 1.0,
+ PAL_STARS, f, t, S, sat=0.9)
+
+ # Layer 2: vortex on different grid
+ canvas_b = _render_vf(r, "md",
+ lambda g, f, t, S: vf_vortex(g, f, t, S, twist=6.0) * 1.2,
+ hf_time_cycle(0.15), PAL_BLOCKS, f, t, S, sat=0.8)
+
+ result = blend_canvas(canvas_b, canvas_a, "screen", 0.7)
+
+ # Layer 3: particles (custom rendering, not _render_vf)
+ g = r.get_grid("sm")
+ if "px" not in S:
+ S["px"], S["py"], S["vx"], S["vy"], S["life"], S["pch"] = (
+ [], [], [], [], [], [])
+ if f.get("beat", 0) > 0.5:
+ chars = list("\u2605\u2736\u2733\u2738\u2726\u2728*+")
+ for _ in range(int(80 + f.get("rms", 0.3) * 120)):
+ ang = random.uniform(0, 2 * math.pi)
+ sp = random.uniform(1, 10) * (0.5 + f.get("sub_r", 0.3) * 2)
+ S["px"].append(float(g.cols // 2))
+ S["py"].append(float(g.rows // 2))
+ S["vx"].append(math.cos(ang) * sp * 2.5)
+ S["vy"].append(math.sin(ang) * sp)
+ S["life"].append(1.0)
+ S["pch"].append(random.choice(chars))
+
+ # Update + draw particles
+ ch_p = np.full((g.rows, g.cols), " ", dtype="U1")
+ co_p = np.zeros((g.rows, g.cols, 3), dtype=np.uint8)
+ i = 0
+ while i < len(S["px"]):
+ S["px"][i] += S["vx"][i]; S["py"][i] += S["vy"][i]
+ S["vy"][i] += 0.03; S["life"][i] -= 0.02
+ if S["life"][i] <= 0:
+ for k in ("px","py","vx","vy","life","pch"): S[k].pop(i)
+ else:
+ pr, pc = int(S["py"][i]), int(S["px"][i])
+ if 0 <= pr < g.rows and 0 <= pc < g.cols:
+ ch_p[pr, pc] = S["pch"][i]
+ co_p[pr, pc] = hsv2rgb_scalar(
+ 0.08 + (1-S["life"][i])*0.15, 0.95, S["life"][i])
+ i += 1
+
+ canvas_p = g.render(ch_p, co_p)
+ result = blend_canvas(result, canvas_p, "add", 0.8)
+
+ # Conditional kaleidoscope on strong beats
+ if f.get("bdecay", 0) > 0.4:
+ result = sh_kaleidoscope(result.copy(), folds=6)
+
+ return result
+```
+
+### Scene with Custom Character Rendering (Matrix Rain)
+
+When you need per-cell control beyond what `_render_vf()` provides:
+
+```python
+def fx_matrix_layered(r, f, t, S):
+ """Matrix rain blended with tunnel — two grids, screen blend."""
+ # Layer 1: Matrix rain (custom per-column rendering)
+ g = r.get_grid("md")
+ rows, cols = g.rows, g.cols
+ pal = PAL_KATA
+
+ if "ry" not in S or len(S["ry"]) != cols:
+ S["ry"] = np.random.uniform(-rows, rows, cols).astype(np.float32)
+ S["rsp"] = np.random.uniform(0.3, 2.0, cols).astype(np.float32)
+ S["rln"] = np.random.randint(8, 35, cols)
+ S["rch"] = np.random.randint(1, len(pal), (rows, cols))
+
+ speed = 0.6 + f.get("bass", 0.3) * 3
+ if f.get("beat", 0) > 0.5: speed *= 2.5
+ S["ry"] += S["rsp"] * speed
+
+ ch = np.full((rows, cols), " ", dtype="U1")
+ co = np.zeros((rows, cols, 3), dtype=np.uint8)
+ heads = S["ry"].astype(int)
+ for c in range(cols):
+ head = heads[c]
+ for i in range(S["rln"][c]):
+ row = head - i
+ if 0 <= row < rows:
+ fade = 1.0 - i / S["rln"][c]
+ ch[row, c] = pal[S["rch"][row, c] % len(pal)]
+ if i == 0:
+ v = int(min(255, fade * 300))
+ co[row, c] = (int(v*0.9), v, int(v*0.9))
+ else:
+ v = int(fade * 240)
+ co[row, c] = (int(v*0.1), v, int(v*0.4))
+ canvas_a = g.render(ch, co)
+
+ # Layer 2: Tunnel on sm grid for depth texture
+ canvas_b = _render_vf(r, "sm",
+ lambda g, f, t, S: vf_tunnel(g, f, t, S, speed=5.0, complexity=10),
+ hf_distance(0.3, 0.02), PAL_BLOCKS, f, t, S, sat=0.6)
+
+ return blend_canvas(canvas_a, canvas_b, "screen", 0.5)
+```
+
+---
+
+## Scene Table
+
+The scene table defines the timeline: which scene plays when, with what configuration.
+
+### Structure
+
+```python
+SCENES = [
+ {
+ "start": 0.0, # start time in seconds
+ "end": 3.96, # end time in seconds
+ "name": "starfield", # identifier (used for clip filenames)
+ "grid": "sm", # default grid (for render_clip setup)
+ "fx": fx_starfield, # scene function reference (must be module-level)
+ "gamma": 0.75, # tonemap gamma override (default 0.75)
+ "shaders": [ # shader chain (applied after tonemap + feedback)
+ ("bloom", {"thr": 120}),
+ ("vignette", {"s": 0.2}),
+ ("grain", {"amt": 8}),
+ ],
+ "feedback": None, # feedback buffer config (None = disabled)
+ # "feedback": {"decay": 0.8, "blend": "screen", "opacity": 0.3,
+ # "transform": "zoom", "transform_amt": 0.02, "hue_shift": 0.02},
+ },
+ {
+ "start": 3.96,
+ "end": 6.58,
+ "name": "matrix_layered",
+ "grid": "md",
+ "fx": fx_matrix_layered,
+ "shaders": [
+ ("crt", {"strength": 0.05}),
+ ("scanlines", {"intensity": 0.12}),
+ ("color_grade", {"tint": (0.7, 1.2, 0.7)}),
+ ("bloom", {"thr": 100}),
+ ],
+ "feedback": {"decay": 0.5, "blend": "add", "opacity": 0.2},
+ },
+ # ... more scenes ...
+]
+```
+
+### Beat-Synced Scene Cutting
+
+Derive cut points from audio analysis:
+
+```python
+# Get beat timestamps
+beats = [fi / FPS for fi in range(N_FRAMES) if features["beat"][fi] > 0.5]
+
+# Group beats into phrase boundaries (every 4-8 beats)
+cuts = [0.0]
+for i in range(0, len(beats), 4): # cut every 4 beats
+ cuts.append(beats[i])
+cuts.append(DURATION)
+
+# Or use the music's structure: silence gaps, energy changes
+energy = features["rms"]
+# Find timestamps where energy drops significantly -> natural break points
+```
+
+### `render_clip()` — The Render Loop
+
+This function renders one scene to a clip file:
+
+```python
+def render_clip(seg, features, clip_path):
+ r = Renderer()
+ r.set_grid(seg["grid"])
+ S = r.S
+ random.seed(hash(seg["id"]) + 42) # deterministic per scene
+
+ # Build shader chain from config
+ chain = ShaderChain()
+ for shader_name, kwargs in seg.get("shaders", []):
+ chain.add(shader_name, **kwargs)
+
+ # Setup feedback buffer
+ fb = None
+ fb_cfg = seg.get("feedback", None)
+ if fb_cfg:
+ fb = FeedbackBuffer()
+
+ fx_fn = seg["fx"]
+
+ # Open ffmpeg pipe
+ cmd = ["ffmpeg", "-y", "-f", "rawvideo", "-pix_fmt", "rgb24",
+ "-s", f"{VW}x{VH}", "-r", str(FPS), "-i", "pipe:0",
+ "-c:v", "libx264", "-preset", "fast", "-crf", "20",
+ "-pix_fmt", "yuv420p", clip_path]
+ stderr_fh = open(clip_path.replace(".mp4", ".log"), "w")
+ pipe = subprocess.Popen(cmd, stdin=subprocess.PIPE,
+ stdout=subprocess.DEVNULL, stderr=stderr_fh)
+
+ for fi in range(seg["frame_start"], seg["frame_end"]):
+ t = fi / FPS
+ feat = {k: float(features[k][fi]) for k in features}
+
+ # 1. Scene renders canvas
+ canvas = fx_fn(r, feat, t, S)
+
+ # 2. Tonemap normalizes brightness
+ canvas = tonemap(canvas, gamma=seg.get("gamma", 0.75))
+
+ # 3. Feedback adds temporal recursion
+ if fb and fb_cfg:
+ canvas = fb.apply(canvas, **{k: fb_cfg[k] for k in fb_cfg})
+
+ # 4. Shader chain adds post-processing
+ canvas = chain.apply(canvas, f=feat, t=t)
+
+ pipe.stdin.write(canvas.tobytes())
+
+ pipe.stdin.close(); pipe.wait(); stderr_fh.close()
+```
+
+### Building Segments from Scene Table
+
+```python
+segments = []
+for i, scene in enumerate(SCENES):
+ segments.append({
+ "id": f"s{i:02d}_{scene['name']}",
+ "name": scene["name"],
+ "grid": scene["grid"],
+ "fx": scene["fx"],
+ "shaders": scene.get("shaders", []),
+ "feedback": scene.get("feedback", None),
+ "gamma": scene.get("gamma", 0.75),
+ "frame_start": int(scene["start"] * FPS),
+ "frame_end": int(scene["end"] * FPS),
+ })
+```
+
+### Parallel Rendering
+
+Scenes are independent units dispatched to a process pool:
+
+```python
+from concurrent.futures import ProcessPoolExecutor, as_completed
+
+with ProcessPoolExecutor(max_workers=N_WORKERS) as pool:
+ futures = {
+ pool.submit(render_clip, seg, features, clip_path): seg["id"]
+ for seg, clip_path in zip(segments, clip_paths)
+ }
+ for fut in as_completed(futures):
+ try:
+ fut.result()
+ except Exception as e:
+ log(f"ERROR {futures[fut]}: {e}")
+```
+
+**Pickling constraint**: `ProcessPoolExecutor` serializes arguments via pickle. Module-level functions can be pickled; lambdas and closures cannot. All `fx_*` scene functions MUST be defined at module level, not as closures or class methods.
+
+### Test-Frame Mode
+
+Render a single frame at a specific timestamp to verify visuals without a full render:
+
+```python
+if args.test_frame >= 0:
+ fi = min(int(args.test_frame * FPS), N_FRAMES - 1)
+ t = fi / FPS
+ feat = {k: float(features[k][fi]) for k in features}
+ scene = next(sc for sc in reversed(SCENES) if t >= sc["start"])
+ r = Renderer()
+ r.set_grid(scene["grid"])
+ canvas = scene["fx"](r, feat, t, r.S)
+ canvas = tonemap(canvas, gamma=scene.get("gamma", 0.75))
+ chain = ShaderChain()
+ for sn, kw in scene.get("shaders", []):
+ chain.add(sn, **kw)
+ canvas = chain.apply(canvas, f=feat, t=t)
+ Image.fromarray(canvas).save(f"test_{args.test_frame:.1f}s.png")
+ print(f"Mean brightness: {canvas.astype(float).mean():.1f}")
+```
+
+CLI: `python reel.py --test-frame 10.0`
+
+---
+
+## Scene Design Checklist
+
+For each scene:
+
+1. **Choose 2-3 grid sizes** — different scales create interference
+2. **Choose different value fields** per layer — don't use the same effect on every grid
+3. **Choose different hue fields** per layer — or at minimum different hue offsets
+4. **Choose different palettes** per layer — mixing PAL_RUNE with PAL_BLOCKS looks different from PAL_RUNE with PAL_DENSE
+5. **Choose a blend mode** that matches the energy — screen for bright, difference for psychedelic, exclusion for subtle
+6. **Add conditional effects** on beat — kaleidoscope, mirror, glitch
+7. **Configure feedback** for trailing/recursive looks — or None for clean cuts
+8. **Set gamma** if using destructive shaders (solarize, posterize)
+9. **Test with --test-frame** at the scene's midpoint before full render
+
+---
+
+## Scene Examples
+
+Copy-paste-ready scene functions at increasing complexity. Each is a complete, working v2 scene function that returns a pixel canvas. See the Scene Protocol section above for the scene protocol and `composition.md` for blend modes and tonemap.
+
+---
+
+### Minimal — Single Grid, Single Effect
+
+### Breathing Plasma
+
+One grid, one value field, one hue field. The simplest possible scene.
+
+```python
+def fx_breathing_plasma(r, f, t, S):
+ """Plasma field with time-cycling hue. Audio modulates brightness."""
+ canvas = _render_vf(r, "md",
+ lambda g, f, t, S: vf_plasma(g, f, t, S) * 1.3,
+ hf_time_cycle(0.08), PAL_DENSE, f, t, S, sat=0.8)
+ return canvas
+```
+
+### Reaction-Diffusion Coral
+
+Single grid, simulation-based field. Evolves organically over time.
+
+```python
+def fx_coral(r, f, t, S):
+ """Gray-Scott reaction-diffusion — coral branching pattern.
+ Slow-evolving, organic. Best for ambient/chill sections."""
+ canvas = _render_vf(r, "sm",
+ lambda g, f, t, S: vf_reaction_diffusion(g, f, t, S,
+ feed=0.037, kill=0.060, steps_per_frame=6, init_mode="center"),
+ hf_distance(0.55, 0.015), PAL_DOTS, f, t, S, sat=0.7)
+ return canvas
+```
+
+### SDF Geometry
+
+Geometric shapes from SDFs. Clean, precise, graphic.
+
+```python
+def fx_sdf_rings(r, f, t, S):
+ """Concentric SDF rings with smooth pulsing."""
+ def val_fn(g, f, t, S):
+ d1 = sdf_ring(g, radius=0.15 + f.get("bass", 0.3) * 0.05, thickness=0.015)
+ d2 = sdf_ring(g, radius=0.25 + f.get("mid", 0.3) * 0.05, thickness=0.012)
+ d3 = sdf_ring(g, radius=0.35 + f.get("hi", 0.3) * 0.04, thickness=0.010)
+ combined = sdf_smooth_union(sdf_smooth_union(d1, d2, 0.05), d3, 0.05)
+ return sdf_glow(combined, falloff=0.08) * (0.5 + f.get("rms", 0.3) * 0.8)
+ canvas = _render_vf(r, "md", val_fn, hf_angle(0.0), PAL_STARS, f, t, S, sat=0.85)
+ return canvas
+```
+
+---
+
+### Standard — Two Grids + Blend
+
+### Tunnel Through Noise
+
+Two grids at different densities, screen blended. The fine noise texture shows through the coarser tunnel characters.
+
+```python
+def fx_tunnel_noise(r, f, t, S):
+ """Tunnel depth on md grid + fBM noise on sm grid, screen blended."""
+ canvas_a = _render_vf(r, "md",
+ lambda g, f, t, S: vf_tunnel(g, f, t, S, speed=4.0, complexity=8) * 1.2,
+ hf_distance(0.5, 0.02), PAL_BLOCKS, f, t, S, sat=0.7)
+
+ canvas_b = _render_vf(r, "sm",
+ lambda g, f, t, S: vf_fbm(g, f, t, S, octaves=4, freq=0.05, speed=0.15) * 1.3,
+ hf_time_cycle(0.06), PAL_RUNE, f, t, S, sat=0.6)
+
+ return blend_canvas(canvas_a, canvas_b, "screen", 0.7)
+```
+
+### Voronoi Cells + Spiral Overlay
+
+Voronoi cell edges with a spiral arm pattern overlaid.
+
+```python
+def fx_voronoi_spiral(r, f, t, S):
+ """Voronoi edge detection on md + logarithmic spiral on lg."""
+ canvas_a = _render_vf(r, "md",
+ lambda g, f, t, S: vf_voronoi(g, f, t, S,
+ n_cells=15, mode="edge", edge_width=2.0, speed=0.4),
+ hf_angle(0.2), PAL_CIRCUIT, f, t, S, sat=0.75)
+
+ canvas_b = _render_vf(r, "lg",
+ lambda g, f, t, S: vf_spiral(g, f, t, S, n_arms=4, tightness=3.0) * 1.2,
+ hf_distance(0.1, 0.03), PAL_BLOCKS, f, t, S, sat=0.9)
+
+ return blend_canvas(canvas_a, canvas_b, "exclusion", 0.6)
+```
+
+### Domain-Warped fBM
+
+Two layers of the same fBM, one domain-warped, difference-blended for psychedelic organic texture.
+
+```python
+def fx_organic_warp(r, f, t, S):
+ """Clean fBM vs domain-warped fBM, difference blended."""
+ canvas_a = _render_vf(r, "sm",
+ lambda g, f, t, S: vf_fbm(g, f, t, S, octaves=5, freq=0.04, speed=0.1),
+ hf_plasma(0.2), PAL_DENSE, f, t, S, sat=0.6)
+
+ canvas_b = _render_vf(r, "md",
+ lambda g, f, t, S: vf_domain_warp(g, f, t, S,
+ warp_strength=20.0, freq=0.05, speed=0.15),
+ hf_time_cycle(0.05), PAL_BRAILLE, f, t, S, sat=0.7)
+
+ return blend_canvas(canvas_a, canvas_b, "difference", 0.7)
+```
+
+---
+
+### Complex — Three Grids + Conditional + Feedback
+
+### Psychedelic Cathedral
+
+Three-grid composition with beat-triggered kaleidoscope and feedback zoom tunnel. The most visually complex pattern.
+
+```python
+def fx_cathedral(r, f, t, S):
+ """Three-layer cathedral: interference + rings + noise, kaleidoscope on beat,
+ feedback zoom tunnel."""
+ # Layer 1: interference pattern on sm grid
+ canvas_a = _render_vf(r, "sm",
+ lambda g, f, t, S: vf_interference(g, f, t, S, n_waves=7) * 1.3,
+ hf_angle(0.0), PAL_MATH, f, t, S, sat=0.8)
+
+ # Layer 2: pulsing rings on md grid
+ canvas_b = _render_vf(r, "md",
+ lambda g, f, t, S: vf_rings(g, f, t, S, n_base=10, spacing_base=3) * 1.4,
+ hf_distance(0.3, 0.02), PAL_STARS, f, t, S, sat=0.9)
+
+ # Layer 3: temporal noise on lg grid (slow morph)
+ canvas_c = _render_vf(r, "lg",
+ lambda g, f, t, S: vf_temporal_noise(g, f, t, S,
+ freq=0.04, t_freq=0.2, octaves=3),
+ hf_time_cycle(0.12), PAL_BLOCKS, f, t, S, sat=0.7)
+
+ # Blend: A screen B, then difference with C
+ result = blend_canvas(canvas_a, canvas_b, "screen", 0.8)
+ result = blend_canvas(result, canvas_c, "difference", 0.5)
+
+ # Beat-triggered kaleidoscope
+ if f.get("bdecay", 0) > 0.3:
+ folds = 6 if f.get("sub_r", 0.3) > 0.4 else 8
+ result = sh_kaleidoscope(result.copy(), folds=folds)
+
+ return result
+
+# Scene table entry with feedback:
+# {"start": 30.0, "end": 50.0, "name": "cathedral", "fx": fx_cathedral,
+# "gamma": 0.65, "shaders": [("bloom", {"thr": 110}), ("chromatic", {"amt": 4}),
+# ("vignette", {"s": 0.2}), ("grain", {"amt": 8})],
+# "feedback": {"decay": 0.75, "blend": "screen", "opacity": 0.35,
+# "transform": "zoom", "transform_amt": 0.012, "hue_shift": 0.015}}
+```
+
+### Masked Reaction-Diffusion with Attractor Overlay
+
+Reaction-diffusion visible only through an animated iris mask, with a strange attractor density field underneath.
+
+```python
+def fx_masked_life(r, f, t, S):
+ """Attractor base + reaction-diffusion visible through iris mask + particles."""
+ g_sm = r.get_grid("sm")
+ g_md = r.get_grid("md")
+
+ # Layer 1: strange attractor density field (background)
+ canvas_bg = _render_vf(r, "sm",
+ lambda g, f, t, S: vf_strange_attractor(g, f, t, S,
+ attractor="clifford", n_points=30000),
+ hf_time_cycle(0.04), PAL_DOTS, f, t, S, sat=0.5)
+
+ # Layer 2: reaction-diffusion (foreground, will be masked)
+ canvas_rd = _render_vf(r, "md",
+ lambda g, f, t, S: vf_reaction_diffusion(g, f, t, S,
+ feed=0.046, kill=0.063, steps_per_frame=4, init_mode="ring"),
+ hf_angle(0.15), PAL_HALFFILL, f, t, S, sat=0.85)
+
+ # Animated iris mask — opens over first 5 seconds of scene
+ scene_start = S.get("_scene_start", t)
+ if "_scene_start" not in S:
+ S["_scene_start"] = t
+ mask = mask_iris(g_md, t, scene_start, scene_start + 5.0,
+ max_radius=0.6)
+ canvas_rd = apply_mask_canvas(canvas_rd, mask, bg_canvas=canvas_bg)
+
+ # Layer 3: flow-field particles following the R-D gradient
+ rd_field = vf_reaction_diffusion(g_sm, f, t, S,
+ feed=0.046, kill=0.063, steps_per_frame=0) # read without stepping
+ ch_p, co_p = update_flow_particles(S, g_sm, f, rd_field,
+ n=300, speed=0.8, char_set=list("·•◦∘°"))
+ canvas_p = g_sm.render(ch_p, co_p)
+
+ result = blend_canvas(canvas_rd, canvas_p, "add", 0.7)
+ return result
+```
+
+### Morphing Field Sequence with Eased Keyframes
+
+Demonstrates temporal coherence: smooth morphing between effects with keyframed parameters.
+
+```python
+def fx_morphing_journey(r, f, t, S):
+ """Morphs through 4 value fields over 20 seconds with eased transitions.
+ Parameters (twist, arm count) also keyframed."""
+ # Keyframed twist parameter
+ twist = keyframe(t, [(0, 1.0), (5, 5.0), (10, 2.0), (15, 8.0), (20, 1.0)],
+ ease_fn=ease_in_out_cubic, loop=True)
+
+ # Sequence of value fields with 2s crossfade
+ fields = [
+ lambda g, f, t, S: vf_plasma(g, f, t, S),
+ lambda g, f, t, S: vf_vortex(g, f, t, S, twist=twist),
+ lambda g, f, t, S: vf_fbm(g, f, t, S, octaves=5, freq=0.04),
+ lambda g, f, t, S: vf_domain_warp(g, f, t, S, warp_strength=15),
+ ]
+ durations = [5.0, 5.0, 5.0, 5.0]
+
+ val_fn = lambda g, f, t, S: vf_sequence(g, f, t, S, fields, durations,
+ crossfade=2.0)
+
+ # Render with slowly rotating hue
+ canvas = _render_vf(r, "md", val_fn, hf_time_cycle(0.06),
+ PAL_DENSE, f, t, S, sat=0.8)
+
+ # Second layer: tiled version of same sequence at smaller grid
+ tiled_fn = lambda g, f, t, S: vf_sequence(
+ make_tgrid(g, *uv_tile(g, 3, 3, mirror=True)),
+ f, t, S, fields, durations, crossfade=2.0)
+ canvas_b = _render_vf(r, "sm", tiled_fn, hf_angle(0.1),
+ PAL_RUNE, f, t, S, sat=0.6)
+
+ return blend_canvas(canvas, canvas_b, "screen", 0.5)
+```
+
+---
+
+### Specialized — Unique State Patterns
+
+### Game of Life with Ghost Trails
+
+Cellular automaton with analog fade trails. Beat injects random cells.
+
+```python
+def fx_life(r, f, t, S):
+ """Conway's Game of Life with fading ghost trails.
+ Beat events inject random live cells for disruption."""
+ canvas = _render_vf(r, "sm",
+ lambda g, f, t, S: vf_game_of_life(g, f, t, S,
+ rule="life", steps_per_frame=1, fade=0.92, density=0.25),
+ hf_fixed(0.33), PAL_BLOCKS, f, t, S, sat=0.8)
+
+ # Overlay: coral automaton on lg grid for chunky texture
+ canvas_b = _render_vf(r, "lg",
+ lambda g, f, t, S: vf_game_of_life(g, f, t, S,
+ rule="coral", steps_per_frame=1, fade=0.85, density=0.15, seed=99),
+ hf_time_cycle(0.1), PAL_HATCH, f, t, S, sat=0.6)
+
+ return blend_canvas(canvas, canvas_b, "screen", 0.5)
+```
+
+### Boids Flock Over Voronoi
+
+Emergent swarm movement over a cellular background.
+
+```python
+def fx_boid_swarm(r, f, t, S):
+ """Flocking boids over animated voronoi cells."""
+ # Background: voronoi cells
+ canvas_bg = _render_vf(r, "md",
+ lambda g, f, t, S: vf_voronoi(g, f, t, S,
+ n_cells=20, mode="distance", speed=0.2),
+ hf_distance(0.4, 0.02), PAL_CIRCUIT, f, t, S, sat=0.5)
+
+ # Foreground: boids
+ g = r.get_grid("md")
+ ch_b, co_b = update_boids(S, g, f, n_boids=150, perception=6.0,
+ max_speed=1.5, char_set=list("▸▹►▻→⟶"))
+ canvas_boids = g.render(ch_b, co_b)
+
+ # Trails for the boids
+ # (boid positions are stored in S["boid_x"], S["boid_y"])
+ S["px"] = list(S.get("boid_x", []))
+ S["py"] = list(S.get("boid_y", []))
+ ch_t, co_t = draw_particle_trails(S, g, max_trail=6, fade=0.6)
+ canvas_trails = g.render(ch_t, co_t)
+
+ result = blend_canvas(canvas_bg, canvas_trails, "add", 0.3)
+ result = blend_canvas(result, canvas_boids, "add", 0.9)
+ return result
+```
+
+### Fire Rising Through SDF Text Stencil
+
+Fire effect visible only through text letterforms.
+
+```python
+def fx_fire_text(r, f, t, S):
+ """Fire columns visible through text stencil. Text acts as window."""
+ g = r.get_grid("lg")
+
+ # Full-screen fire (will be masked)
+ canvas_fire = _render_vf(r, "sm",
+ lambda g, f, t, S: np.clip(
+ vf_fbm(g, f, t, S, octaves=4, freq=0.08, speed=0.8) *
+ (1.0 - g.rr / g.rows) * # fade toward top
+ (0.6 + f.get("bass", 0.3) * 0.8), 0, 1),
+ hf_fixed(0.05), PAL_BLOCKS, f, t, S, sat=0.9) # fire hue
+
+ # Background: dark domain warp
+ canvas_bg = _render_vf(r, "md",
+ lambda g, f, t, S: vf_domain_warp(g, f, t, S,
+ warp_strength=8, freq=0.03, speed=0.05) * 0.3,
+ hf_fixed(0.6), PAL_DENSE, f, t, S, sat=0.4)
+
+ # Text stencil mask
+ mask = mask_text(g, "FIRE", row_frac=0.45)
+ # Expand vertically for multi-row coverage
+ for offset in range(-2, 3):
+ shifted = mask_text(g, "FIRE", row_frac=0.45 + offset / g.rows)
+ mask = mask_union(mask, shifted)
+
+ canvas_masked = apply_mask_canvas(canvas_fire, mask, bg_canvas=canvas_bg)
+ return canvas_masked
+```
+
+### Portrait Mode: Vertical Rain + Quote
+
+Optimized for 9:16. Uses vertical space for long rain trails and stacked text.
+
+```python
+def fx_portrait_rain_quote(r, f, t, S):
+ """Portrait-optimized: matrix rain (long vertical trails) with stacked quote.
+ Designed for 1080x1920 (9:16)."""
+ g = r.get_grid("md") # ~112x100 in portrait
+
+ # Matrix rain — long trails benefit from portrait's extra rows
+ ch, co, S = eff_matrix_rain(g, f, t, S,
+ hue=0.33, bri=0.6, pal=PAL_KATA, speed_base=0.4, speed_beat=2.5)
+ canvas_rain = g.render(ch, co)
+
+ # Tunnel depth underneath for texture
+ canvas_tunnel = _render_vf(r, "sm",
+ lambda g, f, t, S: vf_tunnel(g, f, t, S, speed=3.0, complexity=6) * 0.8,
+ hf_fixed(0.33), PAL_BLOCKS, f, t, S, sat=0.5)
+
+ result = blend_canvas(canvas_tunnel, canvas_rain, "screen", 0.8)
+
+ # Quote text — portrait layout: short lines, many of them
+ g_text = r.get_grid("lg") # ~90x80 in portrait
+ quote_lines = layout_text_portrait(
+ "The code is the art and the art is the code",
+ max_chars_per_line=20)
+ # Center vertically
+ block_start = (g_text.rows - len(quote_lines)) // 2
+ ch_t = np.full((g_text.rows, g_text.cols), " ", dtype="U1")
+ co_t = np.zeros((g_text.rows, g_text.cols, 3), dtype=np.uint8)
+ total_chars = sum(len(l) for l in quote_lines)
+ progress = min(1.0, (t - S.get("_scene_start", t)) / 3.0)
+ if "_scene_start" not in S: S["_scene_start"] = t
+ render_typewriter(ch_t, co_t, quote_lines, block_start, g_text.cols,
+ progress, total_chars, (200, 255, 220), t)
+ canvas_text = g_text.render(ch_t, co_t)
+
+ result = blend_canvas(result, canvas_text, "add", 0.9)
+ return result
+```
+
+---
+
+### Scene Table Template
+
+Wire scenes into a complete video:
+
+```python
+SCENES = [
+ {"start": 0.0, "end": 5.0, "name": "coral",
+ "fx": fx_coral, "grid": "sm", "gamma": 0.70,
+ "shaders": [("bloom", {"thr": 110}), ("vignette", {"s": 0.2})],
+ "feedback": {"decay": 0.8, "blend": "screen", "opacity": 0.3,
+ "transform": "zoom", "transform_amt": 0.01}},
+
+ {"start": 5.0, "end": 15.0, "name": "tunnel_noise",
+ "fx": fx_tunnel_noise, "grid": "md", "gamma": 0.75,
+ "shaders": [("chromatic", {"amt": 3}), ("bloom", {"thr": 120}),
+ ("scanlines", {"intensity": 0.06}), ("grain", {"amt": 8})],
+ "feedback": None},
+
+ {"start": 15.0, "end": 35.0, "name": "cathedral",
+ "fx": fx_cathedral, "grid": "sm", "gamma": 0.65,
+ "shaders": [("bloom", {"thr": 100}), ("chromatic", {"amt": 5}),
+ ("color_wobble", {"amt": 0.2}), ("vignette", {"s": 0.18})],
+ "feedback": {"decay": 0.75, "blend": "screen", "opacity": 0.35,
+ "transform": "zoom", "transform_amt": 0.012, "hue_shift": 0.015}},
+
+ {"start": 35.0, "end": 50.0, "name": "morphing",
+ "fx": fx_morphing_journey, "grid": "md", "gamma": 0.70,
+ "shaders": [("bloom", {"thr": 110}), ("grain", {"amt": 6})],
+ "feedback": {"decay": 0.7, "blend": "screen", "opacity": 0.25,
+ "transform": "rotate_cw", "transform_amt": 0.003}},
+]
+```
diff --git a/src/skills/bundled/ascii-video/references/shaders.md b/src/skills/bundled/ascii-video/references/shaders.md
new file mode 100644
index 0000000..a4cf7a2
--- /dev/null
+++ b/src/skills/bundled/ascii-video/references/shaders.md
@@ -0,0 +1,1385 @@
+# Shader Pipeline & Composable Effects
+
+Post-processing effects applied to the pixel canvas (`numpy uint8 array, shape (H,W,3)`) after character rendering and before encoding. Also covers **pixel-level blend modes**, **feedback buffers**, and the **ShaderChain** compositor.
+
+> **See also:** composition.md (blend modes, tonemap) · effects.md · scenes.md · architecture.md · optimization.md · troubleshooting.md
+>
+> **Blend modes:** For the 20 pixel blend modes and `blend_canvas()`, see `composition.md`. All blending uses `blend_canvas(base, top, mode, opacity)`.
+
+## Design Philosophy
+
+The shader pipeline turns raw ASCII renders into cinematic output. The system is designed for **composability** — every shader, blend mode, and feedback transform is an independent building block. Combining them creates infinite visual variety from a small set of primitives.
+
+Choose shaders that reinforce the mood:
+- **Retro terminal**: CRT + scanlines + grain + green/amber tint
+- **Clean modern**: light bloom + subtle vignette only
+- **Glitch art**: heavy chromatic aberration + glitch bands + color wobble + pixel sort
+- **Cinematic**: bloom + vignette + grain + color grade
+- **Dreamy**: heavy bloom + soft focus + color wobble + low contrast
+- **Harsh/industrial**: high contrast + grain + scanlines + no bloom
+- **Psychedelic**: color wobble + chromatic + kaleidoscope mirror + high saturation + feedback with hue shift
+- **Data corruption**: pixel sort + data bend + block glitch + posterize
+- **Recursive/infinite**: feedback buffer with zoom + screen blend + hue shift
+
+---
+
+## Pixel-Level Blend Modes
+
+All operate on float32 [0,1] canvases for precision. Use `blend_canvas(base, top, mode, opacity)` which handles uint8 <-> float conversion.
+
+### Available Modes
+
+```python
+BLEND_MODES = {
+ "normal": lambda a, b: b,
+ "add": lambda a, b: np.clip(a + b, 0, 1),
+ "subtract": lambda a, b: np.clip(a - b, 0, 1),
+ "multiply": lambda a, b: a * b,
+ "screen": lambda a, b: 1 - (1-a)*(1-b),
+ "overlay": # 2*a*b if a<0.5, else 1-2*(1-a)*(1-b)
+ "softlight": lambda a, b: (1-2*b)*a*a + 2*b*a,
+ "hardlight": # like overlay but keyed on b
+ "difference": lambda a, b: abs(a - b),
+ "exclusion": lambda a, b: a + b - 2*a*b,
+ "colordodge": lambda a, b: a / (1-b),
+ "colorburn": lambda a, b: 1 - (1-a)/b,
+ "linearlight": lambda a, b: a + 2*b - 1,
+ "vividlight": # burn if b<0.5, dodge if b>=0.5
+ "pin_light": # min(a,2b) if b<0.5, max(a,2b-1) if b>=0.5
+ "hard_mix": lambda a, b: 1 if a+b>=1 else 0,
+ "lighten": lambda a, b: max(a, b),
+ "darken": lambda a, b: min(a, b),
+ "grain_extract": lambda a, b: a - b + 0.5,
+ "grain_merge": lambda a, b: a + b - 0.5,
+}
+```
+
+### Usage
+
+```python
+def blend_canvas(base, top, mode="normal", opacity=1.0):
+ """Blend two uint8 canvases (H,W,3) using a named blend mode + opacity."""
+ af = base.astype(np.float32) / 255.0
+ bf = top.astype(np.float32) / 255.0
+ result = BLEND_MODES[mode](af, bf)
+ if opacity < 1.0:
+ result = af * (1-opacity) + result * opacity
+ return np.clip(result * 255, 0, 255).astype(np.uint8)
+
+# Multi-layer compositing
+result = blend_canvas(base, layer_a, "screen", 0.7)
+result = blend_canvas(result, layer_b, "difference", 0.5)
+result = blend_canvas(result, layer_c, "multiply", 0.3)
+```
+
+### Creative Combinations
+
+- **Feedback + difference** = psychedelic color evolution (each frame XORs with the previous)
+- **Screen + screen** = additive glow stacking
+- **Multiply** on two different effects = only shows where both have brightness (intersection)
+- **Exclusion** between two layers = creates complementary patterns where they differ
+- **Color dodge/burn** = extreme contrast enhancement at overlap zones
+- **Hard mix** = reduces everything to pure black/white/color at intersections
+
+---
+
+## Feedback Buffer
+
+Recursive temporal effect: frame N-1 feeds back into frame N with decay and optional spatial transform. Creates trails, echoes, smearing, zoom tunnels, rotation feedback, rainbow trails.
+
+```python
+class FeedbackBuffer:
+ def __init__(self):
+ self.buf = None # previous frame (float32, 0-1)
+
+ def apply(self, canvas, decay=0.85, blend="screen", opacity=0.5,
+ transform=None, transform_amt=0.02, hue_shift=0.0):
+ """Mix current frame with decayed/transformed previous frame.
+
+ Args:
+ canvas: current frame (uint8 H,W,3)
+ decay: how fast old frame fades (0=instant, 1=permanent)
+ blend: blend mode for mixing feedback
+ opacity: strength of feedback mix
+ transform: None, "zoom", "shrink", "rotate_cw", "rotate_ccw",
+ "shift_up", "shift_down", "mirror_h"
+ transform_amt: strength of spatial transform per frame
+ hue_shift: rotate hue of feedback buffer each frame (0-1)
+ """
+```
+
+### Feedback Presets
+
+```python
+# Infinite zoom tunnel
+fb_cfg = {"decay": 0.8, "blend": "screen", "opacity": 0.4,
+ "transform": "zoom", "transform_amt": 0.015}
+
+# Rainbow trails (psychedelic)
+fb_cfg = {"decay": 0.7, "blend": "screen", "opacity": 0.3,
+ "transform": "zoom", "transform_amt": 0.01, "hue_shift": 0.02}
+
+# Ghostly echo (horror)
+fb_cfg = {"decay": 0.9, "blend": "add", "opacity": 0.15,
+ "transform": "shift_up", "transform_amt": 0.01}
+
+# Kaleidoscopic recursion
+fb_cfg = {"decay": 0.75, "blend": "screen", "opacity": 0.35,
+ "transform": "rotate_cw", "transform_amt": 0.005, "hue_shift": 0.01}
+
+# Color evolution (abstract)
+fb_cfg = {"decay": 0.8, "blend": "difference", "opacity": 0.4, "hue_shift": 0.03}
+
+# Multiplied depth
+fb_cfg = {"decay": 0.65, "blend": "multiply", "opacity": 0.3, "transform": "mirror_h"}
+
+# Rising heat haze
+fb_cfg = {"decay": 0.5, "blend": "add", "opacity": 0.2,
+ "transform": "shift_up", "transform_amt": 0.02}
+```
+
+---
+
+## ShaderChain
+
+Composable shader pipeline. Build chains of named shaders with parameters. Order matters — shaders are applied sequentially to the canvas.
+
+```python
+class ShaderChain:
+ """Composable shader pipeline.
+
+ Usage:
+ chain = ShaderChain()
+ chain.add("bloom", thr=120)
+ chain.add("chromatic", amt=5)
+ chain.add("kaleidoscope", folds=6)
+ chain.add("vignette", s=0.2)
+ chain.add("grain", amt=12)
+ canvas = chain.apply(canvas, f=features, t=time)
+ """
+ def __init__(self):
+ self.steps = []
+
+ def add(self, shader_name, **kwargs):
+ self.steps.append((shader_name, kwargs))
+ return self # chainable
+
+ def apply(self, canvas, f=None, t=0):
+ if f is None: f = {}
+ for name, kwargs in self.steps:
+ canvas = _apply_shader_step(canvas, name, kwargs, f, t)
+ return canvas
+```
+
+### `_apply_shader_step()` — Full Dispatch Function
+
+Routes shader names to implementations. Some shaders have **audio-reactive scaling** — the dispatch function reads `f["bdecay"]` and `f["rms"]` to modulate parameters on the beat.
+
+```python
+def _apply_shader_step(canvas, name, kwargs, f, t):
+ """Dispatch a single shader by name with kwargs.
+
+ Args:
+ canvas: uint8 (H,W,3) pixel array
+ name: shader key string (e.g. "bloom", "chromatic")
+ kwargs: dict of shader parameters
+ f: audio features dict (keys: bdecay, rms, sub, etc.)
+ t: current time in seconds (float)
+ Returns:
+ canvas: uint8 (H,W,3) — processed
+ """
+ bd = f.get("bdecay", 0) # beat decay (0-1, high on beat)
+ rms = f.get("rms", 0.3) # audio energy (0-1)
+
+ # --- Geometry ---
+ if name == "crt":
+ return sh_crt(canvas, kwargs.get("strength", 0.05))
+ elif name == "pixelate":
+ return sh_pixelate(canvas, kwargs.get("block", 4))
+ elif name == "wave_distort":
+ return sh_wave_distort(canvas, t,
+ kwargs.get("freq", 0.02), kwargs.get("amp", 8), kwargs.get("axis", "x"))
+ elif name == "kaleidoscope":
+ return sh_kaleidoscope(canvas.copy(), kwargs.get("folds", 6))
+ elif name == "mirror_h":
+ return sh_mirror_h(canvas.copy())
+ elif name == "mirror_v":
+ return sh_mirror_v(canvas.copy())
+ elif name == "mirror_quad":
+ return sh_mirror_quad(canvas.copy())
+ elif name == "mirror_diag":
+ return sh_mirror_diag(canvas.copy())
+
+ # --- Channel ---
+ elif name == "chromatic":
+ base = kwargs.get("amt", 3)
+ return sh_chromatic(canvas, max(1, int(base * (0.4 + bd * 0.8))))
+ elif name == "channel_shift":
+ return sh_channel_shift(canvas,
+ kwargs.get("r", (0,0)), kwargs.get("g", (0,0)), kwargs.get("b", (0,0)))
+ elif name == "channel_swap":
+ return sh_channel_swap(canvas, kwargs.get("order", (2,1,0)))
+ elif name == "rgb_split_radial":
+ return sh_rgb_split_radial(canvas, kwargs.get("strength", 5))
+
+ # --- Color ---
+ elif name == "invert":
+ return sh_invert(canvas)
+ elif name == "posterize":
+ return sh_posterize(canvas, kwargs.get("levels", 4))
+ elif name == "threshold":
+ return sh_threshold(canvas, kwargs.get("thr", 128))
+ elif name == "solarize":
+ return sh_solarize(canvas, kwargs.get("threshold", 128))
+ elif name == "hue_rotate":
+ return sh_hue_rotate(canvas, kwargs.get("amount", 0.1))
+ elif name == "saturation":
+ return sh_saturation(canvas, kwargs.get("factor", 1.5))
+ elif name == "color_grade":
+ return sh_color_grade(canvas, kwargs.get("tint", (1,1,1)))
+ elif name == "color_wobble":
+ return sh_color_wobble(canvas, t, kwargs.get("amt", 0.3) * (0.5 + rms * 0.8))
+ elif name == "color_ramp":
+ return sh_color_ramp(canvas, kwargs.get("ramp", [(0,0,0),(255,255,255)]))
+
+ # --- Glow / Blur ---
+ elif name == "bloom":
+ return sh_bloom(canvas, kwargs.get("thr", 130))
+ elif name == "edge_glow":
+ return sh_edge_glow(canvas, kwargs.get("hue", 0.5))
+ elif name == "soft_focus":
+ return sh_soft_focus(canvas, kwargs.get("strength", 0.3))
+ elif name == "radial_blur":
+ return sh_radial_blur(canvas, kwargs.get("strength", 0.03))
+
+ # --- Noise ---
+ elif name == "grain":
+ return sh_grain(canvas, int(kwargs.get("amt", 10) * (0.5 + rms * 0.8)))
+ elif name == "static":
+ return sh_static_noise(canvas, kwargs.get("density", 0.05), kwargs.get("color", True))
+
+ # --- Lines / Patterns ---
+ elif name == "scanlines":
+ return sh_scanlines(canvas, kwargs.get("intensity", 0.08), kwargs.get("spacing", 3))
+ elif name == "halftone":
+ return sh_halftone(canvas, kwargs.get("dot_size", 6))
+
+ # --- Tone ---
+ elif name == "vignette":
+ return sh_vignette(canvas, kwargs.get("s", 0.22))
+ elif name == "contrast":
+ return sh_contrast(canvas, kwargs.get("factor", 1.3))
+ elif name == "gamma":
+ return sh_gamma(canvas, kwargs.get("gamma", 1.5))
+ elif name == "levels":
+ return sh_levels(canvas,
+ kwargs.get("black", 0), kwargs.get("white", 255), kwargs.get("midtone", 1.0))
+ elif name == "brightness":
+ return sh_brightness(canvas, kwargs.get("factor", 1.5))
+
+ # --- Glitch / Data ---
+ elif name == "glitch_bands":
+ return sh_glitch_bands(canvas, f)
+ elif name == "block_glitch":
+ return sh_block_glitch(canvas, kwargs.get("n_blocks", 8), kwargs.get("max_size", 40))
+ elif name == "pixel_sort":
+ return sh_pixel_sort(canvas, kwargs.get("threshold", 100), kwargs.get("direction", "h"))
+ elif name == "data_bend":
+ return sh_data_bend(canvas, kwargs.get("offset", 1000), kwargs.get("chunk", 500))
+
+ else:
+ return canvas # unknown shader — passthrough
+```
+
+### Audio-Reactive Shaders
+
+Three shaders scale their parameters based on audio features:
+
+| Shader | Reactive To | Effect |
+|--------|------------|--------|
+| `chromatic` | `bdecay` | `amt * (0.4 + bdecay * 0.8)` — aberration kicks on beats |
+| `color_wobble` | `rms` | `amt * (0.5 + rms * 0.8)` — wobble intensity follows energy |
+| `grain` | `rms` | `amt * (0.5 + rms * 0.8)` — grain rougher in loud sections |
+| `glitch_bands` | `bdecay`, `sub` | Number of bands and displacement scale with beat energy |
+
+To make any shader beat-reactive, scale its parameter in the dispatch: `base_val * (low + bd * range)`.
+
+---
+
+## Full Shader Catalog
+
+### Geometry Shaders
+
+| Shader | Key Params | Description |
+|--------|-----------|-------------|
+| `crt` | `strength=0.05` | CRT barrel distortion (cached remap) |
+| `pixelate` | `block=4` | Reduce effective resolution |
+| `wave_distort` | `freq, amp, axis` | Sinusoidal row/column displacement |
+| `kaleidoscope` | `folds=6` | Radial symmetry via polar remapping |
+| `mirror_h` | — | Horizontal mirror |
+| `mirror_v` | — | Vertical mirror |
+| `mirror_quad` | — | 4-fold mirror |
+| `mirror_diag` | — | Diagonal mirror |
+
+### Channel Manipulation
+
+| Shader | Key Params | Description |
+|--------|-----------|-------------|
+| `chromatic` | `amt=3` | R/B channel horizontal shift (beat-reactive) |
+| `channel_shift` | `r=(sx,sy), g, b` | Independent per-channel x,y shifting |
+| `channel_swap` | `order=(2,1,0)` | Reorder RGB channels (BGR, GRB, etc.) |
+| `rgb_split_radial` | `strength=5` | Chromatic aberration radiating from center |
+
+### Color Manipulation
+
+| Shader | Key Params | Description |
+|--------|-----------|-------------|
+| `invert` | — | Negate all colors |
+| `posterize` | `levels=4` | Reduce color depth to N levels |
+| `threshold` | `thr=128` | Binary black/white |
+| `solarize` | `threshold=128` | Invert pixels above threshold |
+| `hue_rotate` | `amount=0.1` | Rotate all hues by amount (0-1) |
+| `saturation` | `factor=1.5` | Scale saturation (>1=more, <1=less) |
+| `color_grade` | `tint=(r,g,b)` | Per-channel multiplier |
+| `color_wobble` | `amt=0.3` | Time-varying per-channel sine modulation |
+| `color_ramp` | `ramp=[(R,G,B),...]` | Map luminance to custom color gradient |
+
+### Glow / Blur
+
+| Shader | Key Params | Description |
+|--------|-----------|-------------|
+| `bloom` | `thr=130` | Bright area glow (4x downsample + box blur) |
+| `edge_glow` | `hue=0.5` | Detect edges, add colored overlay |
+| `soft_focus` | `strength=0.3` | Blend with blurred version |
+| `radial_blur` | `strength=0.03` | Zoom blur from center outward |
+
+### Noise / Grain
+
+| Shader | Key Params | Description |
+|--------|-----------|-------------|
+| `grain` | `amt=10` | 2x-downsampled film grain (beat-reactive) |
+| `static` | `density=0.05, color=True` | Random pixel noise (TV static) |
+
+### Lines / Patterns
+
+| Shader | Key Params | Description |
+|--------|-----------|-------------|
+| `scanlines` | `intensity=0.08, spacing=3` | Darken every Nth row |
+| `halftone` | `dot_size=6` | Halftone dot pattern overlay |
+
+### Tone
+
+| Shader | Key Params | Description |
+|--------|-----------|-------------|
+| `vignette` | `s=0.22` | Edge darkening (cached distance field) |
+| `contrast` | `factor=1.3` | Adjust contrast around midpoint 128 |
+| `gamma` | `gamma=1.5` | Gamma correction (>1=brighter mids) |
+| `levels` | `black, white, midtone` | Levels adjustment (Photoshop-style) |
+| `brightness` | `factor=1.5` | Global brightness multiplier |
+
+### Glitch / Data
+
+| Shader | Key Params | Description |
+|--------|-----------|-------------|
+| `glitch_bands` | (uses `f`) | Beat-reactive horizontal row displacement |
+| `block_glitch` | `n_blocks=8, max_size=40` | Random rectangular block displacement |
+| `pixel_sort` | `threshold=100, direction="h"` | Sort pixels by brightness in rows/columns |
+| `data_bend` | `offset, chunk` | Raw byte displacement (datamoshing) |
+
+---
+
+## Shader Implementations
+
+Every shader function takes a canvas (`uint8 H,W,3`) and returns a canvas of the same shape. The naming convention is `sh_`. Geometry shaders that build coordinate remap tables should **cache** them since the table only depends on resolution + parameters, not on frame content.
+
+### Helpers
+
+Shaders that manipulate hue/saturation need vectorized HSV conversion:
+
+```python
+def rgb2hsv(r, g, b):
+ """Vectorized RGB (0-255 uint8) -> HSV (float32 0-1)."""
+ rf = r.astype(np.float32) / 255.0
+ gf = g.astype(np.float32) / 255.0
+ bf = b.astype(np.float32) / 255.0
+ cmax = np.maximum(np.maximum(rf, gf), bf)
+ cmin = np.minimum(np.minimum(rf, gf), bf)
+ delta = cmax - cmin + 1e-10
+ h = np.zeros_like(rf)
+ m = cmax == rf; h[m] = ((gf[m] - bf[m]) / delta[m]) % 6
+ m = cmax == gf; h[m] = (bf[m] - rf[m]) / delta[m] + 2
+ m = cmax == bf; h[m] = (rf[m] - gf[m]) / delta[m] + 4
+ h = h / 6.0 % 1.0
+ s = np.where(cmax > 0, delta / (cmax + 1e-10), 0)
+ return h, s, cmax
+
+def hsv2rgb(h, s, v):
+ """Vectorized HSV->RGB. h,s,v are numpy float32 arrays."""
+ h = h % 1.0
+ c = v * s; x = c * (1 - np.abs((h * 6) % 2 - 1)); m = v - c
+ r = np.zeros_like(h); g = np.zeros_like(h); b = np.zeros_like(h)
+ mask = h < 1/6; r[mask]=c[mask]; g[mask]=x[mask]
+ mask = (h>=1/6)&(h<2/6); r[mask]=x[mask]; g[mask]=c[mask]
+ mask = (h>=2/6)&(h<3/6); g[mask]=c[mask]; b[mask]=x[mask]
+ mask = (h>=3/6)&(h<4/6); g[mask]=x[mask]; b[mask]=c[mask]
+ mask = (h>=4/6)&(h<5/6); r[mask]=x[mask]; b[mask]=c[mask]
+ mask = h >= 5/6; r[mask]=c[mask]; b[mask]=x[mask]
+ R = np.clip((r+m)*255, 0, 255).astype(np.uint8)
+ G = np.clip((g+m)*255, 0, 255).astype(np.uint8)
+ B = np.clip((b+m)*255, 0, 255).astype(np.uint8)
+ return R, G, B
+
+def mkc(R, G, B, rows, cols):
+ """Stack R,G,B uint8 arrays into (rows,cols,3) canvas."""
+ o = np.zeros((rows, cols, 3), dtype=np.uint8)
+ o[:,:,0] = R; o[:,:,1] = G; o[:,:,2] = B
+ return o
+```
+
+---
+
+### Geometry Shaders
+
+#### CRT Barrel Distortion
+Cache the coordinate remap — it never changes per frame:
+```python
+_crt_cache = {}
+def sh_crt(c, strength=0.05):
+ k = (c.shape[0], c.shape[1], round(strength, 3))
+ if k not in _crt_cache:
+ h, w = c.shape[:2]; cy, cx = h/2, w/2
+ Y = np.arange(h, dtype=np.float32)[:, None]
+ X = np.arange(w, dtype=np.float32)[None, :]
+ ny = (Y - cy) / cy; nx = (X - cx) / cx
+ r2 = nx**2 + ny**2
+ factor = 1 + strength * r2
+ sx = np.clip((nx * factor * cx + cx), 0, w-1).astype(np.int32)
+ sy = np.clip((ny * factor * cy + cy), 0, h-1).astype(np.int32)
+ _crt_cache[k] = (sy, sx)
+ sy, sx = _crt_cache[k]
+ return c[sy, sx]
+```
+
+#### Pixelate
+```python
+def sh_pixelate(c, block=4):
+ """Reduce effective resolution."""
+ sm = c[::block, ::block]
+ return np.repeat(np.repeat(sm, block, axis=0), block, axis=1)[:c.shape[0], :c.shape[1]]
+```
+
+#### Wave Distort
+```python
+def sh_wave_distort(c, t, freq=0.02, amp=8, axis="x"):
+ """Sinusoidal row/column displacement. Uses time t for animation."""
+ h, w = c.shape[:2]
+ out = c.copy()
+ if axis == "x":
+ for y in range(h):
+ shift = int(amp * math.sin(y * freq + t * 3))
+ out[y] = np.roll(c[y], shift, axis=0)
+ else:
+ for x in range(w):
+ shift = int(amp * math.sin(x * freq + t * 3))
+ out[:, x] = np.roll(c[:, x], shift, axis=0)
+ return out
+```
+
+#### Displacement Map
+```python
+def sh_displacement_map(c, dx_map, dy_map, strength=10):
+ """Displace pixels using float32 displacement maps (same HxW as c).
+ dx_map/dy_map: positive = shift right/down."""
+ h, w = c.shape[:2]
+ Y = np.arange(h)[:, None]; X = np.arange(w)[None, :]
+ ny = np.clip((Y + (dy_map * strength).astype(int)), 0, h-1)
+ nx = np.clip((X + (dx_map * strength).astype(int)), 0, w-1)
+ return c[ny, nx]
+```
+
+#### Kaleidoscope
+```python
+def sh_kaleidoscope(c, folds=6):
+ """Radial symmetry by polar coordinate remapping."""
+ h, w = c.shape[:2]; cy, cx = h//2, w//2
+ Y = np.arange(h, dtype=np.float32)[:, None] - cy
+ X = np.arange(w, dtype=np.float32)[None, :] - cx
+ angle = np.arctan2(Y, X)
+ dist = np.sqrt(X**2 + Y**2)
+ wedge = 2 * np.pi / folds
+ folded_angle = np.abs((angle % wedge) - wedge/2)
+ ny = np.clip((cy + dist * np.sin(folded_angle)).astype(int), 0, h-1)
+ nx = np.clip((cx + dist * np.cos(folded_angle)).astype(int), 0, w-1)
+ return c[ny, nx]
+```
+
+#### Mirror Variants
+```python
+def sh_mirror_h(c):
+ """Horizontal mirror — left half reflected to right."""
+ w = c.shape[1]; c[:, w//2:] = c[:, :w//2][:, ::-1]; return c
+
+def sh_mirror_v(c):
+ """Vertical mirror — top half reflected to bottom."""
+ h = c.shape[0]; c[h//2:, :] = c[:h//2, :][::-1, :]; return c
+
+def sh_mirror_quad(c):
+ """4-fold mirror — top-left quadrant reflected to all four."""
+ h, w = c.shape[:2]; hh, hw = h//2, w//2
+ tl = c[:hh, :hw].copy()
+ c[:hh, hw:hw+tl.shape[1]] = tl[:, ::-1]
+ c[hh:hh+tl.shape[0], :hw] = tl[::-1, :]
+ c[hh:hh+tl.shape[0], hw:hw+tl.shape[1]] = tl[::-1, ::-1]
+ return c
+
+def sh_mirror_diag(c):
+ """Diagonal mirror — top-left triangle reflected."""
+ h, w = c.shape[:2]
+ for y in range(h):
+ x_cut = int(w * y / h)
+ if x_cut > 0 and x_cut < w:
+ c[y, x_cut:] = c[y, :x_cut+1][::-1][:w-x_cut]
+ return c
+```
+
+> **Note:** Mirror shaders mutate in-place. The dispatch function passes `canvas.copy()` to avoid corrupting the original.
+
+---
+
+### Channel Manipulation Shaders
+
+#### Chromatic Aberration
+```python
+def sh_chromatic(c, amt=3):
+ """R/B channel horizontal shift. Beat-reactive in dispatch (amt scaled by bdecay)."""
+ if amt < 1: return c
+ a = int(amt)
+ o = c.copy()
+ o[:, a:, 0] = c[:, :-a, 0] # red shifts right
+ o[:, :-a, 2] = c[:, a:, 2] # blue shifts left
+ return o
+```
+
+#### Channel Shift
+```python
+def sh_channel_shift(c, r_shift=(0,0), g_shift=(0,0), b_shift=(0,0)):
+ """Independent per-channel x,y shifting."""
+ o = c.copy()
+ for ch_i, (sx, sy) in enumerate([r_shift, g_shift, b_shift]):
+ if sx != 0: o[:,:,ch_i] = np.roll(c[:,:,ch_i], sx, axis=1)
+ if sy != 0: o[:,:,ch_i] = np.roll(o[:,:,ch_i], sy, axis=0)
+ return o
+```
+
+#### Channel Swap
+```python
+def sh_channel_swap(c, order=(2,1,0)):
+ """Reorder RGB channels. (2,1,0)=BGR, (1,0,2)=GRB, etc."""
+ return c[:, :, list(order)]
+```
+
+#### RGB Split Radial
+```python
+def sh_rgb_split_radial(c, strength=5):
+ """Chromatic aberration radiating from center — stronger at edges."""
+ h, w = c.shape[:2]; cy, cx = h//2, w//2
+ Y = np.arange(h, dtype=np.float32)[:, None]
+ X = np.arange(w, dtype=np.float32)[None, :]
+ dist = np.sqrt((Y-cy)**2 + (X-cx)**2)
+ max_dist = np.sqrt(cy**2 + cx**2)
+ factor = dist / max_dist * strength
+ dy = ((Y-cy) / (dist+1) * factor).astype(int)
+ dx = ((X-cx) / (dist+1) * factor).astype(int)
+ out = c.copy()
+ ry = np.clip(Y.astype(int)+dy, 0, h-1); rx = np.clip(X.astype(int)+dx, 0, w-1)
+ out[:,:,0] = c[ry, rx, 0] # red shifts outward
+ by = np.clip(Y.astype(int)-dy, 0, h-1); bx = np.clip(X.astype(int)-dx, 0, w-1)
+ out[:,:,2] = c[by, bx, 2] # blue shifts inward
+ return out
+```
+
+---
+
+### Color Manipulation Shaders
+
+#### Invert
+```python
+def sh_invert(c):
+ return 255 - c
+```
+
+#### Posterize
+```python
+def sh_posterize(c, levels=4):
+ """Reduce color depth to N levels per channel."""
+ step = 256.0 / levels
+ return (np.floor(c.astype(np.float32) / step) * step).astype(np.uint8)
+```
+
+#### Threshold
+```python
+def sh_threshold(c, thr=128):
+ """Binary black/white at threshold."""
+ gray = c.astype(np.float32).mean(axis=2)
+ out = np.zeros_like(c); out[gray > thr] = 255
+ return out
+```
+
+#### Solarize
+```python
+def sh_solarize(c, threshold=128):
+ """Invert pixels above threshold — classic darkroom effect."""
+ o = c.copy(); mask = c > threshold; o[mask] = 255 - c[mask]
+ return o
+```
+
+#### Hue Rotate
+```python
+def sh_hue_rotate(c, amount=0.1):
+ """Rotate all hues by amount (0-1)."""
+ h, s, v = rgb2hsv(c[:,:,0], c[:,:,1], c[:,:,2])
+ h = (h + amount) % 1.0
+ R, G, B = hsv2rgb(h, s, v)
+ return mkc(R, G, B, c.shape[0], c.shape[1])
+```
+
+#### Saturation
+```python
+def sh_saturation(c, factor=1.5):
+ """Adjust saturation. >1=more saturated, <1=desaturated."""
+ h, s, v = rgb2hsv(c[:,:,0], c[:,:,1], c[:,:,2])
+ s = np.clip(s * factor, 0, 1)
+ R, G, B = hsv2rgb(h, s, v)
+ return mkc(R, G, B, c.shape[0], c.shape[1])
+```
+
+#### Color Grade
+```python
+def sh_color_grade(c, tint):
+ """Per-channel multiplier. tint=(r_mul, g_mul, b_mul)."""
+ o = c.astype(np.float32)
+ o[:,:,0] *= tint[0]; o[:,:,1] *= tint[1]; o[:,:,2] *= tint[2]
+ return np.clip(o, 0, 255).astype(np.uint8)
+```
+
+#### Color Wobble
+```python
+def sh_color_wobble(c, t, amt=0.3):
+ """Time-varying per-channel sine modulation. Audio-reactive in dispatch (amt scaled by rms)."""
+ o = c.astype(np.float32)
+ o[:,:,0] *= 1.0 + amt * math.sin(t * 5.0)
+ o[:,:,1] *= 1.0 + amt * math.sin(t * 5.0 + 2.09)
+ o[:,:,2] *= 1.0 + amt * math.sin(t * 5.0 + 4.19)
+ return np.clip(o, 0, 255).astype(np.uint8)
+```
+
+#### Color Ramp
+```python
+def sh_color_ramp(c, ramp_colors):
+ """Map luminance to a custom color gradient.
+ ramp_colors = list of (R,G,B) tuples, evenly spaced from dark to bright."""
+ gray = c.astype(np.float32).mean(axis=2) / 255.0
+ n = len(ramp_colors)
+ idx = np.clip(gray * (n-1), 0, n-1.001)
+ lo = np.floor(idx).astype(int); hi = np.minimum(lo+1, n-1)
+ frac = idx - lo
+ ramp = np.array(ramp_colors, dtype=np.float32)
+ out = ramp[lo] * (1-frac[:,:,None]) + ramp[hi] * frac[:,:,None]
+ return np.clip(out, 0, 255).astype(np.uint8)
+```
+
+---
+
+### Glow / Blur Shaders
+
+#### Bloom
+```python
+def sh_bloom(c, thr=130):
+ """Bright-area glow: 4x downsample, threshold, 3-pass box blur, screen blend."""
+ sm = c[::4, ::4].astype(np.float32)
+ br = np.where(sm > thr, sm, 0)
+ for _ in range(3):
+ p = np.pad(br, ((1,1),(1,1),(0,0)), mode="edge")
+ br = (p[:-2,:-2]+p[:-2,1:-1]+p[:-2,2:]+p[1:-1,:-2]+p[1:-1,1:-1]+
+ p[1:-1,2:]+p[2:,:-2]+p[2:,1:-1]+p[2:,2:]) / 9.0
+ bl = np.repeat(np.repeat(br, 4, axis=0), 4, axis=1)[:c.shape[0], :c.shape[1]]
+ return np.clip(c.astype(np.float32) + bl * 0.5, 0, 255).astype(np.uint8)
+```
+
+#### Edge Glow
+```python
+def sh_edge_glow(c, hue=0.5):
+ """Detect edges via gradient, add colored overlay."""
+ gray = c.astype(np.float32).mean(axis=2)
+ gx = np.abs(gray[:, 2:] - gray[:, :-2])
+ gy = np.abs(gray[2:, :] - gray[:-2, :])
+ ex = np.zeros_like(gray); ey = np.zeros_like(gray)
+ ex[:, 1:-1] = gx; ey[1:-1, :] = gy
+ edge = np.clip((ex + ey) / 255 * 2, 0, 1)
+ R, G, B = hsv2rgb(np.full_like(edge, hue), np.full_like(edge, 0.8), edge * 0.5)
+ out = c.astype(np.int16).copy()
+ out[:,:,0] = np.clip(out[:,:,0] + R.astype(np.int16), 0, 255)
+ out[:,:,1] = np.clip(out[:,:,1] + G.astype(np.int16), 0, 255)
+ out[:,:,2] = np.clip(out[:,:,2] + B.astype(np.int16), 0, 255)
+ return out.astype(np.uint8)
+```
+
+#### Soft Focus
+```python
+def sh_soft_focus(c, strength=0.3):
+ """Blend original with 2x-downsampled box blur."""
+ sm = c[::2, ::2].astype(np.float32)
+ p = np.pad(sm, ((1,1),(1,1),(0,0)), mode="edge")
+ bl = (p[:-2,:-2]+p[:-2,1:-1]+p[:-2,2:]+p[1:-1,:-2]+p[1:-1,1:-1]+
+ p[1:-1,2:]+p[2:,:-2]+p[2:,1:-1]+p[2:,2:]) / 9.0
+ bl = np.repeat(np.repeat(bl, 2, axis=0), 2, axis=1)[:c.shape[0], :c.shape[1]]
+ return np.clip(c * (1-strength) + bl * strength, 0, 255).astype(np.uint8)
+```
+
+#### Radial Blur
+```python
+def sh_radial_blur(c, strength=0.03, center=None):
+ """Zoom blur from center — motion blur radiating outward."""
+ h, w = c.shape[:2]
+ cy, cx = center if center else (h//2, w//2)
+ Y = np.arange(h, dtype=np.float32)[:, None]
+ X = np.arange(w, dtype=np.float32)[None, :]
+ out = c.astype(np.float32)
+ for s in [strength, strength*2]:
+ dy = (Y - cy) * s; dx = (X - cx) * s
+ sy = np.clip((Y + dy).astype(int), 0, h-1)
+ sx = np.clip((X + dx).astype(int), 0, w-1)
+ out += c[sy, sx].astype(np.float32)
+ return np.clip(out / 3, 0, 255).astype(np.uint8)
+```
+
+---
+
+### Noise / Grain Shaders
+
+#### Film Grain
+```python
+def sh_grain(c, amt=10):
+ """2x-downsampled film grain. Audio-reactive in dispatch (amt scaled by rms)."""
+ noise = np.random.randint(-amt, amt+1, (c.shape[0]//2, c.shape[1]//2, 1), dtype=np.int16)
+ noise = np.repeat(np.repeat(noise, 2, axis=0), 2, axis=1)[:c.shape[0], :c.shape[1]]
+ return np.clip(c.astype(np.int16) + noise, 0, 255).astype(np.uint8)
+```
+
+#### Static Noise
+```python
+def sh_static_noise(c, density=0.05, color=True):
+ """Random pixel noise overlay (TV static)."""
+ mask = np.random.random((c.shape[0]//2, c.shape[1]//2)) < density
+ mask = np.repeat(np.repeat(mask, 2, axis=0), 2, axis=1)[:c.shape[0], :c.shape[1]]
+ out = c.copy()
+ if color:
+ noise = np.random.randint(0, 256, (c.shape[0], c.shape[1], 3), dtype=np.uint8)
+ else:
+ v = np.random.randint(0, 256, (c.shape[0], c.shape[1]), dtype=np.uint8)
+ noise = np.stack([v, v, v], axis=2)
+ out[mask] = noise[mask]
+ return out
+```
+
+---
+
+### Lines / Pattern Shaders
+
+#### Scanlines
+```python
+def sh_scanlines(c, intensity=0.08, spacing=3):
+ """Darken every Nth row."""
+ m = np.ones(c.shape[0], dtype=np.float32)
+ m[::spacing] = 1.0 - intensity
+ return np.clip(c * m[:, None, None], 0, 255).astype(np.uint8)
+```
+
+#### Halftone
+```python
+def sh_halftone(c, dot_size=6):
+ """Halftone dot pattern overlay — circular dots sized by local brightness."""
+ h, w = c.shape[:2]
+ gray = c.astype(np.float32).mean(axis=2) / 255.0
+ out = np.zeros_like(c)
+ for y in range(0, h, dot_size):
+ for x in range(0, w, dot_size):
+ block = gray[y:y+dot_size, x:x+dot_size]
+ if block.size == 0: continue
+ radius = block.mean() * dot_size * 0.5
+ cy_b, cx_b = dot_size//2, dot_size//2
+ for dy in range(min(dot_size, h-y)):
+ for dx in range(min(dot_size, w-x)):
+ if math.sqrt((dy-cy_b)**2 + (dx-cx_b)**2) < radius:
+ out[y+dy, x+dx] = c[y+dy, x+dx]
+ return out
+```
+
+> **Performance note:** Halftone is slow due to Python loops. Acceptable for small resolutions or single test frames. For production, consider a vectorized version using precomputed distance masks.
+
+---
+
+### Tone Shaders
+
+#### Vignette
+```python
+_vig_cache = {}
+def sh_vignette(c, s=0.22):
+ """Edge darkening using cached distance field."""
+ k = (c.shape[0], c.shape[1], round(s, 2))
+ if k not in _vig_cache:
+ h, w = c.shape[:2]
+ Y = np.linspace(-1, 1, h)[:, None]; X = np.linspace(-1, 1, w)[None, :]
+ _vig_cache[k] = np.clip(1.0 - np.sqrt(X**2 + Y**2) * s, 0.15, 1).astype(np.float32)
+ return np.clip(c * _vig_cache[k][:,:,None], 0, 255).astype(np.uint8)
+```
+
+#### Reverse Vignette
+
+Inverted vignette: darkens the **center** and leaves edges bright. Useful when text is centered over busy backgrounds — creates a natural dark zone for readability without a hard-edged box.
+
+Combine with `apply_text_backdrop()` (see composition.md) for per-frame glyph-aware darkening.
+
+```python
+_rvignette_cache = {}
+
+def sh_reverse_vignette(c, strength=0.5):
+ """Center darkening, edge brightening. Cached."""
+ k = ('rv', c.shape[0], c.shape[1], round(strength, 2))
+ if k not in _rvignette_cache:
+ h, w = c.shape[:2]
+ Y = np.linspace(-1, 1, h)[:, None]
+ X = np.linspace(-1, 1, w)[None, :]
+ d = np.sqrt(X**2 + Y**2)
+ # Invert: bright at edges, dark at center
+ mask = np.clip(1.0 - (1.0 - d * 0.7) * strength, 0.2, 1.0)
+ _rvignette_cache[k] = mask[:, :, np.newaxis].astype(np.float32)
+ return np.clip(c.astype(np.float32) * _rvignette_cache[k], 0, 255).astype(np.uint8)
+```
+
+| Param | Default | Effect |
+|-------|---------|--------|
+| `strength` | 0.5 | 0 = no effect, 1.0 = center nearly black |
+
+Add to ShaderChain dispatch:
+```python
+elif name == "reverse_vignette":
+ return sh_reverse_vignette(canvas, kwargs.get("strength", 0.5))
+```
+
+#### Contrast
+```python
+def sh_contrast(c, factor=1.3):
+ """Adjust contrast around midpoint 128."""
+ return np.clip((c.astype(np.float32) - 128) * factor + 128, 0, 255).astype(np.uint8)
+```
+
+#### Gamma
+```python
+def sh_gamma(c, gamma=1.5):
+ """Gamma correction. >1=brighter mids, <1=darker mids."""
+ return np.clip(((c.astype(np.float32)/255.0) ** (1.0/gamma)) * 255, 0, 255).astype(np.uint8)
+```
+
+#### Levels
+```python
+def sh_levels(c, black=0, white=255, midtone=1.0):
+ """Levels adjustment (Photoshop-style). Remap black/white points, apply midtone gamma."""
+ o = (c.astype(np.float32) - black) / max(1, white - black)
+ o = np.clip(o, 0, 1) ** (1.0 / midtone)
+ return (o * 255).astype(np.uint8)
+```
+
+#### Brightness
+```python
+def sh_brightness(c, factor=1.5):
+ """Global brightness multiplier. Prefer tonemap() for scene-level brightness control."""
+ return np.clip(c.astype(np.float32) * factor, 0, 255).astype(np.uint8)
+```
+
+---
+
+### Glitch / Data Shaders
+
+#### Glitch Bands
+```python
+def sh_glitch_bands(c, f):
+ """Beat-reactive horizontal row displacement. f = audio features dict.
+ Uses f["bdecay"] for intensity and f["sub"] for band height."""
+ n = int(3 + f.get("bdecay", 0) * 10)
+ out = c.copy()
+ for _ in range(n):
+ y = random.randint(0, c.shape[0]-1)
+ h = random.randint(1, max(2, int(4 + f.get("sub", 0.3) * 12)))
+ shift = int((random.random()-0.5) * f.get("bdecay", 0) * 60)
+ if shift != 0 and y+h < c.shape[0]:
+ out[y:y+h] = np.roll(out[y:y+h], shift, axis=1)
+ return out
+```
+
+#### Block Glitch
+```python
+def sh_block_glitch(c, n_blocks=8, max_size=40):
+ """Random rectangular block displacement — copy blocks to random positions."""
+ out = c.copy(); h, w = c.shape[:2]
+ for _ in range(n_blocks):
+ bw = random.randint(10, max_size); bh = random.randint(5, max_size//2)
+ sx = random.randint(0, w-bw-1); sy = random.randint(0, h-bh-1)
+ dx = random.randint(0, w-bw-1); dy = random.randint(0, h-bh-1)
+ out[dy:dy+bh, dx:dx+bw] = c[sy:sy+bh, sx:sx+bw]
+ return out
+```
+
+#### Pixel Sort
+```python
+def sh_pixel_sort(c, threshold=100, direction="h"):
+ """Sort pixels by brightness in contiguous bright regions."""
+ gray = c.astype(np.float32).mean(axis=2)
+ out = c.copy()
+ if direction == "h":
+ for y in range(0, c.shape[0], 3): # every 3rd row for speed
+ row_bright = gray[y]
+ mask = row_bright > threshold
+ regions = np.diff(np.concatenate([[0], mask.astype(int), [0]]))
+ starts = np.where(regions == 1)[0]
+ ends = np.where(regions == -1)[0]
+ for s, e in zip(starts, ends):
+ if e - s > 2:
+ indices = np.argsort(gray[y, s:e])
+ out[y, s:e] = c[y, s:e][indices]
+ else:
+ for x in range(0, c.shape[1], 3):
+ col_bright = gray[:, x]
+ mask = col_bright > threshold
+ regions = np.diff(np.concatenate([[0], mask.astype(int), [0]]))
+ starts = np.where(regions == 1)[0]
+ ends = np.where(regions == -1)[0]
+ for s, e in zip(starts, ends):
+ if e - s > 2:
+ indices = np.argsort(gray[s:e, x])
+ out[s:e, x] = c[s:e, x][indices]
+ return out
+```
+
+#### Data Bend
+```python
+def sh_data_bend(c, offset=1000, chunk=500):
+ """Treat raw pixel bytes as data, copy a chunk to another offset — datamosh artifacts."""
+ flat = c.flatten().copy()
+ n = len(flat)
+ src = offset % n; dst = (offset + chunk*3) % n
+ length = min(chunk, n-src, n-dst)
+ if length > 0:
+ flat[dst:dst+length] = flat[src:src+length]
+ return flat.reshape(c.shape)
+```
+
+---
+
+## Tint Presets
+
+```python
+TINT_WARM = (1.15, 1.0, 0.85) # golden warmth
+TINT_COOL = (0.85, 0.95, 1.15) # blue cool
+TINT_MATRIX = (0.7, 1.2, 0.7) # green terminal
+TINT_AMBER = (1.2, 0.9, 0.6) # amber monitor
+TINT_SEPIA = (1.2, 1.05, 0.8) # old film
+TINT_NEON_PINK = (1.3, 0.7, 1.1) # cyberpunk pink
+TINT_ICE = (0.8, 1.0, 1.3) # frozen
+TINT_BLOOD = (1.4, 0.7, 0.7) # horror red
+TINT_FOREST = (0.8, 1.15, 0.75) # natural green
+TINT_VOID = (0.85, 0.85, 1.1) # deep space
+TINT_SUNSET = (1.3, 0.85, 0.7) # orange dusk
+```
+
+---
+
+## Transitions
+
+> **Note:** These operate on character-level `(chars, colors)` arrays (v1 interface). In v2, transitions between scenes are typically handled by hard cuts at beat boundaries (see `scenes.md`), or by rendering both scenes to canvases and using `blend_canvas()` with a time-varying opacity. The character-level transitions below are still useful for within-scene effects.
+
+### Crossfade
+```python
+def tr_crossfade(ch_a, co_a, ch_b, co_b, blend):
+ co = (co_a.astype(np.float32) * (1-blend) + co_b.astype(np.float32) * blend).astype(np.uint8)
+ mask = np.random.random(ch_a.shape) < blend
+ ch = ch_a.copy(); ch[mask] = ch_b[mask]
+ return ch, co
+```
+
+### v2 Canvas-Level Crossfade
+```python
+def tr_canvas_crossfade(canvas_a, canvas_b, blend):
+ """Smooth pixel crossfade between two canvases."""
+ return np.clip(canvas_a * (1-blend) + canvas_b * blend, 0, 255).astype(np.uint8)
+```
+
+### Wipe (directional)
+```python
+def tr_wipe(ch_a, co_a, ch_b, co_b, blend, direction="left"):
+ """direction: left, right, up, down, radial, diagonal"""
+ rows, cols = ch_a.shape
+ if direction == "radial":
+ cx, cy = cols/2, rows/2
+ rr = np.arange(rows)[:, None]; cc = np.arange(cols)[None, :]
+ d = np.sqrt((cc-cx)**2 + (rr-cy)**2)
+ mask = d < blend * np.sqrt(cx**2 + cy**2)
+ ch = ch_a.copy(); co = co_a.copy()
+ ch[mask] = ch_b[mask]; co[mask] = co_b[mask]
+ return ch, co
+```
+
+### Glitch Cut
+```python
+def tr_glitch_cut(ch_a, co_a, ch_b, co_b, blend):
+ if blend < 0.5: ch, co = ch_a.copy(), co_a.copy()
+ else: ch, co = ch_b.copy(), co_b.copy()
+ if 0.3 < blend < 0.7:
+ intensity = 1.0 - abs(blend - 0.5) * 4
+ for _ in range(int(intensity * 20)):
+ y = random.randint(0, ch.shape[0]-1)
+ shift = int((random.random()-0.5) * 40 * intensity)
+ if shift: ch[y] = np.roll(ch[y], shift); co[y] = np.roll(co[y], shift, axis=0)
+ return ch, co
+```
+
+---
+
+## Output Formats
+
+### MP4 (default)
+```python
+cmd = ["ffmpeg", "-y", "-f", "rawvideo", "-pix_fmt", "rgb24",
+ "-s", f"{W}x{H}", "-r", str(fps), "-i", "pipe:0",
+ "-c:v", "libx264", "-preset", "fast", "-crf", str(crf),
+ "-pix_fmt", "yuv420p", output_path]
+```
+
+### GIF
+```python
+cmd = ["ffmpeg", "-y", "-f", "rawvideo", "-pix_fmt", "rgb24",
+ "-s", f"{W}x{H}", "-r", str(fps), "-i", "pipe:0",
+ "-vf", f"fps={fps},scale={W}:{H}:flags=lanczos,split[s0][s1];[s0]palettegen[p];[s1][p]paletteuse",
+ "-loop", "0", output_gif]
+```
+
+### PNG Sequence
+
+For frame-accurate editing, compositing in external tools (After Effects, Nuke), or lossless archival:
+
+```python
+import os
+
+def output_png_sequence(frames, output_dir, W, H, fps, prefix="frame"):
+ """Write frames as numbered PNGs. frames = iterable of uint8 (H,W,3) arrays."""
+ os.makedirs(output_dir, exist_ok=True)
+
+ # Method 1: Direct PIL write (no ffmpeg dependency)
+ from PIL import Image
+ for i, frame in enumerate(frames):
+ img = Image.fromarray(frame)
+ img.save(os.path.join(output_dir, f"{prefix}_{i:06d}.png"))
+
+ # Method 2: ffmpeg pipe (faster for large sequences)
+ cmd = ["ffmpeg", "-y", "-f", "rawvideo", "-pix_fmt", "rgb24",
+ "-s", f"{W}x{H}", "-r", str(fps), "-i", "pipe:0",
+ os.path.join(output_dir, f"{prefix}_%06d.png")]
+```
+
+Reassemble PNG sequence to video:
+```bash
+ffmpeg -framerate 24 -i frame_%06d.png -c:v libx264 -crf 18 -pix_fmt yuv420p output.mp4
+```
+
+### Alpha Channel / Transparent Background (RGBA)
+
+For compositing ASCII art over other video or images. Uses RGBA canvas (4 channels) instead of RGB (3 channels):
+
+```python
+def create_rgba_canvas(H, W):
+ """Transparent canvas — alpha channel starts at 0 (fully transparent)."""
+ return np.zeros((H, W, 4), dtype=np.uint8)
+
+def render_char_rgba(canvas, row, col, char_img, color_rgb, alpha=255):
+ """Render a character with alpha. char_img = PIL glyph mask (grayscale).
+ Alpha comes from the glyph mask — background stays transparent."""
+ r, g, b = color_rgb
+ y0, x0 = row * cell_h, col * cell_w
+ mask = np.array(char_img) # grayscale 0-255
+ canvas[y0:y0+cell_h, x0:x0+cell_w, 0] = np.maximum(canvas[y0:y0+cell_h, x0:x0+cell_w, 0], (mask * r / 255).astype(np.uint8))
+ canvas[y0:y0+cell_h, x0:x0+cell_w, 1] = np.maximum(canvas[y0:y0+cell_h, x0:x0+cell_w, 1], (mask * g / 255).astype(np.uint8))
+ canvas[y0:y0+cell_h, x0:x0+cell_w, 2] = np.maximum(canvas[y0:y0+cell_h, x0:x0+cell_w, 2], (mask * b / 255).astype(np.uint8))
+ canvas[y0:y0+cell_h, x0:x0+cell_w, 3] = np.maximum(canvas[y0:y0+cell_h, x0:x0+cell_w, 3], mask)
+
+def blend_onto_background(rgba_canvas, bg_rgb):
+ """Composite RGBA canvas over a solid or image background."""
+ alpha = rgba_canvas[:, :, 3:4].astype(np.float32) / 255.0
+ fg = rgba_canvas[:, :, :3].astype(np.float32)
+ bg = bg_rgb.astype(np.float32)
+ result = fg * alpha + bg * (1.0 - alpha)
+ return result.astype(np.uint8)
+```
+
+RGBA output via ffmpeg (ProRes 4444 for editing, WebM VP9 for web):
+```bash
+# ProRes 4444 — preserves alpha, widely supported in NLEs
+ffmpeg -y -f rawvideo -pix_fmt rgba -s {W}x{H} -r {fps} -i pipe:0 \
+ -c:v prores_ks -profile:v 4444 -pix_fmt yuva444p10le output.mov
+
+# WebM VP9 — alpha support for web/browser compositing
+ffmpeg -y -f rawvideo -pix_fmt rgba -s {W}x{H} -r {fps} -i pipe:0 \
+ -c:v libvpx-vp9 -pix_fmt yuva420p -crf 30 -b:v 0 output.webm
+
+# PNG sequence with alpha (lossless)
+ffmpeg -y -f rawvideo -pix_fmt rgba -s {W}x{H} -r {fps} -i pipe:0 \
+ frame_%06d.png
+```
+
+**Key constraint**: shaders that operate on `(H,W,3)` arrays need adaptation for RGBA. Either apply shaders to the RGB channels only and preserve alpha, or write RGBA-aware versions:
+
+```python
+def apply_shader_rgba(canvas_rgba, shader_fn, **kwargs):
+ """Apply an RGB shader to the color channels of an RGBA canvas."""
+ rgb = canvas_rgba[:, :, :3]
+ alpha = canvas_rgba[:, :, 3:4]
+ rgb_out = shader_fn(rgb, **kwargs)
+ return np.concatenate([rgb_out, alpha], axis=2)
+```
+
+---
+
+## Real-Time Terminal Rendering
+
+Live ASCII display in the terminal using ANSI escape codes. Useful for previewing scenes during development, live performances, and interactive parameter tuning.
+
+### ANSI Color Escape Codes
+
+```python
+def rgb_to_ansi(r, g, b):
+ """24-bit true color ANSI escape (supported by most modern terminals)."""
+ return f"\033[38;2;{r};{g};{b}m"
+
+ANSI_RESET = "\033[0m"
+ANSI_CLEAR = "\033[2J\033[H" # clear screen + cursor home
+ANSI_HIDE_CURSOR = "\033[?25l"
+ANSI_SHOW_CURSOR = "\033[?25h"
+```
+
+### Frame-to-ANSI Conversion
+
+```python
+def frame_to_ansi(chars, colors):
+ """Convert char+color arrays to a single ANSI string for terminal output.
+
+ Args:
+ chars: (rows, cols) array of single characters
+ colors: (rows, cols, 3) uint8 RGB array
+ Returns:
+ str: ANSI-encoded frame ready for sys.stdout.write()
+ """
+ rows, cols = chars.shape
+ lines = []
+ for r in range(rows):
+ parts = []
+ prev_color = None
+ for c in range(cols):
+ rgb = tuple(colors[r, c])
+ ch = chars[r, c]
+ if ch == " " or rgb == (0, 0, 0):
+ parts.append(" ")
+ else:
+ if rgb != prev_color:
+ parts.append(rgb_to_ansi(*rgb))
+ prev_color = rgb
+ parts.append(ch)
+ parts.append(ANSI_RESET)
+ lines.append("".join(parts))
+ return "\n".join(lines)
+```
+
+### Optimized: Delta Updates
+
+Only redraw characters that changed since the last frame. Eliminates redundant terminal writes for static regions:
+
+```python
+def frame_to_ansi_delta(chars, colors, prev_chars, prev_colors):
+ """Emit ANSI escapes only for cells that changed."""
+ rows, cols = chars.shape
+ parts = []
+ for r in range(rows):
+ for c in range(cols):
+ if (chars[r, c] != prev_chars[r, c] or
+ not np.array_equal(colors[r, c], prev_colors[r, c])):
+ parts.append(f"\033[{r+1};{c+1}H") # move cursor
+ rgb = tuple(colors[r, c])
+ parts.append(rgb_to_ansi(*rgb))
+ parts.append(chars[r, c])
+ return "".join(parts)
+```
+
+### Live Render Loop
+
+```python
+import sys
+import time
+
+def render_live(scene_fn, r, fps=24, duration=None):
+ """Render a scene function live in the terminal.
+
+ Args:
+ scene_fn: v2 scene function (r, f, t, S) -> canvas
+ OR v1-style function that populates a grid
+ r: Renderer instance
+ fps: target frame rate
+ duration: seconds to run (None = run until Ctrl+C)
+ """
+ frame_time = 1.0 / fps
+ S = {}
+ f = {} # synthesize features or connect to live audio
+
+ sys.stdout.write(ANSI_HIDE_CURSOR + ANSI_CLEAR)
+ sys.stdout.flush()
+
+ t0 = time.monotonic()
+ frame_count = 0
+ try:
+ while True:
+ t = time.monotonic() - t0
+ if duration and t > duration:
+ break
+
+ # Synthesize features from time (or connect to live audio via pyaudio)
+ f = synthesize_features(t)
+
+ # Render scene — for terminal, use a small grid
+ g = r.get_grid("sm")
+ # Option A: v2 scene → extract chars/colors from canvas (reverse render)
+ # Option B: call effect functions directly for chars/colors
+ canvas = scene_fn(r, f, t, S)
+
+ # For terminal display, render chars+colors directly
+ # (bypassing the pixel canvas — terminal uses character cells)
+ chars, colors = scene_to_terminal(scene_fn, r, f, t, S, g)
+
+ frame_str = ANSI_CLEAR + frame_to_ansi(chars, colors)
+ sys.stdout.write(frame_str)
+ sys.stdout.flush()
+
+ # Frame timing
+ elapsed = time.monotonic() - t0 - (frame_count * frame_time)
+ sleep_time = frame_time - elapsed
+ if sleep_time > 0:
+ time.sleep(sleep_time)
+ frame_count += 1
+ except KeyboardInterrupt:
+ pass
+ finally:
+ sys.stdout.write(ANSI_SHOW_CURSOR + ANSI_RESET + "\n")
+ sys.stdout.flush()
+
+def scene_to_terminal(scene_fn, r, f, t, S, g):
+ """Run effect functions and return (chars, colors) for terminal display.
+ For terminal mode, skip the pixel canvas and work with character arrays directly."""
+ # Effects that return (chars, colors) work directly
+ # For vf-based effects, render the value field + hue field to chars/colors:
+ val = vf_plasma(g, f, t, S)
+ hue = hf_time_cycle(0.08)(g, t)
+ mask = val > 0.03
+ chars = val2char(val, mask, PAL_DENSE)
+ R, G, B = hsv2rgb(hue, np.full_like(val, 0.8), val)
+ colors = mkc(R, G, B, g.rows, g.cols)
+ return chars, colors
+```
+
+### Curses-Based Rendering (More Robust)
+
+For full-featured terminal UIs with proper resize handling and input:
+
+```python
+import curses
+
+def render_curses(scene_fn, r, fps=24):
+ """Curses-based live renderer with resize handling and key input."""
+
+ def _main(stdscr):
+ curses.start_color()
+ curses.use_default_colors()
+ curses.curs_set(0) # hide cursor
+ stdscr.nodelay(True) # non-blocking input
+
+ # Initialize color pairs (curses supports 256 colors)
+ # Map RGB to nearest curses color pair
+ color_cache = {}
+ next_pair = [1]
+
+ def get_color_pair(r, g, b):
+ key = (r >> 4, g >> 4, b >> 4) # quantize to reduce pairs
+ if key not in color_cache:
+ if next_pair[0] < curses.COLOR_PAIRS - 1:
+ ci = 16 + (r // 51) * 36 + (g // 51) * 6 + (b // 51) # 6x6x6 cube
+ curses.init_pair(next_pair[0], ci, -1)
+ color_cache[key] = next_pair[0]
+ next_pair[0] += 1
+ else:
+ return 0
+ return curses.color_pair(color_cache[key])
+
+ S = {}
+ f = {}
+ frame_time = 1.0 / fps
+ t0 = time.monotonic()
+
+ while True:
+ t = time.monotonic() - t0
+ f = synthesize_features(t)
+
+ # Adapt grid to terminal size
+ max_y, max_x = stdscr.getmaxyx()
+ g = r.get_grid_for_size(max_x, max_y) # dynamic grid sizing
+
+ chars, colors = scene_to_terminal(scene_fn, r, f, t, S, g)
+ rows, cols = chars.shape
+
+ for row in range(min(rows, max_y - 1)):
+ for col in range(min(cols, max_x - 1)):
+ ch = chars[row, col]
+ rgb = tuple(colors[row, col])
+ try:
+ stdscr.addch(row, col, ch, get_color_pair(*rgb))
+ except curses.error:
+ pass # ignore writes outside terminal bounds
+
+ stdscr.refresh()
+
+ # Handle input
+ key = stdscr.getch()
+ if key == ord('q'):
+ break
+
+ time.sleep(max(0, frame_time - (time.monotonic() - t0 - t)))
+
+ curses.wrapper(_main)
+```
+
+### Terminal Rendering Constraints
+
+| Constraint | Value | Notes |
+|-----------|-------|-------|
+| Max practical grid | ~200x60 | Depends on terminal size |
+| Color support | 24-bit (modern), 256 (fallback), 16 (minimal) | Check `$COLORTERM` for truecolor |
+| Frame rate ceiling | ~30 fps | Terminal I/O is the bottleneck |
+| Delta updates | 2-5x faster | Only worth it when <30% of cells change per frame |
+| SSH latency | Kills performance | Local terminals only for real-time |
+
+**Detect color support:**
+```python
+import os
+def get_terminal_color_depth():
+ ct = os.environ.get("COLORTERM", "")
+ if ct in ("truecolor", "24bit"):
+ return 24
+ term = os.environ.get("TERM", "")
+ if "256color" in term:
+ return 8 # 256 colors
+ return 4 # 16 colors basic ANSI
+```
diff --git a/src/skills/bundled/ascii-video/references/troubleshooting.md b/src/skills/bundled/ascii-video/references/troubleshooting.md
new file mode 100644
index 0000000..6b38382
--- /dev/null
+++ b/src/skills/bundled/ascii-video/references/troubleshooting.md
@@ -0,0 +1,367 @@
+# Troubleshooting Reference
+
+> **See also:** composition.md · architecture.md · shaders.md · scenes.md · optimization.md
+
+## Quick Diagnostic
+
+| Symptom | Likely Cause | Fix |
+|---------|-------------|-----|
+| All black output | tonemap gamma too high or no effects rendering | Lower gamma to 0.5, check scene_fn returns non-zero canvas |
+| Washed out / too bright | Linear brightness multiplier instead of tonemap | Replace `canvas * N` with `tonemap(canvas, gamma=0.75)` |
+| ffmpeg hangs mid-render | stderr=subprocess.PIPE deadlock | Redirect stderr to file |
+| "read-only" array error | broadcast_to view without .copy() | Add `.copy()` after broadcast_to |
+| PicklingError | Lambda or closure in SCENES table | Define all fx_* at module level |
+| Random dark holes in output | Font missing Unicode glyphs | Validate palettes at init |
+| Audio-visual desync | Frame timing accumulation | Use integer frame counter, compute t fresh each frame |
+| Single-color flat output | Hue field shape mismatch | Ensure h,s,v arrays all (rows,cols) before hsv2rgb |
+| Text unreadable over busy bg | No contrast between text and background | Use `apply_text_backdrop()` (composition.md) + `reverse_vignette` shader (shaders.md) |
+| Text garbled/mirrored | Kaleidoscope or mirror shader applied to text scene | **Never apply kaleidoscope, mirror_h/v/quad/diag to scenes with readable text** — radial folding destroys legibility. Apply these only to background layers or text-free scenes |
+
+Common bugs, gotchas, and platform-specific issues encountered during ASCII video development.
+
+## NumPy Broadcasting
+
+### The `broadcast_to().copy()` Trap
+
+Hue field generators often return arrays that are broadcast views — they have shape `(1, cols)` or `(rows, 1)` that numpy broadcasts to `(rows, cols)`. These views are **read-only**. If any downstream code tries to modify them in-place (e.g., `h %= 1.0`), numpy raises:
+
+```
+ValueError: output array is read-only
+```
+
+**Fix**: Always `.copy()` after `broadcast_to()`:
+
+```python
+h = np.broadcast_to(h, (g.rows, g.cols)).copy()
+```
+
+This is especially important in `_render_vf()` where hue arrays flow through `hsv2rgb()`.
+
+### The `+=` vs `+` Trap
+
+Broadcasting also fails with in-place operators when operand shapes don't match exactly:
+
+```python
+# FAILS if result is (rows,1) and operand is (rows, cols)
+val += np.sin(g.cc * 0.02 + t * 0.3) * 0.5
+
+# WORKS — creates a new array
+val = val + np.sin(g.cc * 0.02 + t * 0.3) * 0.5
+```
+
+The `vf_plasma()` function had this bug. Use `+` instead of `+=` when mixing different-shaped arrays.
+
+### Shape Mismatch in `hsv2rgb()`
+
+`hsv2rgb(h, s, v)` requires all three arrays to have identical shapes. If `h` is `(1, cols)` and `s` is `(rows, cols)`, the function crashes or produces wrong output.
+
+**Fix**: Ensure all inputs are broadcast and copied to `(rows, cols)` before calling.
+
+---
+
+## Blend Mode Pitfalls
+
+### Overlay Crushes Dark Inputs
+
+`overlay(a, b) = 2*a*b` when `a < 0.5`. Two values of 0.12 produce `2 * 0.12 * 0.12 = 0.03`. The result is darker than either input.
+
+**Impact**: If both layers are dark (which ASCII art usually is), overlay produces near-black output.
+
+**Fix**: Use `screen` for dark source material. Screen always brightens: `1 - (1-a)*(1-b)`.
+
+### Colordodge Division by Zero
+
+`colordodge(a, b) = a / (1 - b)`. When `b = 1.0` (pure white pixels), this divides by zero.
+
+**Fix**: Add epsilon: `a / (1 - b + 1e-6)`. The implementation in `BLEND_MODES` should include this.
+
+### Colorburn Division by Zero
+
+`colorburn(a, b) = 1 - (1-a) / b`. When `b = 0` (pure black pixels), this divides by zero.
+
+**Fix**: Add epsilon: `1 - (1-a) / (b + 1e-6)`.
+
+### Multiply Always Darkens
+
+`multiply(a, b) = a * b`. Since both operands are [0,1], the result is always <= min(a,b). Never use multiply as a feedback blend mode — the frame goes black within a few frames.
+
+**Fix**: Use `screen` for feedback, or `add` with low opacity.
+
+---
+
+## Multiprocessing
+
+### Pickling Constraints
+
+`ProcessPoolExecutor` serializes function arguments via pickle. This constrains what you can pass to workers:
+
+| Can Pickle | Cannot Pickle |
+|-----------|---------------|
+| Module-level functions (`def fx_foo():`) | Lambdas (`lambda x: x + 1`) |
+| Dicts, lists, numpy arrays | Closures (functions defined inside functions) |
+| Class instances (with `__reduce__`) | Instance methods |
+| Strings, numbers | File handles, sockets |
+
+**Impact**: All scene functions referenced in the SCENES table must be defined at module level with `def`. If you use a lambda or closure, you get:
+
+```
+_pickle.PicklingError: Can't pickle at 0x...>
+```
+
+**Fix**: Define all scene functions at module top level. Lambdas used inside `_render_vf()` as val_fn/hue_fn are fine because they execute within the worker process — they're not pickled across process boundaries.
+
+### macOS spawn vs Linux fork
+
+On macOS, `multiprocessing` defaults to `spawn` (full serialization). On Linux, it defaults to `fork` (copy-on-write). This means:
+
+- **macOS**: Feature arrays are serialized per worker (~57KB for 30s video, but scales with duration). Each worker re-imports the entire module.
+- **Linux**: Feature arrays are shared via COW. Workers inherit the parent's memory.
+
+**Impact**: On macOS, module-level code (like `detect_hardware()`) runs in every worker process. If it has side effects (e.g., subprocess calls), those happen N+1 times.
+
+### Per-Worker State Isolation
+
+Each worker creates its own:
+- `Renderer` instance (with fresh grid cache)
+- `FeedbackBuffer` (feedback doesn't cross scene boundaries)
+- Random seed (`random.seed(hash(seg_id) + 42)`)
+
+This means:
+- Particle state doesn't carry between scenes (expected)
+- Feedback trails reset at scene cuts (expected)
+- `np.random` state is NOT seeded by `random.seed()` — they use separate RNGs
+
+**Fix for deterministic noise**: Use `np.random.RandomState(seed)` explicitly:
+
+```python
+rng = np.random.RandomState(hash(seg_id) + 42)
+noise = rng.random((rows, cols))
+```
+
+---
+
+## Brightness Issues
+
+### Dark Scenes After Tonemap
+
+If a scene is still dark after tonemap, check:
+
+1. **Gamma too high**: Lower gamma (0.5-0.6) for scenes with destructive post-processing
+2. **Shader destroying brightness**: Solarize, posterize, or contrast adjustments in the shader chain can undo tonemap's work. Move destructive shaders earlier in the chain, or increase gamma to compensate.
+3. **Feedback with multiply**: Multiply feedback darkens every frame. Switch to screen or add.
+4. **Overlay blend in scene**: If the scene function uses `blend_canvas(..., "overlay", ...)` with dark layers, switch to screen.
+
+### Diagnostic: Test-Frame Brightness
+
+```bash
+python reel.py --test-frame 10.0
+# Output: Mean brightness: 44.3, max: 255
+```
+
+If mean < 20, the scene needs attention. Common fixes:
+- Lower gamma in the SCENES entry
+- Change internal blend modes from overlay/multiply to screen/add
+- Increase value field multipliers (e.g., `vf_plasma(...) * 1.5`)
+- Check that the shader chain doesn't have an aggressive solarize or threshold
+
+### v1 Brightness Pattern (Deprecated)
+
+The old pattern used a linear multiplier:
+
+```python
+# OLD — don't use
+canvas = np.clip(canvas.astype(np.float32) * 2.0, 0, 255).astype(np.uint8)
+```
+
+This fails because:
+- Dark scenes (mean 8): `8 * 2.0 = 16` — still dark
+- Bright scenes (mean 130): `130 * 2.0 = 255` — clipped, lost detail
+
+Use `tonemap()` instead. See `composition.md` § Adaptive Tone Mapping.
+
+---
+
+## ffmpeg Issues
+
+### Pipe Deadlock
+
+The #1 production bug. If you use `stderr=subprocess.PIPE`:
+
+```python
+# DEADLOCK — stderr buffer fills at 64KB, blocks ffmpeg, blocks your writes
+pipe = subprocess.Popen(cmd, stdin=subprocess.PIPE, stderr=subprocess.PIPE)
+```
+
+**Fix**: Always redirect stderr to a file:
+
+```python
+stderr_fh = open(err_path, "w")
+pipe = subprocess.Popen(cmd, stdin=subprocess.PIPE,
+ stdout=subprocess.DEVNULL, stderr=stderr_fh)
+```
+
+### Frame Count Mismatch
+
+If the number of frames written to the pipe doesn't match what ffmpeg expects (based on `-r` and duration), the output may have:
+- Missing frames at the end
+- Incorrect duration
+- Audio-video desync
+
+**Fix**: Calculate frame count explicitly: `n_frames = int(duration * FPS)`. Don't use `range(int(start*FPS), int(end*FPS))` without verifying the total matches.
+
+### Concat Fails with "unsafe file name"
+
+```
+[concat @ ...] Unsafe file name
+```
+
+**Fix**: Always use `-safe 0`:
+```python
+["ffmpeg", "-f", "concat", "-safe", "0", "-i", concat_path, ...]
+```
+
+---
+
+## Font Issues
+
+### Cell Height (macOS Pillow)
+
+`textbbox()` and `getbbox()` return incorrect heights on some macOS Pillow versions. Use `getmetrics()`:
+
+```python
+ascent, descent = font.getmetrics()
+cell_height = ascent + descent # correct
+# NOT: font.getbbox("M")[3] # wrong on some versions
+```
+
+### Missing Unicode Glyphs
+
+Not all fonts render all Unicode characters. If a palette character isn't in the font, the glyph renders as a blank or tofu box, appearing as a dark hole in the output.
+
+**Fix**: Validate at init:
+
+```python
+all_chars = set()
+for pal in [PAL_DEFAULT, PAL_DENSE, PAL_RUNE, ...]:
+ all_chars.update(pal)
+
+valid_chars = set()
+for c in all_chars:
+ if c == " ":
+ valid_chars.add(c)
+ continue
+ img = Image.new("L", (20, 20), 0)
+ ImageDraw.Draw(img).text((0, 0), c, fill=255, font=font)
+ if np.array(img).max() > 0:
+ valid_chars.add(c)
+ else:
+ log(f"WARNING: '{c}' (U+{ord(c):04X}) missing from font")
+```
+
+### Platform Font Paths
+
+| Platform | Common Paths |
+|----------|-------------|
+| macOS | `/System/Library/Fonts/Menlo.ttc`, `/System/Library/Fonts/Monaco.ttf` |
+| Linux | `/usr/share/fonts/truetype/dejavu/DejaVuSansMono.ttf` |
+| Windows | `C:\Windows\Fonts\consola.ttf` (Consolas) |
+
+Always probe multiple paths and fall back gracefully. See `architecture.md` § Font Selection.
+
+---
+
+## Performance
+
+### Slow Shaders
+
+Some shaders use Python loops and are very slow at 1080p:
+
+| Shader | Issue | Fix |
+|--------|-------|-----|
+| `wave_distort` | Per-row Python loop | Use vectorized fancy indexing |
+| `halftone` | Triple-nested loop | Vectorize with block reduction |
+| `matrix rain` | Per-column per-trail loop | Accumulate index arrays, bulk assign |
+
+### Render Time Scaling
+
+If render is taking much longer than expected:
+1. Check grid count — each extra grid adds ~100-150ms/frame for init
+2. Check particle count — cap at quality-appropriate limits
+3. Check shader count — each shader adds 2-25ms
+4. Check for accidental Python loops in effects (should be numpy only)
+
+---
+
+## Common Mistakes
+
+### Using `r.S` vs the `S` Parameter
+
+The v2 scene protocol passes `S` (the state dict) as an explicit parameter. But `S` IS `r.S` — they're the same object. Both work:
+
+```python
+def fx_scene(r, f, t, S):
+ S["counter"] = S.get("counter", 0) + 1 # via parameter (preferred)
+ r.S["counter"] = r.S.get("counter", 0) + 1 # via renderer (also works)
+```
+
+Use the `S` parameter for clarity. The explicit parameter makes it obvious that the function has persistent state.
+
+### Forgetting to Handle Empty Feature Values
+
+Audio features default to 0.0 if the audio is silent. Use `.get()` with sensible defaults:
+
+```python
+energy = f.get("bass", 0.3) # default to 0.3, not 0
+```
+
+If you default to 0, effects go blank during silence.
+
+### Writing New Files Instead of Editing Existing State
+
+A common bug in particle systems: creating new arrays every frame instead of updating persistent state.
+
+```python
+# WRONG — particles reset every frame
+S["px"] = []
+for _ in range(100):
+ S["px"].append(random.random())
+
+# RIGHT — only initialize once, update each frame
+if "px" not in S:
+ S["px"] = []
+# ... emit new particles based on beats
+# ... update existing particles
+```
+
+### Not Clipping Value Fields
+
+Value fields should be [0, 1]. If they exceed this range, `val2char()` produces index errors:
+
+```python
+# WRONG — vf_plasma() * 1.5 can exceed 1.0
+val = vf_plasma(g, f, t, S) * 1.5
+
+# RIGHT — clip after scaling
+val = np.clip(vf_plasma(g, f, t, S) * 1.5, 0, 1)
+```
+
+The `_render_vf()` helper clips automatically, but if you're building custom scenes, clip explicitly.
+
+## Brightness Best Practices
+
+- Dense animated backgrounds — never flat black, always fill the grid
+- Vignette minimum clamped to 0.15 (not 0.12)
+- Bloom threshold 130 (not 170) so more pixels contribute to glow
+- Use `screen` blend mode (not `overlay`) for dark ASCII layers — overlay squares dark values: `2 * 0.12 * 0.12 = 0.03`
+- FeedbackBuffer decay minimum 0.5 — below that, feedback disappears too fast to see
+- Value field floor: `vf * 0.8 + 0.05` ensures no cell is truly zero
+- Per-scene gamma overrides: default 0.75, solarize 0.55, posterize 0.50, bright scenes 0.85
+- Test frames early: render single frames at key timestamps before committing to full render
+
+**Quick checklist before full render:**
+1. Render 3 test frames (start, middle, end)
+2. Check `canvas.mean() > 8` after tonemap
+3. Check no scene is visually flat black
+4. Verify per-section variation (different bg/palette/color per scene)
+5. Confirm shader chain includes bloom (threshold 130)
+6. Confirm vignette strength ≤ 0.25
diff --git a/src/skills/bundled/ast-grep/LICENSE b/src/skills/bundled/ast-grep/LICENSE
new file mode 100644
index 0000000..09aac3c
--- /dev/null
+++ b/src/skills/bundled/ast-grep/LICENSE
@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Yeongyu Kim
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
diff --git a/src/skills/bundled/ast-grep/SKILL.md b/src/skills/bundled/ast-grep/SKILL.md
new file mode 100644
index 0000000..581c235
--- /dev/null
+++ b/src/skills/bundled/ast-grep/SKILL.md
@@ -0,0 +1,288 @@
+---
+name: ast-grep
+description: "AST-aware structural code search and rewrite via ast-grep."
+version: 1.0.0
+author: Yeongyu Kim (code-yeongyu), adapted by Hermes Agent
+license: MIT
+platforms: [linux, macos, windows]
+metadata:
+ hermes:
+ tags: [ast, codemod, refactoring, structural-search, code-search, rewrite, tree-sitter]
+ category: software-development
+ related_skills: [simplify-code, systematic-debugging]
+---
+
+# ast-grep
+
+`ast-grep` (binary also named `sg`) is an **AST-aware search and rewrite tool** across 25 languages. It treats your pattern as code, parses it the same way it parses your project, and matches structurally. It is the right tool whenever your question depends on **code shape** rather than text bytes.
+
+This skill ships a Python wrapper at `scripts/ast_grep_helper.py` and platform install scripts at `install.sh` (POSIX) and `install.ps1` (Windows). The helper adds offline pattern validation, the two-pass write trick, and binary auto-resolution. Use it as your default entry point.
+
+Upstream source: vendored from [code-yeongyu/ast-grep-skill](https://github.com/code-yeongyu/ast-grep-skill) (MIT), as shipped in oh-my-openagent's shared-skills bundle.
+
+---
+
+## When to use this skill
+
+Use it whenever the question is about **code structure**, not bytes:
+
+- "Find every function that takes a `Request` parameter."
+- "Rewrite every `console.log(x)` to `logger.info(x)`."
+- "Strip every `as any` cast."
+- "Replace `require(...)` with `import` across the repo."
+- "Find empty catch blocks."
+- "Migrate `Optional[X]` to `X | None`."
+- "Apply this codemod across these 200 files."
+- "Run our YAML lint rules and surface violations."
+
+Switch to `search_files` (or plain `rg`) when the question is text-shaped (string literal contents, comments, license headers, file names, cross-language regex). When in doubt, ask: "does the answer depend on the language's syntax tree, or just on the file's bytes?" If the former, ast-grep. If the latter, search_files.
+
+Hermes integration notes:
+- Run the helper and `sg` through the `terminal` tool. Single-quote every pattern so the shell never expands `$VAR`.
+- For find→read chains around matches, use `--json-out` and process with `execute_code` rather than piping through interpreters.
+- This complements (does not replace) Hermes's `patch` tool: `patch` is for targeted edits you author; ast-grep is for pattern-driven bulk rewrites across many sites.
+
+---
+
+## Three things the agent must internalize
+
+### 1. ast-grep is NOT regex
+
+The wildcards are `$VAR` (one AST node) and `$$$` (zero or more nodes). Regex syntax fails silently:
+
+| You wrote | What ast-grep saw | What you wanted |
+|---|---|---|
+| `foo\|bar` | bitwise-or of `foo` and `bar` | run two separate searches |
+| `.*foo` | not parseable | `$$$ foo` (if `$$$` is a list of nodes) or use rg |
+| `\w+` | not parseable | `$VAR` to capture any identifier |
+| `[a-z]` | character class, not parseable | switch to rg |
+
+The full anti-pattern table is in `references/pitfalls.md` §1. The helper's `validate` subcommand catches these mechanically — call it before debugging "no matches" by hand.
+
+### 2. Patterns must be valid code
+
+The pattern itself must parse. `def $FN($$$):` fails because the trailing `:` makes it incomplete; use `def $FN($$$)`. `function $NAME` without params/body fails; use `function $NAME($$$) { $$$ }`. Full table per language in `references/pitfalls.md` §2.
+
+### 3. `--update-all` and `--json` are mutually exclusive (silently)
+
+This is the single biggest gotcha when scripting. `sg run -p P -r R --json --update-all` returns the JSON but **does not mutate files**. To both preview AND apply, run **two passes**:
+
+```bash
+sg run -p P -r R --json=compact . # pass 1: see what would change
+sg run -p P -r R --update-all . # pass 2: actually apply
+```
+
+The helper does this automatically when you call `replace --apply`. Read `references/pitfalls.md` §9.
+
+---
+
+## The helper script — `scripts/ast_grep_helper.py`
+
+A single-file Python 3 stdlib wrapper. Same on every OS. The agent's default entry point.
+
+### `search` — find all matches of a pattern
+
+```bash
+python scripts/ast_grep_helper.py search 'console.log($MSG)' --lang ts src/
+```
+
+Validates the pattern offline first. If the pattern looks like regex (`\w`, `.*`, `|`, etc.) the helper exits with a hint and never calls `sg` — saves a round-trip. Pass `--force` to skip validation.
+
+Flags:
+- `--lang ts` (or any of the 25 languages; aliases like `js`, `py`, `rs`, `kt` accepted)
+- `--globs '!**/*.test.ts'` (repeatable; prefix `!` to exclude)
+- `-C 3` (context lines)
+- `--json-out` (raw JSON instead of human format)
+
+### `replace` — rewrite by pattern, dry-run by default
+
+```bash
+# Dry-run preview (default — no files mutated)
+python scripts/ast_grep_helper.py replace 'console.log($MSG)' 'logger.info($MSG)' --lang ts src/
+
+# Actually apply
+python scripts/ast_grep_helper.py replace 'console.log($MSG)' 'logger.info($MSG)' --lang ts src/ --apply
+```
+
+The helper:
+1. Validates both `pattern` and `rewrite` for hint-detectable mistakes.
+2. Runs pass 1 with `--json=compact` to collect matches and show a preview.
+3. If `--apply` is set, runs pass 2 with `--update-all` to mutate files.
+
+### `scan` — run YAML rules
+
+```bash
+# Discover sgconfig.yml from cwd and run all rules
+python scripts/ast_grep_helper.py scan src/
+
+# Run a single rule file
+python scripts/ast_grep_helper.py scan -r rules/no-console.yml src/
+
+# Apply auto-fixes
+python scripts/ast_grep_helper.py scan -U src/
+
+# CI-friendly GitHub annotations
+python scripts/ast_grep_helper.py scan --report-style short src/
+```
+
+### `validate` — offline pattern check (no `sg` call)
+
+Useful for CI lints, pre-commit hooks, and quick sanity checks:
+
+```bash
+python scripts/ast_grep_helper.py validate '\w+' --lang ts
+# → exit 2: regex \w not supported. Use $VAR for identifiers.
+
+python scripts/ast_grep_helper.py validate 'console.log($MSG)' --lang ts
+# → exit 0: pattern looks plausible for ast-grep.
+```
+
+### `langs` / `doctor` / `install`
+
+```bash
+python scripts/ast_grep_helper.py langs # list 25 supported languages and aliases
+python scripts/ast_grep_helper.py doctor # check ast-grep binary availability
+python scripts/ast_grep_helper.py install # delegate to install.sh / install.ps1
+```
+
+`new` and `test` subcommands proxy directly to `sg new` and `sg test`.
+
+---
+
+## Direct `sg` use (when the helper isn't enough)
+
+The helper is opinionated. For full control, drop to `sg`. The skill ships a CLI cheat sheet in `references/cli.md`. The minimal idioms:
+
+```bash
+# Search
+sg run -p 'console.log($MSG)' --lang ts src/
+
+# Search with JSON for scripting
+sg run -p 'console.log($MSG)' --lang ts --json=compact src/
+
+# Rewrite, dry-run
+sg run -p 'console.log($MSG)' -r 'logger.info($MSG)' --lang ts --json=compact src/
+
+# Rewrite, apply
+sg run -p 'console.log($MSG)' -r 'logger.info($MSG)' --lang ts --update-all src/
+
+# Pattern from stdin (great for ad-hoc experiments)
+echo 'console.log("hi")' | sg run -p 'console.log($MSG)' --lang js --stdin
+
+# Debug a pattern that returns 0 matches
+sg run -p '' --lang --debug-query=ast --stdin <<< ''
+
+# Run YAML rules
+sg scan src/
+
+# Inline YAML rule (one-off)
+sg scan --inline-rules '
+id: no-todo
+language: TypeScript
+severity: warning
+rule: { pattern: TODO }' src/
+```
+
+When using `sg` directly in a shell, **always single-quote patterns** so `$VAR` is not expanded by the shell.
+
+---
+
+## Decision tree — what to use, when
+
+```
+USER asks for "find/rewrite/codemod"
+│
+├─ structural pattern (function shape, call, class, import, control flow)
+│ └→ ast-grep (this skill)
+│
+├─ text pattern (regex, alternation, character classes, file names)
+│ └→ search_files / rg
+│
+├─ semantic question (what variable does this refer to? does this throw?)
+│ └→ LSP tools, TypeScript compiler, Pyright, Semgrep with type inference
+│
+└─ multiple repos / federated search
+ └→ a search engine + then ast-grep / rg / LSP per-repo
+```
+
+If the user says "find all" or "every", default to ast-grep when the target is shaped (function, class, call, import, statement). Default to search_files when the target is text (string content, comment, license header, file name, identifier substring).
+
+---
+
+## Always run dry-run first when rewriting
+
+A bad pattern silently rewrites the wrong thing. The helper's `replace` defaults to dry-run for this reason. The flow is:
+
+1. Search to confirm matches: `helper search '' --lang X .`
+2. Dry-run rewrite: `helper replace '' '' --lang X .` (no `--apply`)
+3. Inspect the dry-run summary: number of matches, files affected, the per-location preview.
+4. If wrong: refine pattern, go back to step 1.
+5. If right: `helper replace '' '' --lang X . --apply`.
+
+Never apply a rewrite that you have not first dry-run. After an `--apply` in a git repo, review with `git diff --stat` before committing.
+
+---
+
+## When `sg` returns 0 matches but you know the code is there
+
+In priority order:
+
+1. **Run `helper validate '' --lang `** — catches regex misuse, missing function bodies, Python trailing colons.
+2. **Check `--lang`** — `sg` infers from extension; if you pass a `.tsx` file with `--lang ts` (not `tsx`), JSX won't parse.
+3. **Inspect the parsed pattern**: `sg run -p '' --lang --debug-query=ast --stdin <<< ''`. If it shows `ERROR` nodes, the pattern is malformed.
+4. **Check the AST of the target file**: `sg run -p '$_' --lang --debug-query=cst path/to/file | head -40` — find the `kind` you're trying to match.
+5. **Try the playground**: — paste code + pattern, see what's happening.
+
+Do not blindly retry with variations. Each failure has a reason; surface it.
+
+---
+
+## When to use YAML rules vs inline `-p` patterns
+
+**Use inline `-p`** when:
+- One-off ad-hoc query.
+- The pattern is simple (no constraints, no fix template).
+- You're exploring.
+
+**Use YAML rules** (file under `rules/`, run via `sg scan`) when:
+- The pattern is reused (lint rule, codemod that runs in CI).
+- You need `constraints`, `transform`, complex `inside`/`has`, or composite logic.
+- You want auto-fix (`fix:` field).
+- You want to test the rule (snapshot tests via `sg test`).
+
+The full YAML rule schema is in `references/yaml-rules.md`. Project setup (`sgconfig.yml`, `ruleDirs`, `utilDirs`) is in `references/sgconfig.md`.
+
+---
+
+## Output discipline
+
+- `sg run --json=compact` produces an array of match objects: `{ file, range: {start, end}, text, replacement?, lines, language, ... }`.
+- Without `--json`, `sg` produces human-readable colored output suitable for terminals.
+- The helper's default output is human-readable (file:line:column + match preview). Pass `--json-out` for raw JSON.
+- The helper's `replace` always summarizes: number of matches, number of files, per-location preview.
+
+When summarizing for the user, **always include the count of files affected**, not just the count of matches. Users care about blast radius.
+
+---
+
+## Required reading (in order of priority)
+
+1. `references/patterns.md` — meta-variables, naming rules, strictness levels. Read when you're unsure why a pattern doesn't match.
+2. `references/pitfalls.md` — the failure-mode field guide. Read when 0 matches surprises you.
+3. `references/recipes.md` — copy-paste patterns by language. Read first when you start a new task.
+4. `references/cli.md` — `sg run`, `sg scan`, `sg test`, `sg new`, `sg lsp`. Read when the helper isn't enough.
+5. `references/yaml-rules.md` — YAML rule schema. Read when you outgrow inline patterns.
+6. `references/sgconfig.md` — project-level configuration. Read when you set up `sg scan` for a real project.
+7. `references/install.md` — per-OS install methods. Read only if `install.sh` / `install.ps1` fail.
+
+---
+
+## Invariants (do not break)
+
+- **Validate before searching.** When emitting a pattern programmatically, call `helper validate` first. It catches the regex-misuse class of mistakes that account for ~70% of "0 matches" debug sessions.
+- **Dry-run before applying.** Never run `sg run -r ... --update-all` without first inspecting the matches. The helper's `replace` enforces this by default.
+- **Two-pass writes.** When using `sg` directly to both preview and apply, run two invocations — `--json` ignores `--update-all`.
+- **Single-quote patterns in shell.** `'$VAR'` not `"$VAR"`. The shell expands `$VAR` to the empty string in double quotes, breaking the pattern.
+- **Pattern is code, not regex.** When the pattern would need `|`, `.*`, `\w`, or `[a-z]`, switch to search_files instead. Don't try to force ast-grep into a regex shape.
+- **`--lang` is required for stdin.** When piping with `--stdin`, set `--lang` explicitly; `sg` cannot infer from extension.
+- **Linux: prefer `ast-grep` over `sg`** because `sg` collides with `setgroups` from `util-linux`. The helper handles this; if you call `sg` directly, alias it: `alias sg=ast-grep`.
diff --git a/src/skills/bundled/ast-grep/install.ps1 b/src/skills/bundled/ast-grep/install.ps1
new file mode 100644
index 0000000..65dd6d8
--- /dev/null
+++ b/src/skills/bundled/ast-grep/install.ps1
@@ -0,0 +1,235 @@
+#Requires -Version 5.1
+<#
+.SYNOPSIS
+ Install the ast-grep binary on Windows.
+
+.DESCRIPTION
+ Tries package managers in priority order, then falls back to downloading a
+ pinned release zip from GitHub into /bin/sg.exe.
+
+ Order:
+ 1. Already installed? -> nothing to do
+ 2. Scoop (most common Windows dev tool installer)
+ 3. Winget (Microsoft built-in)
+ 4. Chocolatey (choco)
+ 5. npm (@ast-grep/cli)
+ 6. cargo binstall / cargo install
+ 7. pip (ast-grep-cli)
+ 8. GitHub release zip -> /bin/sg.exe
+
+.PARAMETER Method
+ Force one method: scoop | winget | choco | npm | cargo | pip | github
+
+.PARAMETER Version
+ Pin a specific version when downloading from GitHub. Default: 0.45.0
+
+.PARAMETER NoFallback
+ Don't fall back to GitHub zip; fail if all package managers miss
+
+.PARAMETER Quiet
+ Suppress non-error output
+
+.EXAMPLE
+ .\install.ps1
+ .\install.ps1 -Method scoop
+ .\install.ps1 -Version 0.42.0 -Method github
+#>
+
+param(
+ [string]$Method = "",
+ [string]$Version = "0.45.0",
+ [switch]$NoFallback,
+ [switch]$Quiet
+)
+
+$ErrorActionPreference = 'Stop'
+$ScriptDir = Split-Path -Parent $MyInvocation.MyCommand.Path
+$CacheBinDir = if ($env:OMO_AST_GREP_BIN_DIR) { $env:OMO_AST_GREP_BIN_DIR } else { Join-Path $ScriptDir "bin" }
+
+function Log([string]$msg) {
+ if (-not $Quiet) {
+ [Console]::Error.WriteLine("[install.ps1] $msg")
+ }
+}
+
+function Err([string]$msg) {
+ [Console]::Error.WriteLine("[install.ps1] error: $msg")
+}
+
+function Has-Cmd([string]$name) {
+ $null -ne (Get-Command $name -ErrorAction SilentlyContinue)
+}
+
+function Test-AstGrep {
+ if (Has-Cmd 'ast-grep') { return $true }
+ if (Has-Cmd 'sg') { return $true }
+ if (Test-Path (Join-Path $CacheBinDir 'sg.exe')) { return $true }
+ if (Test-Path (Join-Path $CacheBinDir 'ast-grep.exe')) { return $true }
+ return $false
+}
+
+if (-not $Method -and (Test-AstGrep)) {
+ Log "ast-grep already installed"
+ exit 0
+}
+
+function Detect-Arch {
+ $a = $env:PROCESSOR_ARCHITECTURE
+ switch -Wildcard ($a) {
+ 'AMD64' { return 'x86_64' }
+ 'ARM64' { return 'aarch64' }
+ default { return 'unknown' }
+ }
+}
+
+$Arch = Detect-Arch
+
+function Try-Scoop {
+ if (-not (Has-Cmd 'scoop')) { return $false }
+ Log "trying: scoop install main/ast-grep"
+ try { scoop install main/ast-grep; return $LASTEXITCODE -eq 0 }
+ catch { return $false }
+}
+
+function Try-Winget {
+ if (-not (Has-Cmd 'winget')) { return $false }
+ Log "trying: winget install --id ast-grep.ast-grep"
+ try { winget install --id ast-grep.ast-grep --silent --accept-package-agreements --accept-source-agreements; return $LASTEXITCODE -eq 0 }
+ catch { return $false }
+}
+
+function Try-Choco {
+ if (-not (Has-Cmd 'choco')) { return $false }
+ Log "trying: choco install ast-grep -y"
+ try { choco install ast-grep -y; return $LASTEXITCODE -eq 0 }
+ catch { return $false }
+}
+
+function Try-Npm {
+ if (-not (Has-Cmd 'npm')) { return $false }
+ Log "trying: npm install -g @ast-grep/cli"
+ try { npm install -g '@ast-grep/cli'; return $LASTEXITCODE -eq 0 }
+ catch { return $false }
+}
+
+function Try-Cargo {
+ if (Has-Cmd 'cargo-binstall') {
+ Log "trying: cargo binstall -y ast-grep"
+ try { cargo binstall -y ast-grep; if ($LASTEXITCODE -eq 0) { return $true } } catch {}
+ }
+ if (-not (Has-Cmd 'cargo')) { return $false }
+ Log "trying: cargo install ast-grep --locked"
+ try { cargo install ast-grep --locked; return $LASTEXITCODE -eq 0 }
+ catch { return $false }
+}
+
+function Try-Pip {
+ $pip = $null
+ foreach ($p in 'pip3','pip','py') {
+ if (Has-Cmd $p) { $pip = $p; break }
+ }
+ if (-not $pip) { return $false }
+ Log "trying: $pip install --user ast-grep-cli"
+ try {
+ if ($pip -eq 'py') { py -m pip install --user ast-grep-cli }
+ else { & $pip install --user ast-grep-cli }
+ return $LASTEXITCODE -eq 0
+ } catch { return $false }
+}
+
+function Triple-For-Windows {
+ switch ($Arch) {
+ 'x86_64' { return 'x86_64-pc-windows-msvc' }
+ 'aarch64' { return 'aarch64-pc-windows-msvc' }
+ default { return '' }
+ }
+}
+
+function Try-Github {
+ $triple = Triple-For-Windows
+ if (-not $triple) {
+ Err "no GitHub release asset for arch $Arch"
+ return $false
+ }
+
+ $asset = "app-$triple.zip"
+ $url = "https://github.com/ast-grep/ast-grep/releases/download/$Version/$asset"
+ $tmp = Join-Path $env:TEMP ("ast-grep-install-" + [guid]::NewGuid().ToString('N').Substring(0,8))
+ New-Item -ItemType Directory -Path $tmp -Force | Out-Null
+ try {
+ Log "downloading $url"
+ Invoke-WebRequest -Uri $url -OutFile (Join-Path $tmp $asset) -UseBasicParsing
+ Expand-Archive -Path (Join-Path $tmp $asset) -DestinationPath (Join-Path $tmp 'extract') -Force
+
+ New-Item -ItemType Directory -Path $CacheBinDir -Force | Out-Null
+
+ $candidates = @(
+ (Join-Path $tmp 'extract/ast-grep.exe'),
+ (Join-Path $tmp 'extract/sg.exe')
+ )
+ $src = $null
+ foreach ($c in $candidates) {
+ if (Test-Path $c) { $src = $c; break }
+ }
+ if (-not $src) {
+ Err "no ast-grep.exe or sg.exe found inside $asset"
+ return $false
+ }
+
+ $dest = Join-Path $CacheBinDir 'sg.exe'
+ Copy-Item -Path $src -Destination $dest -Force
+ Log "installed cached binary: $dest"
+ Log "verify: & '$dest' --version"
+ Log ""
+ Log "Add to PATH for direct sg use:"
+ Log " `$env:Path = '$CacheBinDir;' + `$env:Path"
+ return $true
+ } finally {
+ Remove-Item -Recurse -Force $tmp -ErrorAction SilentlyContinue
+ }
+}
+
+function Run-Method([string]$m) {
+ switch ($m) {
+ 'scoop' { return Try-Scoop }
+ 'winget' { return Try-Winget }
+ 'choco' { return Try-Choco }
+ 'npm' { return Try-Npm }
+ 'cargo' { return Try-Cargo }
+ 'pip' { return Try-Pip }
+ 'github' { return Try-Github }
+ default { Err "unknown method: $m"; return $false }
+ }
+}
+
+if ($Method) {
+ if (Run-Method $Method) { exit 0 }
+ Err "method '$Method' failed"
+ exit 2
+}
+
+$methods = @('scoop', 'winget', 'choco', 'npm', 'cargo', 'pip')
+foreach ($m in $methods) {
+ if (Run-Method $m) {
+ Log "installed via $m"
+ exit 0
+ }
+ Log "$m unavailable or failed; trying next"
+}
+
+if (-not $NoFallback) {
+ Log "all package managers failed; falling back to GitHub release"
+ if (Try-Github) { exit 0 }
+}
+
+Err "all install methods failed."
+Err ""
+Err "Manual options:"
+Err " scoop install main/ast-grep # Scoop"
+Err " winget install --id ast-grep.ast-grep # Winget"
+Err " choco install ast-grep # Chocolatey"
+Err " npm install -g @ast-grep/cli # any OS with Node"
+Err " cargo install ast-grep --locked # any OS with Rust"
+Err " pip install ast-grep-cli # any OS with Python"
+Err " https://github.com/ast-grep/ast-grep/releases # manual binary"
+exit 2
diff --git a/src/skills/bundled/ast-grep/install.sh b/src/skills/bundled/ast-grep/install.sh
new file mode 100644
index 0000000..178e7fc
--- /dev/null
+++ b/src/skills/bundled/ast-grep/install.sh
@@ -0,0 +1,286 @@
+#!/usr/bin/env bash
+#
+# install.sh - install the ast-grep binary on POSIX systems (macOS, Linux, WSL, Git Bash).
+#
+# Tries package managers in priority order, then falls back to downloading a
+# pinned release binary from GitHub into /bin/sg.
+#
+# Order:
+# 1. Already installed? -> nothing to do
+# 2. Homebrew (brew)
+# 3. npm (@ast-grep/cli)
+# 4. cargo binstall (faster) or cargo install (slower)
+# 5. pip (ast-grep-cli)
+# 6. nix-env (NixOS / Nix users)
+# 7. mise (asdf successor)
+# 8. GitHub release tarball -> /bin/sg
+#
+# Flags:
+# --method= Force one method: brew | npm | cargo | pip | nix | mise | github
+# --version= Pin a specific version when downloading from GitHub
+# --no-fallback Don't fall back to GitHub tarball; fail if all package managers miss
+# --quiet, -q Suppress non-error output
+#
+# Exit codes:
+# 0 Installed (or already present)
+# 1 Argument error
+# 2 All install methods failed
+# 3 Network failure during GitHub fallback
+
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+SKILL_ROOT="$SCRIPT_DIR"
+CACHE_BIN_DIR="${OMO_AST_GREP_BIN_DIR:-$SKILL_ROOT/bin}"
+
+PINNED_VERSION="0.45.0"
+FORCED_METHOD=""
+USE_FALLBACK=1
+QUIET=0
+
+log() {
+ if [ "$QUIET" -eq 0 ]; then
+ printf '[install.sh] %s\n' "$*" >&2
+ fi
+}
+
+err() {
+ printf '[install.sh] error: %s\n' "$*" >&2
+}
+
+usage() {
+ sed -n '2,/^set -/p' "${BASH_SOURCE[0]}" | sed 's/^# \{0,1\}//;/^set -/d'
+ exit "${1:-0}"
+}
+
+while [ "$#" -gt 0 ]; do
+ case "$1" in
+ --method=*) FORCED_METHOD="${1#*=}" ;;
+ --version=*) PINNED_VERSION="${1#*=}" ;;
+ --no-fallback) USE_FALLBACK=0 ;;
+ --quiet|-q) QUIET=1 ;;
+ --help|-h) usage 0 ;;
+ *) err "unknown argument: $1"; usage 1 ;;
+ esac
+ shift
+done
+
+# --- detect platform -----------------------------------------------------
+
+detect_os() {
+ case "$(uname -s)" in
+ Darwin) echo "darwin" ;;
+ Linux) echo "linux" ;;
+ MINGW*|MSYS*|CYGWIN*) echo "windows" ;;
+ *) echo "unknown" ;;
+ esac
+}
+
+detect_arch() {
+ case "$(uname -m)" in
+ arm64|aarch64) echo "aarch64" ;;
+ x86_64|amd64) echo "x86_64" ;;
+ *) echo "unknown" ;;
+ esac
+}
+
+OS="$(detect_os)"
+ARCH="$(detect_arch)"
+
+# --- already installed? --------------------------------------------------
+
+ast_grep_present() {
+ if command -v ast-grep >/dev/null 2>&1; then
+ return 0
+ fi
+ if command -v sg >/dev/null 2>&1; then
+ if [ "$OS" = "linux" ]; then
+ if "$(command -v sg)" --version 2>/dev/null | grep -qi 'ast-grep'; then
+ return 0
+ fi
+ return 1
+ fi
+ return 0
+ fi
+ if [ -x "$CACHE_BIN_DIR/sg" ] || [ -x "$CACHE_BIN_DIR/ast-grep" ]; then
+ return 0
+ fi
+ return 1
+}
+
+if [ -z "$FORCED_METHOD" ] && ast_grep_present; then
+ log "ast-grep already installed: $(command -v ast-grep 2>/dev/null || command -v sg)"
+ exit 0
+fi
+
+# --- per-method installers -----------------------------------------------
+
+try_brew() {
+ command -v brew >/dev/null 2>&1 || return 1
+ log "trying: brew install ast-grep"
+ brew install ast-grep && return 0 || return 1
+}
+
+try_npm() {
+ command -v npm >/dev/null 2>&1 || return 1
+ log "trying: npm install -g @ast-grep/cli"
+ npm install -g @ast-grep/cli && return 0 || return 1
+}
+
+try_cargo() {
+ if command -v cargo-binstall >/dev/null 2>&1; then
+ log "trying: cargo binstall ast-grep"
+ cargo binstall -y ast-grep && return 0 || true
+ fi
+ if command -v cargo >/dev/null 2>&1; then
+ log "trying: cargo install ast-grep --locked"
+ cargo install ast-grep --locked && return 0 || return 1
+ fi
+ return 1
+}
+
+try_pip() {
+ if command -v pipx >/dev/null 2>&1; then
+ log "trying: pipx install ast-grep-cli"
+ pipx install ast-grep-cli && return 0 || true
+ fi
+ command -v pip3 >/dev/null 2>&1 || command -v pip >/dev/null 2>&1 || return 1
+ PIP="$(command -v pip3 || command -v pip)"
+ log "trying: $PIP install --user ast-grep-cli"
+ $PIP install --user ast-grep-cli && return 0 || return 1
+}
+
+try_nix() {
+ command -v nix-env >/dev/null 2>&1 || return 1
+ log "trying: nix-env -iA nixpkgs.ast-grep"
+ nix-env -iA nixpkgs.ast-grep && return 0 || return 1
+}
+
+try_mise() {
+ command -v mise >/dev/null 2>&1 || return 1
+ log "trying: mise use -g ast-grep"
+ mise use -g ast-grep && return 0 || return 1
+}
+
+# Tarball assets are named like:
+# app-aarch64-apple-darwin.zip
+# app-x86_64-apple-darwin.zip
+# app-aarch64-unknown-linux-gnu.zip
+# app-x86_64-unknown-linux-gnu.zip
+# app-x86_64-pc-windows-msvc.zip (.zip only on windows)
+
+triple_for() {
+ case "$OS-$ARCH" in
+ darwin-aarch64) echo "aarch64-apple-darwin" ;;
+ darwin-x86_64) echo "x86_64-apple-darwin" ;;
+ linux-aarch64) echo "aarch64-unknown-linux-gnu" ;;
+ linux-x86_64) echo "x86_64-unknown-linux-gnu" ;;
+ *) echo "" ;;
+ esac
+}
+
+try_github() {
+ TRIPLE="$(triple_for)"
+ if [ -z "$TRIPLE" ]; then
+ err "no GitHub release asset for $OS-$ARCH; install via package manager or build from source."
+ return 1
+ fi
+
+ command -v curl >/dev/null 2>&1 || command -v wget >/dev/null 2>&1 || {
+ err "need curl or wget for GitHub fallback"
+ return 1
+ }
+
+ ASSET="app-${TRIPLE}.zip"
+ URL="https://github.com/ast-grep/ast-grep/releases/download/${PINNED_VERSION}/${ASSET}"
+ TMP="$(mktemp -d -t ast-grep-install-XXXXXX)"
+ trap 'rm -rf "$TMP"' RETURN
+
+ log "downloading $URL"
+ if command -v curl >/dev/null 2>&1; then
+ curl -fsSL "$URL" -o "$TMP/$ASSET" || return 3
+ else
+ wget -q "$URL" -O "$TMP/$ASSET" || return 3
+ fi
+
+ command -v unzip >/dev/null 2>&1 || {
+ err "need 'unzip' to extract GitHub release archives"
+ return 1
+ }
+
+ unzip -q "$TMP/$ASSET" -d "$TMP/extract"
+ mkdir -p "$CACHE_BIN_DIR"
+ if [ -f "$TMP/extract/ast-grep" ]; then
+ mv "$TMP/extract/ast-grep" "$CACHE_BIN_DIR/sg"
+ elif [ -f "$TMP/extract/sg" ]; then
+ mv "$TMP/extract/sg" "$CACHE_BIN_DIR/sg"
+ else
+ err "no ast-grep or sg binary found inside $ASSET"
+ return 1
+ fi
+ chmod +x "$CACHE_BIN_DIR/sg"
+
+ log "installed cached binary: $CACHE_BIN_DIR/sg"
+ log "verify: $CACHE_BIN_DIR/sg --version"
+ log ""
+ log "Add to PATH for direct sg use:"
+ log " export PATH=\"$CACHE_BIN_DIR:\$PATH\""
+ return 0
+}
+
+# --- dispatch ------------------------------------------------------------
+
+run_method() {
+ case "$1" in
+ brew) try_brew ;;
+ npm) try_npm ;;
+ cargo) try_cargo ;;
+ pip) try_pip ;;
+ nix) try_nix ;;
+ mise) try_mise ;;
+ github) try_github ;;
+ *) err "unknown method: $1"; return 1 ;;
+ esac
+}
+
+if [ -n "$FORCED_METHOD" ]; then
+ if run_method "$FORCED_METHOD"; then
+ exit 0
+ else
+ err "method '$FORCED_METHOD' failed"
+ exit 2
+ fi
+fi
+
+# Try methods in OS-aware priority order.
+case "$OS" in
+ darwin) METHODS=(brew npm cargo pip mise) ;;
+ linux) METHODS=(npm cargo pip nix mise brew) ;;
+ windows) METHODS=(npm cargo pip mise) ;;
+ *) METHODS=(npm cargo pip) ;;
+esac
+
+for m in "${METHODS[@]}"; do
+ if run_method "$m"; then
+ log "installed via $m"
+ exit 0
+ fi
+ log "$m unavailable or failed; trying next"
+done
+
+if [ "$USE_FALLBACK" -eq 1 ]; then
+ log "all package managers failed; falling back to GitHub release"
+ if try_github; then
+ exit 0
+ fi
+fi
+
+err "all install methods failed."
+err ""
+err "Manual options:"
+err " brew install ast-grep # macOS / linuxbrew"
+err " npm install -g @ast-grep/cli # any OS with Node"
+err " cargo install ast-grep --locked # any OS with Rust"
+err " pip install ast-grep-cli # any OS with Python"
+err " https://github.com/ast-grep/ast-grep/releases # manual binary"
+exit 2
diff --git a/src/skills/bundled/ast-grep/references/cli.md b/src/skills/bundled/ast-grep/references/cli.md
new file mode 100644
index 0000000..d887e1f
--- /dev/null
+++ b/src/skills/bundled/ast-grep/references/cli.md
@@ -0,0 +1,231 @@
+# CLI reference — `sg` / `ast-grep`
+
+Compact reference for the underlying `sg` binary that the helper wraps. Use this when the helper isn't enough or when you want to invoke `sg` directly.
+
+> **Binary name on Linux**: prefer `ast-grep` over `sg` because `sg` collides with `setgroups` from `util-linux`.
+
+---
+
+## `sg run` — one-shot search/rewrite
+
+The default subcommand. `sg -p 'foo'` is shorthand for `sg run -p 'foo'`.
+
+```bash
+sg run [OPTIONS] --pattern [PATHS...]
+```
+
+| Flag | Purpose |
+|---|---|
+| `-p, --pattern
` | AST pattern to match. **Always single-quote** in shell to prevent `$VAR` expansion. |
+| `-r, --rewrite ` | Replacement pattern. Used with `-U` to apply. |
+| `-l, --lang ` | Language. Inferred from path extension if omitted. |
+| `--selector ` | When the pattern is ambiguous, extract only this AST kind. |
+| `--strictness ` | `cst` \| `smart` (default) \| `ast` \| `relaxed` \| `signature` |
+| `--debug-query[=]` | Print parsed pattern. F: `pattern` \| `ast` \| `cst` \| `sexp` |
+| `--stdin` | Read code from stdin instead of files. Lang must be set. |
+| `--globs ` | Include/exclude glob (repeatable; prefix `!` to exclude). |
+| `--follow` | Follow symlinks. |
+| `--no-ignore ` | Disable a class of ignore: `hidden`, `dot`, `exclude`, `global`, `parent`, `vcs`. |
+| `-i, --interactive` | Step through matches and confirm each rewrite. |
+| `-U, --update-all` | Apply all rewrites without confirmation. **Mutually exclusive with `--json`** (silently). |
+| `--json[=]` | Emit JSON. S: `pretty` \| `stream` \| `compact` (compact is best for piping). |
+| `--color ` | `auto` \| `always` \| `ansi` \| `never` |
+| `--inspect ` | Detail level: `nothing` \| `summary` \| `entity` |
+| `-A, -B, -C ` | Context lines after / before / around each match. |
+| `-j, --threads ` | Thread count (default: heuristic; `0` = auto). |
+
+### `--update-all` + `--json` — the trap
+
+`sg` silently ignores `--update-all` when `--json` is set. To preview AND apply, run **two passes**:
+
+```bash
+# Pass 1: preview
+sg run -p 'foo()' -r 'bar()' --json=compact src/
+
+# Pass 2: apply
+sg run -p 'foo()' -r 'bar()' --update-all src/
+```
+
+The `ast_grep_helper.py replace --apply` subcommand does this automatically.
+
+### Examples
+
+```bash
+# Basic search
+sg run -p 'console.log($MSG)' --lang ts src/
+
+# Search with context lines
+sg run -p 'eval($CODE)' --lang js -C 3 .
+
+# Rewrite, dry-run preview as JSON
+sg run -p 'console.log($MSG)' -r 'logger.info($MSG)' --json=compact --lang ts src/
+
+# Rewrite, apply
+sg run -p 'console.log($MSG)' -r 'logger.info($MSG)' --update-all --lang ts src/
+
+# Pattern from stdin
+echo 'console.log("x")' | sg run -p 'console.log($MSG)' --lang js --stdin
+
+# Limit to specific files
+sg run -p 'foo()' --lang ts --globs 'src/**/*.ts' --globs '!**/*.test.ts' .
+
+# Debug a pattern that returns 0 matches
+sg run -p 'def $F($$$):' --lang py --debug-query=ast --stdin <<< 'def foo(): pass'
+```
+
+---
+
+## `sg scan` — YAML rule scanner
+
+Run a configuration of YAML rules across files. Used for project-wide lints and codemods.
+
+```bash
+sg scan [OPTIONS] [PATHS...]
+```
+
+| Flag | Purpose |
+|---|---|
+| `-c, --config ` | Path to `sgconfig.yml` (default: walk up from cwd looking for one). |
+| `-r, --rule ` | Run a **single** rule file. Mutually exclusive with `--config`. |
+| `--inline-rules ` | Pass YAML rule text inline. Use `---` to separate multiple rules. |
+| `--filter ` | Only run rules whose `id` matches this regex. |
+| `--include-metadata` | Include rule `metadata` field in JSON output. |
+| `-U, --update-all` | Apply fixes from `fix:` automatically. |
+| `--report-style ` | `rich` \| `medium` \| `short` |
+| `--format ` | `github` \| `sarif` (CI-friendly outputs). |
+| `--error[=ID]`, `--warning[=ID]`, `--info[=ID]`, `--hint[=ID]`, `--off[=ID]` | Promote/demote severity. |
+| `-i, --interactive` | Confirm each fix interactively. |
+| `--json[=]` | JSON output. |
+
+### Examples
+
+```bash
+# Run all rules in sgconfig.yml-discovered ruleDirs
+sg scan src/
+
+# Run a single rule file (no sgconfig.yml needed)
+sg scan -r rules/no-console.yml src/
+
+# Inline rule (great for one-offs and CI)
+sg scan --inline-rules '
+id: no-todo
+language: TypeScript
+severity: warning
+rule: { pattern: TODO }' src/
+
+# Apply all auto-fixes
+sg scan -U src/
+
+# CI-friendly GitHub annotations
+sg scan --format github src/
+
+# SARIF for security scanners
+sg scan --format sarif src/ > sarif.json
+```
+
+---
+
+## `sg test` — run rule snapshot tests
+
+```bash
+sg test [OPTIONS]
+```
+
+| Flag | Purpose |
+|---|---|
+| `-c, --config ` | Path to `sgconfig.yml`. |
+| `-t, --test-dir ` | Test directory. |
+| `--snapshot-dir ` | Snapshot directory (default: `__snapshots__`). |
+| `--skip-snapshot-tests` | Validate test code parses; don't compare snapshots. |
+| `-U, --update-all` | Update all changed snapshots. |
+| `-f, --filter ` | Filter test cases by glob on rule id. |
+| `--include-off` | Include rules with severity `off`. |
+| `-i, --interactive` | Step through changed snapshots and accept/reject each. |
+
+A test directory looks like:
+
+```
+test/
+├── no-console.yml # `valid:` and `invalid:` snippets
+└── no-console-test.yml # alternative test file format
+__snapshots__/
+└── no-console-snapshot.yml # expected match locations
+```
+
+---
+
+## `sg new` — scaffold
+
+```bash
+sg new [NAME] [OPTIONS]
+```
+
+| Subcommand | Creates |
+|---|---|
+| `project` | `sgconfig.yml`, `rules/`, `utils/`, `__snapshots__/` directory tree |
+| `rule` | A new YAML rule file in the first `ruleDirs` entry |
+| `test` | A new test file in `testConfigs[0].testDir` |
+| `util` | A new utility rule in the first `utilDirs` entry |
+
+```bash
+# New project in current dir
+sg new project --yes
+
+# New rule
+sg new rule no-console --lang typescript
+
+# New test
+sg new test no-console --yes
+```
+
+---
+
+## `sg lsp` — language server
+
+```bash
+sg lsp -c sgconfig.yml
+```
+
+Speak LSP over stdin/stdout. Configure your editor (VS Code extension, Neovim `nvim-lspconfig`, Helix `languages.toml`) to spawn this command for live diagnostics.
+
+---
+
+## `sg completions` — shell completions
+
+```bash
+sg completions bash >> ~/.bashrc
+sg completions zsh > "${fpath[1]}/_sg"
+sg completions fish > ~/.config/fish/completions/sg.fish
+sg completions powershell >> $PROFILE
+```
+
+---
+
+## Useful one-liners
+
+```bash
+# Count matches per file
+sg run -p 'console.log($_)' --lang ts --json=compact . \
+ | jq -r '.[].file' | sort | uniq -c | sort -rn
+
+# Find all unique kinds in a file (great for figuring out kind names)
+sg run -p '$_' --lang ts --debug-query=cst src/foo.ts \
+ | grep -oE 'kind: [a-z_]+' | sort -u
+
+# Rewrite only in a subset of files
+sg run -p 'foo()' -r 'bar()' --update-all --globs 'src/**/*.ts' --globs '!src/legacy/**' .
+
+# Apply fixes from many rules but only ones matching a pattern in their id
+sg scan --filter 'no-' -U src/
+
+# Use ast-grep as a linter in pre-commit
+sg scan --format github src/ || exit 1
+```
+
+---
+
+## See also
+
+- `references/yaml-rules.md` — rule schema (`pattern`, `kind`, `regex`, `inside`, `has`, `all`, `any`, `not`, `matches`, `transform`, `fix`).
+- `references/sgconfig.md` — project configuration.
+- Official:
diff --git a/src/skills/bundled/ast-grep/references/install.md b/src/skills/bundled/ast-grep/references/install.md
new file mode 100644
index 0000000..49a3e41
--- /dev/null
+++ b/src/skills/bundled/ast-grep/references/install.md
@@ -0,0 +1,166 @@
+# Install ast-grep
+
+The skill ships an `install.sh` (POSIX) and `install.ps1` (Windows) that try every reasonable method in priority order and fall back to a GitHub release download as a last resort. **You usually do not need to read this page.** Run the installer:
+
+```bash
+bash install.sh # macOS / Linux / WSL / Git Bash
+pwsh -File install.ps1 # Windows PowerShell
+```
+
+This page exists for the (rare) case the installer cannot find a working method, or you want to install ast-grep manually.
+
+---
+
+## Per-OS install commands (verbatim, copy-paste)
+
+### macOS
+
+```bash
+brew install ast-grep # Homebrew - the primary path
+sudo port install ast-grep # MacPorts
+npm install -g @ast-grep/cli # if you have Node already
+cargo install ast-grep --locked # if you have Rust already
+```
+
+### Linux
+
+```bash
+# Universal (works on every distro)
+npm install -g @ast-grep/cli
+cargo install ast-grep --locked
+pip install ast-grep-cli
+
+# Distro-specific
+nix-env -iA nixpkgs.ast-grep # NixOS / Nix
+brew install ast-grep # Linuxbrew
+
+# NixOS shell.nix
+nix-shell -p ast-grep
+```
+
+> **Linux gotcha**: the binary is named `sg`, but on most Linux systems `sg` is also the **`setgroups` command** from `util-linux`. The shell sees `setgroups` first and ignores ast-grep. Two options:
+>
+> 1. Always invoke `ast-grep` (full name).
+> 2. Add an alias: `alias sg=ast-grep` in your `~/.bashrc` / `~/.zshrc`.
+>
+> The `ast_grep_helper.py` script in `scripts/` already handles this — when it sees `sg` on PATH on Linux, it runs `--version` and rejects the binary if it isn't ast-grep.
+
+### Windows
+
+```powershell
+scoop install main/ast-grep # Scoop (most common on dev machines)
+winget install --id ast-grep.ast-grep # Winget (Microsoft built-in)
+choco install ast-grep # Chocolatey
+npm install -g @ast-grep/cli # any OS with Node
+cargo install ast-grep --locked # any OS with Rust
+```
+
+### WSL / Git Bash on Windows
+
+Treat as Linux. Use `npm`, `cargo`, `pip`, or `bash install.sh`.
+
+---
+
+## Cross-platform / language-ecosystem methods
+
+These work on every OS:
+
+| Method | Command | Pros | Cons |
+|---|---|---|---|
+| **npm** | `npm install -g @ast-grep/cli` | Fast, prebuilt platform binaries | Needs Node 18+ |
+| **cargo** | `cargo install ast-grep --locked` | Always builds latest from source | Slow (~3-5 min compile) |
+| **cargo binstall** | `cargo binstall ast-grep` | Fast (downloads release binary) | Needs `cargo-binstall` first |
+| **pip** | `pip install ast-grep-cli` | Works in any Python venv | Needs Python 3.8+ |
+| **pipx** | `pipx install ast-grep-cli` | Isolated install | Needs pipx |
+| **mise** | `mise use -g ast-grep` | asdf successor, version-pinning | Needs mise |
+| **GitHub release** | manual download | Pure binary, no toolchain | Manual PATH setup |
+
+---
+
+## GitHub release manual install
+
+If every package manager fails:
+
+```bash
+# 1. Pick the right asset for your OS+arch from the latest release:
+# https://github.com/ast-grep/ast-grep/releases/latest
+#
+# Naming pattern:
+# app-aarch64-apple-darwin.zip macOS Apple Silicon
+# app-x86_64-apple-darwin.zip macOS Intel
+# app-aarch64-unknown-linux-gnu.zip Linux ARM64 (glibc)
+# app-x86_64-unknown-linux-gnu.zip Linux x86_64 (glibc)
+# app-x86_64-pc-windows-msvc.zip Windows x86_64
+# app-aarch64-pc-windows-msvc.zip Windows ARM64
+
+# 2. Download and extract:
+VERSION=0.45.0
+TRIPLE=aarch64-apple-darwin
+curl -fsSL "https://github.com/ast-grep/ast-grep/releases/download/${VERSION}/app-${TRIPLE}.zip" -o /tmp/ast-grep.zip
+unzip /tmp/ast-grep.zip -d /tmp/ast-grep
+sudo mv /tmp/ast-grep/ast-grep /usr/local/bin/sg
+sudo chmod +x /usr/local/bin/sg
+
+# 3. Verify:
+sg --version
+```
+
+The skill's `install.sh` does steps 1-3 automatically and drops the binary in `/bin/sg` so you can use it without sudo.
+
+---
+
+## Build from source
+
+```bash
+git clone https://github.com/ast-grep/ast-grep.git
+cd ast-grep
+cargo install --path ./crates/cli --locked
+```
+
+Requires Rust 1.74+. Slowest path; only useful when you need a specific commit or unreleased fix.
+
+---
+
+## Verifying the install
+
+```bash
+ast-grep --version # or `sg --version`
+# ast-grep 0.45.0
+```
+
+Then sanity-check a real query:
+
+```bash
+echo 'console.log("hello")' | sg run -p 'console.log($MSG)' --lang js --stdin
+```
+
+Expected: a single match with the `console.log("hello")` call highlighted.
+
+---
+
+## Editor integration
+
+After installing the CLI, set up your editor:
+
+- **VS Code**: install the [`ast-grep`](https://marketplace.visualstudio.com/items?itemName=ast-grep.ast-grep-vscode) extension. Requires `sgconfig.yml` in workspace root for live diagnostics.
+- **Neovim**: configure `nvim-lspconfig` with `ast_grep` server, or install [`telescope-ast-grep.nvim`](https://github.com/ray-x/telescope-ast-grep.nvim).
+- **Helix**: add `ast-grep lsp` as a language server in `languages.toml`.
+- **Emacs**: install [`ast-grep.el`](https://github.com/SunskyXH/ast-grep.el).
+
+See `references/cli.md` for `ast-grep lsp` flags.
+
+---
+
+## Uninstall
+
+| Method | Command |
+|---|---|
+| brew | `brew uninstall ast-grep` |
+| npm | `npm uninstall -g @ast-grep/cli` |
+| cargo | `cargo uninstall ast-grep` |
+| pip | `pip uninstall ast-grep-cli` |
+| pipx | `pipx uninstall ast-grep-cli` |
+| scoop | `scoop uninstall ast-grep` |
+| winget | `winget uninstall --id ast-grep.ast-grep` |
+| choco | `choco uninstall ast-grep` |
+| GitHub binary | `rm /bin/sg` |
diff --git a/src/skills/bundled/ast-grep/references/patterns.md b/src/skills/bundled/ast-grep/references/patterns.md
new file mode 100644
index 0000000..6c0b5bb
--- /dev/null
+++ b/src/skills/bundled/ast-grep/references/patterns.md
@@ -0,0 +1,147 @@
+# Pattern syntax — meta-variables and how patterns parse
+
+ast-grep is **not regex**. Patterns are written in the **same syntax as the target language** (TypeScript, Python, Go, etc.), and ast-grep matches them against the AST of every file. The wildcards are called **meta-variables**.
+
+This page is the canonical primer. If a pattern fails, 90% of the time it is one of the issues on this page.
+
+---
+
+## The three meta-variables
+
+| Syntax | Matches | Capture |
+|---|---|---|
+| `$VAR` | exactly **one** AST node | yes, by name |
+| `$$$` | **zero or more** AST nodes (a list) | no (anonymous) |
+| `$$$VAR` | zero or more AST nodes | yes, by name |
+| `$_` | one AST node | no (anonymous) |
+
+A meta-variable always replaces a **whole AST node**, never a substring of a node. `$VAR` cannot match the first three characters of an identifier, only an entire identifier (or expression, or statement, depending on context).
+
+### Naming rules
+
+- Must start with `$`.
+- Then uppercase letters `A-Z`, digits, or underscores.
+- **Valid**: `$X`, `$VAR`, `$VAR_1`, `$_`, `$_VAR`, `$ARG1`.
+- **Invalid**: `$lower`, `$kebab-case`, `$1` (digit first), `$$single` (use `$_` for anonymous).
+
+### Same-name = same content
+
+Two occurrences of the same metavariable in a pattern must capture **identical text**:
+
+```ts
+// Pattern
+$X === $X
+
+// Matches
+a === a
+foo.bar === foo.bar
+
+// Does NOT match
+a === b
+foo === foo.bar
+```
+
+Useful for finding redundant comparisons, double assignments, etc.
+
+### `$$$` is greedy
+
+When you write `foo($$$A, b, $$$C)`, the matcher does **not** backtrack or try every possible split. It greedily fills `$$$A` until the pattern can match `b`, then everything left goes into `$$$C`.
+
+```ts
+// Pattern
+foo($$$A, b, $$$C)
+
+// Input
+foo(a, c, b, b, c)
+
+// Capture
+$$$A = [a, c]
+$$$C = [b, c]
+```
+
+If you need a different split, restructure the pattern (e.g. add a constraint).
+
+---
+
+## Patterns must be valid code
+
+The pattern itself must parse with the target language's grammar. ast-grep treats `$VAR` and `$$$` as identifiers/argument lists during parsing, then matches structurally.
+
+### What goes wrong
+
+| Bad pattern | Why it fails | Fix |
+|---|---|---|
+| `function $NAME` | Function declaration without body — not a valid AST node in JS/TS/Go/Rust. | `function $NAME($$$) { $$$ }` |
+| `def $FN($$$):` | Trailing colon. ast-grep parses as a complete function definition; the colon makes it a statement. | `def $FN($$$)` |
+| `class Foo:` | Same — Python class without body. | `class Foo($$$)` |
+| `fn $NAME` | Rust fn without signature. | `fn $NAME($$$) -> $RET { $$$ }` |
+| `if x` | Incomplete `if` — most languages require the body. | `if x { $$$ }` (curly-brace languages) or `if x: $$$` (Python uses `pattern.context`/`selector` instead) |
+| `"key": "$VAL"` | JSON pattern — a key/value pair on its own isn't valid JSON. | Use `pattern: { context: '{"key": "$VAL"}', selector: pair }` |
+
+### When a sub-expression isn't valid on its own
+
+Sometimes you want to match an *expression* that the language only allows inside a larger context. Use the `pattern` object form:
+
+```yaml
+pattern:
+ context: 'class A { $FIELD = $INIT }'
+ selector: field_definition
+```
+
+This says: parse `class A { $FIELD = $INIT }` as a whole, then keep only the `field_definition` sub-tree as the actual pattern.
+
+---
+
+## Strictness levels
+
+When CST nodes don't match exactly (extra whitespace, different unnamed punctuation), ast-grep can be more or less forgiving. Pass `--strictness ` on the CLI, or set it in a YAML rule:
+
+| Level | Matches |
+|---|---|
+| `cst` | Every node, including unnamed (commas, parens, etc.) |
+| `smart` (default) | All except unnamed nodes in the **target** that aren't in the pattern |
+| `ast` | Only named AST nodes |
+| `relaxed` | Named AST nodes, ignoring comments |
+| `signature` | Only node kinds — text and unnamed nodes ignored |
+
+`smart` is almost always what you want. Reach for `signature` when you want to match "any function called `foo`" regardless of arguments.
+
+---
+
+## Testing a pattern
+
+Two tools help you confirm a pattern parses the way you expect:
+
+```bash
+# Print the AST of the pattern itself
+sg run -p 'console.log($MSG)' --lang ts --debug-query=ast
+
+# Print the parsed CST of a file (great for figuring out kind names)
+sg run -p '$_' --lang ts --debug-query=cst src/example.ts | head -40
+```
+
+`--debug-query=ast` shows the named AST nodes only (cleaner). `--debug-query=cst` shows everything including punctuation. Both go to stderr, so they don't interfere with stdout JSON.
+
+The web playground is also fast: .
+
+---
+
+## When ast-grep is the wrong tool
+
+If your pattern is fundamentally text-shaped, switch to `grep` / `rg`:
+
+- Match across multiple files **for any text** → `rg`
+- Cross-language regex with alternation → `rg -e foo -e bar`
+- Match comments only → `rg --type ts '^\s*//.*TODO'`
+- Match URLs, emails, license headers → `rg`
+
+ast-grep is for **code structure**: function shapes, call patterns, control flow, type annotations, imports, error handling. If your "pattern" only depends on the bytes of the file and not on the syntax, regex is the right tool.
+
+---
+
+## See also
+
+- `references/pitfalls.md` — concrete regex anti-patterns and language-specific traps.
+- `references/recipes.md` — copy-paste-ready patterns for TS/JS/Py/Go/Rust.
+- `references/yaml-rules.md` — `kind`, `regex`, `inside`, `has`, `all`, `any`, `not`, `matches`.
+- Official:
diff --git a/src/skills/bundled/ast-grep/references/pitfalls.md b/src/skills/bundled/ast-grep/references/pitfalls.md
new file mode 100644
index 0000000..16a6356
--- /dev/null
+++ b/src/skills/bundled/ast-grep/references/pitfalls.md
@@ -0,0 +1,303 @@
+# Pitfalls — what breaks patterns and how to fix them
+
+This is the failure-mode field guide. The `scripts/ast_grep_helper.py validate` subcommand mechanically checks for the items in §1 before calling `sg`; the rest are lower-frequency but still common.
+
+---
+
+## 1. Regex syntax does not work
+
+ast-grep does **not** interpret regex inside patterns. The following all fail:
+
+| Bad | Why | Use instead |
+|---|---|---|
+| `foo\|bar` | `\|` is regex alternation. ast-grep does not alternate. | Two separate calls, OR `any: [pattern: foo, pattern: bar]` in a YAML rule, OR `rg -e foo -e bar`. |
+| `foo.*bar` | `.*` is a regex wildcard. | `foo($$$) bar` if the gap is a list of nodes; otherwise switch to `rg`. |
+| `\w+`, `\d+`, `\s` | Regex character classes. | `$VAR` to capture any identifier. For digits-only, use `kind: number_literal`. |
+| `[a-z]+` | Regex character class. | No AST equivalent — switch to `rg`. |
+| `^foo$` | Regex anchors. | Anchor by AST: use `kind: program > expression_statement` or use `inside`/`not has`. |
+
+**Why this happens**: LLMs default to regex thinking. The mental switch is "ast-grep patterns are *code*, not *strings*."
+
+When you genuinely need regex, use the `regex` rule field in YAML (matches node text with Rust regex):
+
+```yaml
+rule:
+ all:
+ - kind: identifier
+ - regex: '^[A-Z][a-z]+$' # CamelCase identifiers only
+```
+
+Note: `regex` matches the **whole node text** — no partial matches. Combine with `kind` or `pattern` for performance.
+
+---
+
+## 2. Incomplete AST nodes
+
+Patterns must be valid code that the parser accepts as a complete node. Common mistakes:
+
+```text
+# JS/TS
+function foo ❌ no params, no body
+function $NAME($$$) { $$$ } ✅
+
+async function $NAME ❌
+async function $NAME($$$) { $$$ } ✅
+
+# Python
+def foo: ❌ trailing colon makes it a statement
+def $FN($$$) ✅
+class Foo: ❌
+class $C($$$) ✅
+
+# Go
+func foo ❌
+func $NAME($$$) { $$$ } ✅
+
+# Rust
+fn foo ❌
+fn $NAME($$$) -> $RET { $$$ } ✅
+fn $NAME($$$) { $$$ } ✅ (-> () inferred)
+
+# Java
+public void foo ❌
+public void $NAME($$$) { $$$ } ✅
+```
+
+If a pattern returns 0 matches and looks correct, run `sg run -p '' --lang --debug-query=ast --stdin <<< 'echo'` and see what the parser thinks the pattern is. If it returns an `ERROR` node, the pattern is malformed.
+
+---
+
+## 3. Pattern parses as the wrong kind
+
+A class field initializer `a = 123` *also* parses as an assignment expression. If you want only field definitions, you must disambiguate:
+
+```yaml
+# WRONG — pattern parses as assignment_expression, not field_definition
+pattern: a = 123
+kind: field_definition
+
+# CORRECT — use pattern object with context + selector
+pattern:
+ context: 'class C { a = 123 }'
+ selector: field_definition
+```
+
+`kind` and `pattern` are **independent constraints**, not modifiers of each other. ast-grep does not change *how* it parses based on `kind`.
+
+---
+
+## 4. The `|` ambiguity
+
+A bare `|` in a pattern is interpreted as bitwise-or in most languages, **not** alternation. So:
+
+```yaml
+pattern: foo | bar # parses as: foo bitwise-or'd with bar
+```
+
+…matches expressions like `x | y`, not "either foo or bar". To get alternation, use `any`:
+
+```yaml
+rule:
+ any:
+ - pattern: foo
+ - pattern: bar
+```
+
+In TypeScript union types (`A | B`), `|` is part of the type syntax — `pattern: A | B` correctly parses as a union type and matches that.
+
+---
+
+## 5. Same-name metavars collide
+
+```ts
+// Pattern: $X = $X
+// Captures only when both sides are TEXTUALLY identical.
+
+// Matches:
+a = a
+foo.bar = foo.bar
+
+// Does NOT match:
+a = b
+let x = compute() // because $X needs to bind once and re-use
+```
+
+If you actually want two independent captures, name them differently: `$X = $Y`.
+
+---
+
+## 6. `$$$` is greedy then commits
+
+`$$$` does **not** backtrack. It captures as much as possible, then commits. If your pattern needs a non-greedy match, structure it differently:
+
+```ts
+// You want "match foo($X), where $X is any single arg"
+// BAD: foo($$$X) // matches foo(a), foo(a, b), foo(a, b, c) - too broad
+// GOOD: foo($X) // matches only single-arg calls
+
+// You want "match foo() with at least one arg"
+// BAD: foo($$$X) // also matches foo()
+// GOOD: foo($X, $$$REST) // forces at least one arg
+```
+
+---
+
+## 7. `kind` names depend on tree-sitter grammar
+
+`kind: function_declaration` works for JavaScript, but Python uses `function_definition`, Rust uses `function_item`, Go uses `function_declaration` (same as JS by coincidence). To find the right name, parse a known-good file:
+
+```bash
+sg run -p '$_' --lang python --debug-query=cst path/to/example.py | grep -i function
+```
+
+Or open and click on a node to see its `kind`.
+
+---
+
+## 8. `inside` / `has` defaults to `stopBy: neighbor`
+
+```yaml
+inside:
+ kind: function_declaration # only checks the IMMEDIATE parent
+```
+
+If you want "anywhere inside a function (any depth)":
+
+```yaml
+inside:
+ kind: function_declaration
+ stopBy: end # walks up to the file root
+```
+
+Same for `has` (descendants):
+
+```yaml
+has:
+ kind: return_statement
+ stopBy: end # walks down the whole subtree
+```
+
+Without `stopBy: end`, `has` only matches direct children.
+
+---
+
+## 9. CLI silently ignores `--update-all` when `--json` is set
+
+This is the single biggest gotcha when scripting ast-grep. If you run:
+
+```bash
+sg run -p 'foo()' -r 'bar()' --json=compact --update-all .
+```
+
+…you get the JSON output but **no files are mutated**. ast-grep silently drops `--update-all` when `--json` is on. To both preview and apply, run **two passes**:
+
+```bash
+# Pass 1: preview as JSON
+sg run -p 'foo()' -r 'bar()' --json=compact .
+
+# Pass 2: actually apply
+sg run -p 'foo()' -r 'bar()' --update-all .
+```
+
+`scripts/ast_grep_helper.py replace` does this automatically when `--apply` is set.
+
+---
+
+## 10. Composite rules apply to a single node
+
+`all` and `any` evaluate against **one target node** at a time:
+
+```yaml
+# WRONG — wants "node has BOTH a number child AND a string child"
+has:
+ all:
+ - kind: number # impossible: one node cannot be both at once
+ - kind: string
+
+# CORRECT
+all:
+ - has: { kind: number }
+ - has: { kind: string }
+```
+
+Lift relational rules out of composites when the relation is "the surrounding node has X children matching Y."
+
+---
+
+## 11. Field order is not guaranteed
+
+When a rule object has multiple fields:
+
+```yaml
+rule:
+ pattern: $X = compute()
+ has: { kind: number }
+```
+
+…ast-grep evaluates them as an implicit `all`, but the **order** in which metavariables are captured is not guaranteed. If your `transform` or `fix` depends on capture order, use an explicit `all` array:
+
+```yaml
+rule:
+ all:
+ - pattern: function $F() { $$$ }
+ - has: { pattern: $F() } # $F captured by pattern first; here we just check
+```
+
+---
+
+## 12. `regex` without `kind` is slow
+
+`regex` alone scans every node text in the file. On large repos this is noticeably slow. Always combine:
+
+```yaml
+# Slow
+rule:
+ regex: '^TODO'
+
+# Fast
+rule:
+ all:
+ - kind: comment
+ - regex: '^//\s*TODO'
+```
+
+---
+
+## 13. No scope / type / data-flow analysis
+
+ast-grep is a **structural** matcher. It does NOT know:
+
+- Whether two `foo` references point to the same variable.
+- Whether a variable is shadowed.
+- Whether a function is async, throws, returns a Promise.
+- Whether a value flows from input to output.
+
+For those questions, use a real type-aware tool: TypeScript LSP, Pyright, Semgrep with type inference, CodeQL, etc.
+
+ast-grep is great when *the syntactic shape* is what you care about: "find every call to `eval(...)`", "find every `as any`", "find every empty catch block." It is weak for "find every variable that's never used."
+
+---
+
+## 14. Pattern testing is the fastest debugger
+
+When a pattern returns 0 matches and you can't see why:
+
+1. Open .
+2. Paste your code into the left pane, your pattern into the top-right.
+3. The bottom-right shows the parsed AST and which nodes matched (highlighted) or failed.
+
+Or locally:
+
+```bash
+sg run -p '' --lang --debug-query=ast --stdin <<< ''
+```
+
+stderr shows the parsed pattern; stdout shows the JSON match result. If the pattern shows up as `ERROR (XXX)`, it doesn't parse.
+
+---
+
+## See also
+
+- `references/patterns.md` — meta-variables, strictness, naming rules.
+- `references/recipes.md` — known-good patterns by language.
+- `references/cli.md` — `--debug-query`, `--strictness`, `--update-all`.
diff --git a/src/skills/bundled/ast-grep/references/recipes.md b/src/skills/bundled/ast-grep/references/recipes.md
new file mode 100644
index 0000000..b5add75
--- /dev/null
+++ b/src/skills/bundled/ast-grep/references/recipes.md
@@ -0,0 +1,402 @@
+# Recipes — copy-paste patterns by language
+
+Every pattern in this file has been verified against the canonical syntax. They are starting points; tweak metavariable names and constraints to fit your case.
+
+Use them with the helper:
+
+```bash
+ast-grep-helper search '' --lang [path]
+ast-grep-helper replace '' '' --lang [path] # dry-run
+ast-grep-helper replace '' '' --lang [path] --apply
+```
+
+Or directly:
+
+```bash
+sg run -p '' --lang [path]
+sg run -p '' -r '' --update-all --lang [path]
+```
+
+---
+
+## TypeScript / TSX / JavaScript
+
+### Find structural patterns
+
+```typescript
+// Every function declaration
+function $NAME($$$PARAMS) { $$$BODY }
+
+// Every async function
+async function $NAME($$$PARAMS) { $$$BODY }
+
+// Every arrow function (any param shape)
+($$$PARAMS) => $$$BODY
+
+// Every method on a class
+class $C { $$$ $METHOD($$$P) { $$$B } $$$ }
+
+// Every import statement
+import { $$$NAMES } from '$MOD'
+import $DEFAULT from '$MOD'
+import * as $NS from '$MOD'
+
+// Every console.* call
+console.$METHOD($$$ARGS)
+
+// Every JSX element of a given name
+$$$CHILDREN
+
+// Every try/catch
+try { $$$BODY } catch ($E) { $$$HANDLER }
+
+// Every throw
+throw $EXPR
+
+// Every new expression
+new $CLASS($$$ARGS)
+
+// Every type assertion to any (anti-pattern!)
+$EXPR as any
+$EXPR as unknown as $T
+```
+
+### Common rewrites
+
+```bash
+# console.log -> logger.info
+sg run -p 'console.log($$$A)' -r 'logger.info($$$A)' --lang ts --update-all .
+
+# require -> import (one-arg case)
+sg run -p 'const $V = require($M)' -r 'import $V from $M' --lang ts --update-all .
+
+# .then(callback) -> await on the same line (use with caution; needs async function context)
+sg run -p '$P.then($CB)' -r 'const $TMP = await $P; $CB($TMP)' --lang ts --update-all .
+
+# Strip `as any`
+sg run -p '$E as any' -r '$E' --lang ts --update-all .
+
+# Rename a function call site
+sg run -p 'oldName($$$A)' -r 'newName($$$A)' --lang ts --update-all .
+```
+
+---
+
+## Python
+
+> **Reminder**: never end a Python pattern with `:`. Patterns parse as a complete statement, so `def foo($$$):` is invalid.
+
+```python
+# Every function definition
+def $FN($$$PARAMS)
+
+# Every class definition
+class $C($$$BASES)
+
+# Every decorator usage
+@$DEC
+def $FN($$$P)
+
+# Every print call (Python 3)
+print($$$ARGS)
+
+# Every f-string
+f"$STR"
+
+# Every with-statement
+with $CTX as $VAR: $$$BODY
+
+# Every try/except
+try: $$$BODY
+except $EXC: $$$HANDLER
+
+# Every list comprehension
+[$EXPR for $VAR in $ITER]
+
+# Every async def
+async def $FN($$$PARAMS)
+
+# Type hints — Optional[X]
+Optional[$T]
+
+# Type hints — X | None (PEP 604)
+$T | None
+```
+
+### Common rewrites
+
+```bash
+# print(...) -> logger.info(...)
+sg run -p 'print($$$A)' -r 'logger.info($$$A)' --lang py --update-all .
+
+# Optional[X] -> X | None
+sg run -p 'Optional[$T]' -r '$T | None' --lang py --update-all .
+
+# from typing import List -> remove (built-in list works in 3.9+)
+sg run -p 'from typing import List' -r 'from typing import List # TODO: remove, use list' --lang py --update-all .
+```
+
+---
+
+## Go
+
+```go
+// Every function
+func $NAME($$$PARAMS) $$$RET { $$$BODY }
+
+// Every method
+func ($RECV $TYPE) $NAME($$$PARAMS) $$$RET { $$$BODY }
+
+// The classic err nil-check
+if err != nil { $$$BODY }
+
+// Every fmt.Println / fmt.Printf / fmt.Sprintf
+fmt.$METHOD($$$ARGS)
+
+// Every defer
+defer $EXPR
+
+// Every goroutine
+go $EXPR
+
+// Every channel send/recv
+$CH <- $VAL
+$VAL := <-$CH
+
+// Every type assertion
+$EXPR.($TYPE)
+```
+
+### Common rewrites
+
+```bash
+# fmt.Println -> log.Println
+sg run -p 'fmt.Println($$$A)' -r 'log.Println($$$A)' --lang go --update-all .
+
+# Add error wrapping
+sg run -p 'return $ERR' -r 'return fmt.Errorf("operation failed: %w", $ERR)' --lang go --update-all .
+```
+
+---
+
+## Rust
+
+```rust
+// Every fn
+fn $NAME($$$PARAMS) -> $RET { $$$BODY }
+fn $NAME($$$PARAMS) { $$$BODY } // no return type
+
+// Every async fn
+async fn $NAME($$$PARAMS) -> $RET { $$$BODY }
+
+// Every method on impl
+impl $TYPE { fn $METHOD($$$P) -> $R { $$$B } }
+
+// Every trait impl
+impl $TRAIT for $TYPE { $$$ITEMS }
+
+// Every match expression
+match $EXPR { $$$ARMS }
+
+// Every Result-returning fn that uses ?
+fn $N($$$P) -> Result<$T, $E> { $$$ }
+
+// .unwrap() / .expect() (anti-patterns)
+$EXPR.unwrap()
+$EXPR.expect($MSG)
+
+// Every println!/eprintln!/format!
+println!($$$ARGS)
+format!($$$ARGS)
+```
+
+### Common rewrites
+
+```bash
+# unwrap() -> ? in Result-returning fns (caution: needs context)
+sg run -p '$E.unwrap()' -r '$E?' --lang rust --update-all .
+
+# eprintln! -> log::error!
+sg run -p 'eprintln!($$$A)' -r 'log::error!($$$A)' --lang rust --update-all .
+```
+
+---
+
+## Java
+
+```java
+// Every public class
+public class $NAME { $$$BODY }
+
+// Every method (any modifier)
+$$$MOD $RET $NAME($$$P) { $$$BODY }
+
+// Every System.out.println / System.err.println
+System.$STREAM.println($$$ARGS)
+
+// Every try-with-resources
+try ($$$RES) { $$$BODY } catch ($EXC $E) { $$$HANDLER }
+
+// Every annotation usage
+@$ANNOTATION
+$DECL
+```
+
+---
+
+## C / C++
+
+```cpp
+// Every printf-family call
+printf($$$ARGS)
+sprintf($$$ARGS)
+fprintf($$$ARGS)
+
+// Every malloc / free pair (find-only — pairing requires data flow)
+malloc($SIZE)
+free($PTR)
+
+// Every for-loop
+for ($INIT; $COND; $POST) { $$$BODY }
+
+// C++ smart pointer make
+std::make_shared<$T>($$$ARGS)
+std::make_unique<$T>($$$ARGS)
+```
+
+### Rewrites
+
+```bash
+# malloc(N * sizeof(T)) -> calloc(N, sizeof(T)) - safer
+sg run -p 'malloc($N * sizeof($T))' -r 'calloc($N, sizeof($T))' --lang c --update-all .
+```
+
+---
+
+## CSS
+
+```css
+/* Every rule with a specific property */
+{ $$$ color: $VAL; $$$ }
+
+/* Every @media query */
+@media $QUERY { $$$BODY }
+
+/* Every var() reference */
+var($NAME)
+```
+
+---
+
+## HTML
+
+```html
+
+
+
+
+
+
+
+
+```
+
+---
+
+## Bash / Shell
+
+```bash
+# Every for-loop
+for $VAR in $$$LIST; do $$$BODY; done
+
+# Every if-statement
+if $$$COND; then $$$BODY; fi
+
+# Every function definition
+$NAME() { $$$BODY }
+
+# Every subshell call
+$( $$$CMD )
+```
+
+---
+
+## YAML rule recipes (for `sg scan`)
+
+These are full YAML rules you can drop in `rules/*.yml` and run via `sg scan`. See `references/yaml-rules.md` for the full schema.
+
+### no-console (TypeScript)
+
+```yaml
+id: no-console
+language: TypeScript
+severity: warning
+message: "Avoid console.* in production"
+rule:
+ pattern: console.$METHOD($$$ARGS)
+fix: logger.$METHOD($$$ARGS)
+```
+
+### no-as-any (TypeScript)
+
+```yaml
+id: no-as-any
+language: TypeScript
+severity: error
+message: "`as any` defeats type safety. Use a proper type."
+rule:
+ pattern: $EXPR as any
+fix: $EXPR
+```
+
+### empty-catch (JavaScript)
+
+```yaml
+id: empty-catch
+language: JavaScript
+severity: error
+message: "Empty catch swallows errors silently."
+rule:
+ all:
+ - pattern: try { $$$T } catch ($E) { $$$H }
+ - has:
+ kind: catch_clause
+ has:
+ kind: statement_block
+ not:
+ has:
+ kind: statement
+ stopBy: end
+```
+
+### print-to-logger (Python)
+
+```yaml
+id: print-to-logger
+language: Python
+severity: hint
+message: "Use logger.info instead of print"
+rule:
+ pattern: print($$$ARGS)
+fix: logger.info($$$ARGS)
+```
+
+### no-unwrap (Rust)
+
+```yaml
+id: no-unwrap
+language: Rust
+severity: warning
+message: "Avoid .unwrap() in production code; propagate or handle the error."
+rule:
+ pattern: $EXPR.unwrap()
+```
+
+---
+
+## See also
+
+- `references/patterns.md` — meta-variable rules.
+- `references/yaml-rules.md` — full YAML rule schema (atomic / relational / composite / transform / fix).
+- `references/cli.md` — `sg run`, `sg scan`, `sg test`, `sg new`.
+- Official catalog: (community-maintained, browse by language).
diff --git a/src/skills/bundled/ast-grep/references/sgconfig.md b/src/skills/bundled/ast-grep/references/sgconfig.md
new file mode 100644
index 0000000..489716e
--- /dev/null
+++ b/src/skills/bundled/ast-grep/references/sgconfig.md
@@ -0,0 +1,248 @@
+# sgconfig.yml — project configuration
+
+`sgconfig.yml` lives at your project root (the same place as `package.json`, `Cargo.toml`, `pyproject.toml`, etc.) and tells `sg scan`/`sg test` where to find rules and tests.
+
+`sg` walks **upward** from the current directory until it finds an `sgconfig.yml`. You can also pass `--config ` explicitly.
+
+---
+
+## Minimal project layout
+
+```
+my-project/
+├── sgconfig.yml
+├── rules/
+│ ├── no-console.yml
+│ └── no-as-any.yml
+├── utils/
+│ └── is-literal.yml
+├── tests/
+│ ├── no-console.yml
+│ └── __snapshots__/
+│ └── no-console-snapshot.yml
+└── src/
+ └── ...
+```
+
+```yaml
+# sgconfig.yml
+ruleDirs:
+ - rules
+
+testConfigs:
+ - testDir: tests
+ snapshotDir: __snapshots__
+
+utilDirs:
+ - utils
+```
+
+That's it. `sg scan src/` will load every `.yml` in `rules/`, find every `.ts`/`.py`/whatever matching the rule's `language`, and report violations.
+
+---
+
+## Full schema
+
+```yaml
+# Rule directories — required
+ruleDirs:
+ - rules
+ - team-rules
+ - vendor/sg-rules
+
+# Test directories — optional
+testConfigs:
+ - testDir: tests
+ snapshotDir: __snapshots__
+ - testDir: integration-tests
+
+# Utility rule directories — optional
+# Files here become global utilities accessible via `matches: ` from any rule.
+utilDirs:
+ - utils
+ - team-utils
+
+# Override file-extension -> language mapping — optional
+# Useful when your code uses non-standard extensions.
+languageGlobs:
+ html:
+ - '*.vue'
+ - '*.svelte'
+ - '*.astro'
+ json:
+ - '.eslintrc'
+ - '.prettierrc'
+ cpp:
+ - '*.c' # treat C as C++
+ tsx:
+ - '*.ts' # treat all .ts as TSX (so TSX rules work everywhere)
+
+# Custom tree-sitter languages (experimental) — optional
+customLanguages:
+ mojo:
+ libraryPath: tree-sitter-mojo.so
+ extensions: [mojo, '🔥']
+ expandoChar: _ # Replace $ in patterns when language uses $ syntactically
+ languageSymbol: tree_sitter_mojo
+
+# Language injection — embedded code in another language (experimental) — optional
+# Example: CSS inside styled-components template literals.
+languageInjections:
+ - hostLanguage: js
+ rule:
+ pattern: 'styled.$TAG`$CONTENT`'
+ injected: css
+```
+
+---
+
+## Field-by-field
+
+### `ruleDirs` (required)
+
+`Array` — directories containing rule YAML files. Resolved relative to `sgconfig.yml`.
+
+Each `.yml`/`.yaml` file in these directories is loaded as a rule. One file can contain multiple rules separated by `---`.
+
+### `testConfigs`
+
+`Array` where each entry has:
+
+- `testDir` (required): directory of test YAML files.
+- `snapshotDir` (optional, default `__snapshots__`): directory for snapshots.
+
+Each test file looks like:
+
+```yaml
+id: no-console
+valid:
+ - 'logger.info("hi")'
+invalid:
+ - 'console.log("hi")'
+```
+
+`sg test` runs every test, compares matches against the snapshot, and fails on diff. Snapshots are created on first run with `-U`.
+
+### `utilDirs`
+
+`Array` — directories with global utility rules. Each util file must have `id` and `language`. Utils become referenceable via `matches: ` from any rule in the project.
+
+### `languageGlobs`
+
+`HashMap>` — override which extensions map to which language. Takes precedence over the built-in defaults.
+
+Useful for:
+
+- Custom file extensions (`.eslintrc` is JSON).
+- Force-treating `.ts` files as TSX (so JSX-shaped patterns work).
+- Vue/Svelte/Astro files (HTML host language).
+
+### `customLanguages` (experimental)
+
+Register a tree-sitter parser that ast-grep doesn't ship with. Requires:
+
+- `libraryPath`: path to a built `.so` / `.dylib` / `.dll` containing the grammar.
+- `extensions`: file extensions to recognize.
+- `languageSymbol`: the C symbol exported by the grammar (typically `tree_sitter_`).
+- `expandoChar` (optional): character to substitute for `$` in patterns when the host language uses `$` syntactically (PHP, jQuery, etc.).
+
+This is **rarely needed** — ast-grep already supports 25 languages out of the box.
+
+### `languageInjections` (experimental)
+
+Match patterns inside embedded languages. Example: CSS inside JS template literals (styled-components, emotion).
+
+```yaml
+languageInjections:
+ - hostLanguage: js
+ rule:
+ pattern: 'styled.$TAG`$CONTENT`'
+ injected: css
+```
+
+After this, a `css` rule with pattern `color: $C` will match `$CONTENT` strings.
+
+---
+
+## Common configurations
+
+### Monorepo with shared rules
+
+```
+monorepo/
+├── sgconfig.yml # root config — applies to entire monorepo
+├── shared-rules/
+│ ├── no-todo.yml
+│ └── no-as-any.yml
+└── packages/
+ ├── frontend/
+ │ ├── sgconfig.yml # extends root with frontend-specific rules
+ │ └── rules/
+ └── backend/
+ ├── sgconfig.yml # extends root with backend-specific rules
+ └── rules/
+```
+
+Each package's `sgconfig.yml` references both the package-local rules and the shared ones:
+
+```yaml
+# packages/frontend/sgconfig.yml
+ruleDirs:
+ - rules
+ - ../../shared-rules
+```
+
+### Single rule file (no project)
+
+For one-offs, skip `sgconfig.yml` entirely:
+
+```bash
+sg scan -r path/to/single-rule.yml src/
+```
+
+### Inline rule (no file)
+
+```bash
+sg scan --inline-rules '
+id: no-todo
+language: TypeScript
+severity: warning
+rule: { pattern: TODO }' src/
+```
+
+Multiple rules separated by `---`:
+
+```bash
+sg scan --inline-rules '
+id: no-todo
+language: TypeScript
+rule: { pattern: TODO }
+---
+id: no-fixme
+language: TypeScript
+rule: { pattern: FIXME }' src/
+```
+
+---
+
+## Editor integration
+
+VS Code / Neovim / Helix detect `sgconfig.yml` automatically and surface diagnostics from every rule. Without `sgconfig.yml`, the LSP runs without any rules loaded.
+
+To enable schema validation in your editor, add a header to each rule file:
+
+```yaml
+# yaml-language-server: $schema=https://raw.githubusercontent.com/ast-grep/ast-grep/main/schemas/rule.json
+id: no-console
+language: TypeScript
+rule:
+ pattern: console.log($_)
+```
+
+---
+
+## See also
+
+- `references/yaml-rules.md` — rule schema (atomic / relational / composite / transform / fix).
+- `references/cli.md` — `sg scan`, `sg test`, `sg new project`.
+- Official: ,
diff --git a/src/skills/bundled/ast-grep/references/yaml-rules.md b/src/skills/bundled/ast-grep/references/yaml-rules.md
new file mode 100644
index 0000000..7ced6fd
--- /dev/null
+++ b/src/skills/bundled/ast-grep/references/yaml-rules.md
@@ -0,0 +1,509 @@
+# YAML rule reference — atomic, relational, composite, transform, fix
+
+Use this when you outgrow inline `sg run -p ...` patterns and need a reusable, testable rule. A YAML rule is the unit of work for `sg scan`. Drop one or more files in `ruleDirs/` (configured via `sgconfig.yml`) and they get loaded automatically.
+
+This page is the practical reference. The full upstream docs live at:
+
+-
+-
+-
+
+---
+
+## Skeleton
+
+A single YAML file can hold multiple rules separated by `---`.
+
+```yaml
+id: no-console
+language: TypeScript
+severity: warning
+message: "Avoid console.* in production"
+note: |
+ Use a proper logger so we can route logs to stderr in production
+ and silence them in tests.
+url: https://internal.docs/rules/no-console
+
+rule:
+ pattern: console.$METHOD($$$ARGS)
+
+fix: logger.$METHOD($$$ARGS)
+
+constraints:
+ METHOD:
+ not:
+ regex: '^(error|warn)$'
+
+files:
+ - 'src/**/*.ts'
+ignores:
+ - 'src/**/*.test.ts'
+
+metadata:
+ category: logging
+```
+
+---
+
+## Top-level fields
+
+| Field | Required | Description |
+|---|---|---|
+| `id` | yes | Unique identifier. Use `kebab-case`. |
+| `language` | yes | One of: `Bash`, `C`, `Cpp`, `CSharp`, `Css`, `Elixir`, `Go`, `Haskell`, `Html`, `Java`, `JavaScript`, `Json`, `Kotlin`, `Lua`, `Nix`, `Php`, `Python`, `Ruby`, `Rust`, `Scala`, `Solidity`, `Swift`, `TypeScript`, `Tsx`, `Yaml`. **Capitalized PascalCase** is canonical, but lowercase often works. |
+| `rule` | yes | The matching logic. Object containing one or more atomic / relational / composite rules. |
+| `constraints` | no | Filter on captured single-metavariables (`$VAR`, not `$$$`). |
+| `utils` | no | Local utility rules referenced by `matches:` in this file. |
+| `transform` | no | Manipulate metavariable strings before `fix`. |
+| `fix` | no | String or `FixConfig` for auto-rewrite. |
+| `rewriters` | no | Rewriter rules for the `rewrite` transform. |
+| `severity` | no | `hint` \| `info` \| `warning` \| `error` \| `off` (default: `hint`). |
+| `message` | no | Concise lint message. May reference `$VAR` capture text. |
+| `note` | no | Detailed markdown explanation (no `$VAR` interpolation). |
+| `labels` | no | Custom diagnostic highlighting per-metavariable. |
+| `files` | no | Glob include list. |
+| `ignores` | no | Glob exclude list. |
+| `url` | no | Doc link shown in editor diagnostics. |
+| `metadata` | no | Free-form data ignored by `sg`, useful for external tooling. |
+
+---
+
+## Atomic rules — match a single node
+
+### `pattern`
+
+Match by structural pattern. The most common rule.
+
+```yaml
+# String form
+rule:
+ pattern: console.log($MSG)
+
+# Object form (when context is needed)
+rule:
+ pattern:
+ context: 'class C { $FIELD = $INIT }'
+ selector: field_definition
+ strictness: relaxed # optional, default: smart
+```
+
+### `kind`
+
+Match by AST node type name. Tree-sitter grammar-specific.
+
+```yaml
+rule:
+ kind: call_expression
+```
+
+ast-grep 0.39+ supports limited ESQuery selectors:
+
+```yaml
+rule:
+ kind: call_expression > identifier # direct child
+ kind: call_expression + identifier # next sibling
+ kind: call_expression ~ identifier # following sibling
+ kind: call_expression identifier # descendant
+```
+
+To find the right `kind`, parse a known-good file:
+
+```bash
+sg run -p '$_' --lang ts --debug-query=cst src/foo.ts | head -40
+```
+
+### `regex`
+
+Match node text against a Rust regex. Whole-text match (no partial). Always combine with `kind` or `pattern` for performance.
+
+```yaml
+rule:
+ all:
+ - kind: identifier
+ - regex: '^[A-Z][a-z]+$' # PascalCase
+```
+
+Inline flags work: `(?i)apple`, `(?m)^foo`. No look-around, no backreferences.
+
+### `nthChild`
+
+Match by 1-based index among **named** siblings. Inspired by CSS `:nth-child`.
+
+```yaml
+rule:
+ nthChild: 1 # first sibling
+
+# Functional form
+rule:
+ nthChild: 2n+1 # odd siblings
+
+# With reverse and ofRule
+rule:
+ nthChild:
+ position: 1
+ reverse: true # last
+ ofRule:
+ kind: function_declaration
+```
+
+### `range`
+
+Match by character range. Useful for tooling that pinpoints a known location.
+
+```yaml
+rule:
+ range:
+ start: { line: 0, column: 0 }
+ end: { line: 0, column: 11 }
+```
+
+---
+
+## Relational rules — match by relation to other nodes
+
+All four take a sub-rule object plus optional `stopBy` and (for `inside`/`has`) `field`.
+
+### `inside` — target is inside parent/ancestor matching sub-rule
+
+```yaml
+rule:
+ pattern: this.$PROP
+ inside:
+ kind: class_body
+ stopBy: end # walk up to file root, default: neighbor
+```
+
+### `has` — target has child/descendant matching sub-rule
+
+```yaml
+rule:
+ kind: function_declaration
+ has:
+ kind: throw_statement
+ stopBy: end
+```
+
+### `precedes` — target appears before sibling matching sub-rule
+
+```yaml
+rule:
+ kind: import_statement
+ precedes:
+ kind: function_declaration
+```
+
+### `follows` — target appears after sibling matching sub-rule
+
+```yaml
+rule:
+ pattern: super($$$)
+ follows:
+ pattern: $X = $Y
+```
+
+### `stopBy`
+
+| Value | Behavior |
+|---|---|
+| `"neighbor"` (default) | Stop at immediate parent/child/sibling. |
+| `"end"` | Walk all the way to root / leaf / sequence boundary. |
+| Rule object | Stop when sub-rule matches (inclusive). |
+
+### `field`
+
+Specify the semantic role of the target inside its parent (e.g. `name`, `body`, `value`, `key`).
+
+```yaml
+rule:
+ kind: pair
+ has:
+ field: key
+ regex: '^password$'
+```
+
+---
+
+## Composite rules — combine sub-rules
+
+| Rule | Meaning |
+|---|---|
+| `all` | All sub-rules must match the same target node. Metavariables from all sub-rules merge. |
+| `any` | At least one sub-rule must match. Only metavars from the matched branch survive. |
+| `not` | Inverse: target must NOT match the sub-rule. |
+| `matches` | Reference a utility rule by id. |
+
+```yaml
+rule:
+ all:
+ - kind: call_expression
+ - pattern: $FN($$$ARGS)
+ - inside:
+ kind: function_declaration
+ stopBy: end
+
+rule:
+ any:
+ - pattern: console.log($X)
+ - pattern: console.warn($X)
+ - pattern: console.error($X)
+
+rule:
+ all:
+ - pattern: $E.unwrap()
+ - not:
+ inside:
+ kind: function_item
+ has:
+ kind: result_type
+ stopBy: end
+
+rule:
+ matches: is-react-component
+```
+
+> Composites apply to a **single** target. To express "node X has BOTH a number child AND a string child," use two relational rules at the top level, not `all` inside `has`. See `references/pitfalls.md` §10.
+
+---
+
+## Implicit `all` — multiple rule fields
+
+A rule object with multiple fields is treated as an implicit `all`:
+
+```yaml
+# These two are equivalent
+rule:
+ pattern: this.$PROP
+ inside: { kind: class_body }
+
+rule:
+ all:
+ - pattern: this.$PROP
+ - inside: { kind: class_body }
+```
+
+Use the explicit `all` array when capture order matters (rare, but possible with downstream `transform`).
+
+---
+
+## `constraints` — post-match metavariable filtering
+
+After the main `rule` matches, additional checks on captured single metavariables:
+
+```yaml
+rule:
+ pattern: function $NAME($$$P) { $$$B }
+
+constraints:
+ NAME:
+ regex: '^[a-z][a-zA-Z0-9]*$' # camelCase only
+ not:
+ regex: '^_' # not starting with _
+```
+
+Constraints **only apply to single metavars** (`$VAR`), not multi (`$$$VAR`).
+
+---
+
+## `utils` — local reusable sub-rules
+
+```yaml
+utils:
+ is-literal:
+ any:
+ - kind: number
+ - kind: string
+ - kind: 'true'
+ - kind: 'false'
+
+rule:
+ all:
+ - pattern: $X = $Y
+ - has:
+ matches: is-literal # references utils.is-literal
+```
+
+For utils accessible across multiple rule files, use `utilDirs` in `sgconfig.yml` and put each util in its own YAML file with `id` and `language`.
+
+---
+
+## `transform` — manipulate captures before `fix`
+
+Operations: `replace`, `substring`, `convert`, `rewrite`.
+
+### `replace` — regex search/replace on a captured string
+
+```yaml
+rule:
+ pattern: $OLD_FN($$$A)
+constraints:
+ OLD_FN:
+ regex: '^debug_'
+transform:
+ NEW_FN:
+ replace:
+ source: $OLD_FN
+ replace: '^debug_'
+ by: 'release_'
+fix: $NEW_FN($$$A)
+```
+
+### `substring` — character slicing (negative indices supported)
+
+```yaml
+transform:
+ INNER:
+ substring:
+ source: $WRAPPED
+ startChar: 1
+ endChar: -1
+```
+
+### `convert` — case conversion
+
+```yaml
+transform:
+ KEBAB:
+ convert:
+ source: $CAMEL
+ toCase: kebabCase # camelCase | snakeCase | kebabCase | pascalCase | upperCase | lowerCase | capitalize
+ separatedBy: [underscore] # optional: dash | dot | space | slash | underscore | caseChange
+```
+
+### `rewrite` — apply other rewriter rules (experimental)
+
+```yaml
+rewriters:
+ - id: stringify
+ rule: { pattern: "'' + $A" }
+ fix: "String($A)"
+
+rule:
+ pattern: stringify-all($EXPR)
+transform:
+ REWRITTEN:
+ rewrite:
+ source: $EXPR
+ rewriters: [stringify]
+ joinBy: "\n"
+fix: $REWRITTEN
+```
+
+### Transforms can chain
+
+Later transforms can reference variables produced by earlier ones:
+
+```yaml
+transform:
+ KEBABED:
+ convert: { source: $X, toCase: kebabCase }
+ PREFIXED:
+ replace:
+ source: $KEBABED
+ replace: '^'
+ by: 'css-'
+fix: $PREFIXED
+```
+
+---
+
+## `fix` — auto-rewrite
+
+### String form
+
+```yaml
+fix: logger.log($$$ARGS)
+
+# Empty string deletes the match
+fix: ""
+```
+
+### FixConfig form (for list-item deletion that needs to expand the range)
+
+When deleting one item from a comma-separated list, you also need to remove the trailing comma. Use `expandEnd`:
+
+```yaml
+rule:
+ kind: pair
+ has:
+ field: key
+ regex: '^password$'
+
+fix:
+ template: ''
+ expandEnd:
+ regex: ','
+```
+
+`expandStart` and `expandEnd` accept `regex` matching characters that should be absorbed into the rewrite range.
+
+---
+
+## `rewriters` — sub-rule library for `rewrite` transform
+
+Top-level field defining one or more named rewriters:
+
+```yaml
+rewriters:
+ - id: nullable-to-optional
+ rule: { pattern: $X | null }
+ fix: '$X | undefined'
+
+ - id: stringify
+ rule: { pattern: "'' + $A" }
+ fix: 'String($A)'
+```
+
+Used inside `transform` via the `rewrite` operation (see above).
+
+---
+
+## `labels` — custom diagnostic highlighting
+
+```yaml
+rule:
+ pattern: $FN($$$ARGS)
+
+labels:
+ FN:
+ style: primary
+ message: "this function shouldn't be called"
+ ARGS:
+ style: secondary
+ message: "with these arguments"
+```
+
+Editor extensions render the diagnostic with these labels. Defaults are usually fine.
+
+---
+
+## `files` and `ignores` — file selection per-rule
+
+```yaml
+files:
+ - 'src/**/*.ts'
+ - 'lib/**/*.ts'
+
+ignores:
+ - 'src/**/*.test.ts'
+ - '**/__generated__/**'
+```
+
+If omitted, the rule runs on every file matching its `language`. These globs override `sgconfig.yml`-level globs for this rule only.
+
+Object form (rare):
+
+```yaml
+files:
+ - pattern: 'src/**/*.ts'
+ case_sensitive: true
+```
+
+---
+
+## See also
+
+- `references/recipes.md` — copy-paste rules by language.
+- `references/cli.md` — `sg scan`, `sg test`.
+- `references/sgconfig.md` — project-level configuration.
+- Official rule reference:
+- Cheat sheets: ,
diff --git a/src/skills/bundled/ast-grep/scripts/ast_grep_helper.py b/src/skills/bundled/ast-grep/scripts/ast_grep_helper.py
new file mode 100644
index 0000000..053c050
--- /dev/null
+++ b/src/skills/bundled/ast-grep/scripts/ast_grep_helper.py
@@ -0,0 +1,761 @@
+#!/usr/bin/env python3
+"""ast-grep-helper: a thin LLM-friendly wrapper around `sg` (ast-grep).
+
+Single-file Python 3 stdlib. No deps. Works on macOS, Linux, Windows, WSL.
+
+WHAT IT ADDS over plain `sg`:
+ 1. Binary auto-resolution: cached -> @ast-grep/cli -> PATH -> Homebrew -> error with install hint
+ 2. Pattern hint validation: detects regex misuse (\\w, .*, |, [a-z]) and language-specific
+ mistakes (Python trailing colon, JS/Go/Rust missing function body) BEFORE calling sg
+ 3. Two-pass replace: ast-grep silently ignores --update-all when --json is set, so we run
+ a JSON pass to collect matches, then a separate --update-all pass to mutate files
+ 4. Stable JSON output: parses sg --json=compact, salvages truncated output, normalizes shape
+ 5. Cross-OS path handling: works the same on POSIX and Windows (uses pathlib + shutil)
+
+USAGE
+ ast_grep_helper.py search PATTERN [PATH...] [--lang LANG] [--globs GLOB ...] [-C N]
+ ast_grep_helper.py replace PATTERN REWRITE [PATH...] [--lang LANG] [--apply] [--globs GLOB ...]
+ ast_grep_helper.py scan RULE_FILE [PATH...] [--apply] [--report-style STYLE]
+ ast_grep_helper.py test [-c CONFIG] [-t TEST_DIR] [-U]
+ ast_grep_helper.py new {project,rule,test,util} [NAME] [--lang LANG]
+ ast_grep_helper.py langs # list 25 supported languages
+ ast_grep_helper.py doctor # check binary availability + version
+ ast_grep_helper.py install # delegate to ../install.sh / install.ps1
+ ast_grep_helper.py validate PATTERN [--lang LANG] # offline pattern hint check only
+ ast_grep_helper.py --version
+ ast_grep_helper.py --help
+
+EXAMPLES
+ # Find all console.log calls in TypeScript
+ ast_grep_helper.py search 'console.log($MSG)' --lang ts src/
+
+ # Migrate console.log -> logger.info (dry-run preview)
+ ast_grep_helper.py replace 'console.log($MSG)' 'logger.info($MSG)' --lang ts src/
+
+ # Apply the same replacement
+ ast_grep_helper.py replace 'console.log($MSG)' 'logger.info($MSG)' --lang ts src/ --apply
+
+ # Validate a pattern offline (no sg call, no filesystem access)
+ ast_grep_helper.py validate '\\w+' --lang ts
+ # -> exit 2, hint: "regex \\w not supported. Use $VAR for identifiers."
+
+EXIT CODES
+ 0 Success (matches found OR replacement applied OR validation passed)
+ 1 Argument error
+ 2 Pattern hint failure (regex misuse, missing body, etc.) - call would have failed
+ 3 ast-grep binary not found and auto-install declined
+ 4 ast-grep call failed (returned non-zero, with stderr forwarded)
+ 5 Timeout (5 minutes per call by default)
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import os
+import platform
+import re
+import shutil
+import subprocess
+import sys
+from pathlib import Path
+from typing import Optional
+
+VERSION = "0.1.0"
+
+# 25 CLI languages supported by ast-grep, with their aliases (mirrors official docs)
+LANGUAGES: dict[str, list[str]] = {
+ "bash": [".bash", ".sh", ".zsh"],
+ "c": [".c", ".h"],
+ "cpp": [".cc", ".cpp", ".cxx", ".hpp", ".hxx"],
+ "csharp": [".cs"],
+ "css": [".css"],
+ "elixir": [".ex", ".exs"],
+ "go": [".go"],
+ "haskell": [".hs"],
+ "html": [".html", ".htm"],
+ "java": [".java"],
+ "javascript": [".js", ".jsx", ".cjs", ".mjs"],
+ "json": [".json"],
+ "kotlin": [".kt", ".kts"],
+ "lua": [".lua"],
+ "nix": [".nix"],
+ "php": [".php"],
+ "python": [".py", ".pyi"],
+ "ruby": [".rb"],
+ "rust": [".rs"],
+ "scala": [".scala"],
+ "solidity": [".sol"],
+ "swift": [".swift"],
+ "typescript": [".ts", ".cts", ".mts"],
+ "tsx": [".tsx"],
+ "yaml": [".yml", ".yaml"],
+}
+
+# Aliases that ast-grep CLI accepts; we normalize to the canonical name.
+LANG_ALIASES: dict[str, str] = {
+ "js": "javascript", "jsx": "javascript",
+ "ts": "typescript",
+ "py": "python", "py3": "python",
+ "rb": "ruby",
+ "rs": "rust",
+ "kt": "kotlin",
+ "ex": "elixir",
+ "hs": "haskell",
+ "sh": "bash", "zsh": "bash",
+ "cc": "cpp", "c++": "cpp", "cxx": "cpp",
+ "cs": "csharp",
+ "yml": "yaml",
+ "sol": "solidity",
+ "golang": "go",
+}
+
+# Default search timeout (5 min). ast-grep calls can be slow on huge repos.
+DEFAULT_TIMEOUT_S = 300
+
+
+# ---------- logging ----------
+
+def trace(msg: str) -> None:
+ """Print a trace line to stderr (suppressible via --quiet, default off)."""
+ if not _QUIET:
+ print(f"[ast-grep-helper] {msg}", file=sys.stderr, flush=True)
+
+
+def err(msg: str) -> None:
+ """Print an error line to stderr (always shown)."""
+ print(f"[ast-grep-helper] error: {msg}", file=sys.stderr, flush=True)
+
+
+_QUIET = False
+
+
+# ---------- binary resolution ----------
+
+def script_dir() -> Path:
+ return Path(__file__).resolve().parent
+
+
+def skill_root() -> Path:
+ return script_dir().parent
+
+
+def cached_binary() -> Optional[Path]:
+ """Look in /bin/ for a previously downloaded binary."""
+ binname = "sg.exe" if os.name == "nt" else "sg"
+ altname = "ast-grep.exe" if os.name == "nt" else "ast-grep"
+ for name in (binname, altname):
+ p = skill_root() / "bin" / name
+ if p.is_file() and os.access(p, os.X_OK):
+ return p
+ return None
+
+
+def npm_binary() -> Optional[Path]:
+ """If @ast-grep/cli is installed globally via npm, find its binary."""
+ # `sg` shipped by @ast-grep/cli is on PATH when npm prefix bin is on PATH.
+ # We rely on shutil.which for that case.
+ return None # handled by which_binary
+
+
+def which_binary() -> Optional[Path]:
+ """Use shutil.which to find sg or ast-grep on PATH.
+
+ On Linux, plain `sg` collides with the setgroups command from util-linux
+ (sometimes called via /usr/bin/sg) which has flag --version that returns
+ non-zero, so we prefer `ast-grep` when both are on PATH and the `sg` we find
+ is the wrong one.
+ """
+ for name in ("ast-grep", "sg"):
+ found = shutil.which(name)
+ if found:
+ p = Path(found)
+ # On Linux, double-check by trying --version. The util-linux `sg`
+ # rejects --version, while ast-grep prints "ast-grep ".
+ if name == "sg" and platform.system() == "Linux":
+ try:
+ out = subprocess.run(
+ [str(p), "--version"],
+ capture_output=True,
+ text=True,
+ timeout=5,
+ )
+ if out.returncode != 0 or "ast-grep" not in (out.stdout + out.stderr).lower():
+ continue
+ except Exception:
+ continue
+ return p
+ return None
+
+
+def homebrew_binary() -> Optional[Path]:
+ """Common Homebrew install paths."""
+ candidates = [
+ Path("/opt/homebrew/bin/ast-grep"),
+ Path("/opt/homebrew/bin/sg"),
+ Path("/usr/local/bin/ast-grep"),
+ Path("/usr/local/bin/sg"),
+ ]
+ for p in candidates:
+ if p.is_file() and os.access(p, os.X_OK):
+ return p
+ return None
+
+
+# --- OMO runtime resolution (vendored patch) ---
+
+def omo_env_binary() -> Optional[Path]:
+ raw_path = os.environ.get("OMO_AST_GREP_SG_PATH")
+ if not raw_path:
+ return None
+ path = Path(raw_path).expanduser()
+ if path.is_file() and os.access(path, os.X_OK):
+ return path
+ return None
+
+
+def omo_runtime_slug() -> str:
+ if sys.platform.startswith("win"):
+ os_slug = "win32"
+ elif sys.platform == "darwin":
+ os_slug = "darwin"
+ else:
+ os_slug = "linux"
+
+ machine = platform.machine().lower()
+ arch_slug = "arm64" if machine in {"arm64", "aarch64"} else "x64"
+ return f"{os_slug}-{arch_slug}"
+
+
+def omo_runtime_binary() -> Optional[Path]:
+ binary_name = "sg.exe" if sys.platform.startswith("win") else "sg"
+ slug = omo_runtime_slug()
+ candidates: list[Path] = []
+
+ codex_home = os.environ.get("CODEX_HOME")
+ if codex_home:
+ candidates.append(Path(codex_home) / "runtime" / "ast-grep" / slug / binary_name)
+ candidates.append(Path.home() / ".omo" / "runtime" / "ast-grep" / slug / binary_name)
+
+ for path in candidates:
+ if path.is_file() and os.access(path, os.X_OK):
+ return path
+ return None
+
+
+def resolve_binary() -> Optional[Path]:
+ """Resolve the ast-grep binary in priority order.
+
+ 1. OMO_AST_GREP_SG_PATH override
+ 2. OMO runtime dirs
+ 3. Cached binary in /bin/
+ 4. PATH (via shutil.which)
+ 5. Homebrew default paths
+ """
+ for fn in (omo_env_binary, omo_runtime_binary, cached_binary, which_binary, homebrew_binary):
+ result = fn()
+ if result:
+ return result
+ return None
+
+
+def require_binary() -> Path:
+ """Resolve binary, or print an actionable install hint and exit 3."""
+ p = resolve_binary()
+ if p:
+ return p
+ err("ast-grep binary not found.")
+ err("")
+ err("Install via one of:")
+ err(f" bash {skill_root()}/install.sh # POSIX (auto-detects best method)")
+ err(f" pwsh {skill_root()}/install.ps1 # Windows")
+ err("")
+ err("Or manually:")
+ err(" brew install ast-grep # macOS / linuxbrew")
+ err(" npm install -g @ast-grep/cli # any OS with Node")
+ err(" cargo install ast-grep --locked # any OS with Rust")
+ err(" pip install ast-grep-cli # any OS with Python")
+ err(" scoop install main/ast-grep # Windows / Scoop")
+ err("")
+ err("See references/install.md for the full table.")
+ sys.exit(3)
+
+
+# ---------- pattern hint validation ----------
+
+# Regex anti-patterns that ast-grep does NOT support but LLMs frequently emit.
+# Each tuple: (regex_to_detect, hint_message)
+REGEX_ANTIPATTERNS: list[tuple[re.Pattern[str], str]] = [
+ (re.compile(r"\\w|\\d|\\s|\\b"),
+ "Backslash escapes (\\w, \\d, \\s, \\b) are regex syntax, not ast-grep. "
+ "Use $VAR to capture any identifier, or switch to grep for text patterns."),
+ (re.compile(r"(? bool:
+ """Detect a literal '|' that is not inside a string/template literal.
+
+ Heuristic - mark as alternation if `|` appears outside obvious string contexts.
+ """
+ # Strip simple string contents to reduce false positives in patterns like
+ # `'a|b'` or `"x|y"`. This is a heuristic, not a parser.
+ stripped = re.sub(r"'[^']*'|\"[^\"]*\"|`[^`]*`", "", pattern)
+ # Require word chars on both sides to avoid catching bitwise or ||
+ return bool(re.search(r"\w\s*\|\s*\w", stripped)) and "||" not in stripped
+
+
+def lang_specific_hints(pattern: str, lang: Optional[str]) -> list[str]:
+ """Return a list of hints for language-specific common mistakes."""
+ if not lang:
+ return []
+ canonical = LANG_ALIASES.get(lang.lower(), lang.lower())
+ hints: list[str] = []
+
+ if canonical == "python":
+ # def foo($$$): <-- trailing colon breaks the parse
+ if re.search(r"^\s*(def|class)\s+\$?\w+[^:]*:\s*$", pattern, re.MULTILINE):
+ hints.append(
+ "Python pattern has trailing ':'. ast-grep parses pattern as a complete "
+ "definition - drop the trailing colon. Try: 'def $FUNC($$$)' or 'class $C($$$)'."
+ )
+
+ if canonical in ("javascript", "typescript", "tsx"):
+ if re.search(r"^\s*(async\s+)?function\s+\$?\w+\s*$", pattern):
+ hints.append(
+ "JS/TS function pattern is incomplete. Add params and body: "
+ "'function $NAME($$$) { $$$ }'."
+ )
+
+ if canonical == "go":
+ if re.search(r"^\s*func\s+\$?\w+\s*$", pattern):
+ hints.append(
+ "Go function pattern is incomplete. Add params and body: "
+ "'func $NAME($$$) { $$$ }'."
+ )
+
+ if canonical == "rust":
+ if re.search(r"^\s*fn\s+\$?\w+\s*$", pattern):
+ hints.append(
+ "Rust fn pattern is incomplete. Add params, return type, and body: "
+ "'fn $NAME($$$) -> $RET { $$$ }' (or '-> ()' if returning unit)."
+ )
+
+ return hints
+
+
+def validate_pattern(pattern: str, lang: Optional[str]) -> list[str]:
+ """Return a list of hints. Empty list = pattern looks plausible."""
+ hints: list[str] = []
+
+ for rx, msg in REGEX_ANTIPATTERNS:
+ if rx.search(pattern):
+ hints.append(msg)
+
+ if find_alternation(pattern):
+ hints.append(
+ "Literal '|' alternation is regex syntax, not ast-grep. "
+ "Run two separate ast-grep calls (one per alternative), or switch to grep."
+ )
+
+ hints.extend(lang_specific_hints(pattern, lang))
+
+ return hints
+
+
+def normalize_lang(lang: Optional[str]) -> Optional[str]:
+ if not lang:
+ return None
+ canonical = LANG_ALIASES.get(lang.lower(), lang.lower())
+ if canonical not in LANGUAGES:
+ err(f"unknown language '{lang}'. Run 'ast_grep_helper.py langs' for the full list.")
+ sys.exit(1)
+ return canonical
+
+
+# ---------- subprocess helpers ----------
+
+def run_sg(
+ binary: Path,
+ args: list[str],
+ *,
+ timeout: int = DEFAULT_TIMEOUT_S,
+ capture: bool = True,
+) -> subprocess.CompletedProcess[str]:
+ """Spawn `sg ` with a hard timeout. Capture stdout/stderr by default."""
+ cmd = [str(binary), *args]
+ trace(f"exec: {' '.join(cmd)}")
+ try:
+ return subprocess.run(
+ cmd,
+ capture_output=capture,
+ text=True,
+ timeout=timeout,
+ )
+ except subprocess.TimeoutExpired:
+ err(f"ast-grep call timed out after {timeout}s")
+ sys.exit(5)
+
+
+# ---------- subcommands ----------
+
+def cmd_search(args: argparse.Namespace) -> int:
+ pattern: str = args.pattern
+ lang = normalize_lang(args.lang)
+ hints = validate_pattern(pattern, lang)
+ if hints:
+ err("pattern looks invalid for ast-grep:")
+ for h in hints:
+ err(f" - {h}")
+ if not args.force:
+ err("(pass --force to call ast-grep anyway)")
+ return 2
+
+ binary = require_binary()
+ sg_args = ["run", "-p", pattern, "--json=compact"]
+ if lang:
+ sg_args.extend(["--lang", lang])
+ if args.context:
+ sg_args.extend(["-C", str(args.context)])
+ for g in args.globs or []:
+ sg_args.extend(["--globs", g])
+ sg_args.extend(args.paths or ["."])
+
+ proc = run_sg(binary, sg_args)
+ if proc.returncode not in (0, 1): # 0=match, 1=no match - both fine
+ sys.stderr.write(proc.stderr or "")
+ return 4
+
+ matches = parse_compact_json(proc.stdout)
+ if args.json_out:
+ json.dump(matches, sys.stdout, indent=2)
+ print()
+ else:
+ format_matches(matches)
+
+ if not matches:
+ # Re-run pattern hints in case empty result was caused by something subtle.
+ # Already done above; here we just give a generic suggestion.
+ trace("no matches. If you expected matches, double-check --lang and the pattern shape.")
+ return 0
+
+
+def cmd_replace(args: argparse.Namespace) -> int:
+ pattern: str = args.pattern
+ rewrite: str = args.rewrite
+ lang = normalize_lang(args.lang)
+
+ pattern_hints = validate_pattern(pattern, lang)
+ rewrite_hints = validate_pattern(rewrite, lang)
+ all_hints = []
+ if pattern_hints:
+ all_hints.append("pattern issues:")
+ all_hints.extend(f" - {h}" for h in pattern_hints)
+ if rewrite_hints:
+ all_hints.append("rewrite issues:")
+ all_hints.extend(f" - {h}" for h in rewrite_hints)
+ if all_hints:
+ err("input looks invalid for ast-grep:")
+ for line in all_hints:
+ err(line)
+ if not args.force:
+ err("(pass --force to call ast-grep anyway)")
+ return 2
+
+ binary = require_binary()
+
+ # Pass 1: dry-run via JSON to collect what would change.
+ sg_args1 = ["run", "-p", pattern, "-r", rewrite, "--json=compact"]
+ if lang:
+ sg_args1.extend(["--lang", lang])
+ for g in args.globs or []:
+ sg_args1.extend(["--globs", g])
+ sg_args1.extend(args.paths or ["."])
+
+ proc1 = run_sg(binary, sg_args1)
+ if proc1.returncode not in (0, 1):
+ sys.stderr.write(proc1.stderr or "")
+ return 4
+
+ matches = parse_compact_json(proc1.stdout)
+ if not matches:
+ trace("no matches; nothing to replace.")
+ return 0
+
+ if not args.apply:
+ # Show the dry-run preview and exit.
+ print(f"DRY-RUN: would rewrite {len(matches)} match(es) across "
+ f"{len({m['file'] for m in matches})} file(s):")
+ format_matches(matches, show_replacement=True)
+ print()
+ print("Re-run with --apply to mutate files.")
+ return 0
+
+ # Pass 2: apply with --update-all (no --json; sg silently ignores --update-all
+ # when --json is present, so we MUST run a second invocation).
+ sg_args2 = ["run", "-p", pattern, "-r", rewrite, "--update-all"]
+ if lang:
+ sg_args2.extend(["--lang", lang])
+ for g in args.globs or []:
+ sg_args2.extend(["--globs", g])
+ sg_args2.extend(args.paths or ["."])
+
+ proc2 = run_sg(binary, sg_args2)
+ if proc2.returncode not in (0, 1):
+ sys.stderr.write(proc2.stderr or "")
+ return 4
+
+ print(f"APPLIED: rewrote {len(matches)} match(es) across "
+ f"{len({m['file'] for m in matches})} file(s).")
+ return 0
+
+
+def cmd_scan(args: argparse.Namespace) -> int:
+ binary = require_binary()
+ sg_args = ["scan"]
+ if args.config:
+ sg_args.extend(["-c", args.config])
+ if args.rule:
+ sg_args.extend(["-r", args.rule])
+ if args.inline_rules:
+ sg_args.extend(["--inline-rules", args.inline_rules])
+ if args.report_style:
+ sg_args.extend(["--report-style", args.report_style])
+ if args.apply:
+ sg_args.append("-U")
+ sg_args.extend(args.paths or [])
+
+ proc = run_sg(binary, sg_args, capture=False)
+ return proc.returncode
+
+
+def cmd_test(args: argparse.Namespace) -> int:
+ binary = require_binary()
+ sg_args = ["test"]
+ if args.config:
+ sg_args.extend(["-c", args.config])
+ if args.test_dir:
+ sg_args.extend(["-t", args.test_dir])
+ if args.update:
+ sg_args.append("-U")
+ proc = run_sg(binary, sg_args, capture=False)
+ return proc.returncode
+
+
+def cmd_new(args: argparse.Namespace) -> int:
+ binary = require_binary()
+ sg_args = ["new", args.what]
+ if args.name:
+ sg_args.append(args.name)
+ if args.lang:
+ sg_args.extend(["--lang", args.lang])
+ if args.yes:
+ sg_args.append("--yes")
+ proc = run_sg(binary, sg_args, capture=False)
+ return proc.returncode
+
+
+def cmd_langs(_args: argparse.Namespace) -> int:
+ print("ast-grep supported languages (25):")
+ for lang, exts in sorted(LANGUAGES.items()):
+ print(f" {lang:<12} {' '.join(exts)}")
+ print()
+ print("Aliases accepted by --lang:")
+ for alias, canonical in sorted(LANG_ALIASES.items()):
+ print(f" {alias:<8} -> {canonical}")
+ return 0
+
+
+def cmd_doctor(_args: argparse.Namespace) -> int:
+ print(f"ast-grep-helper v{VERSION}")
+ print(f"Python: {sys.version.split()[0]}")
+ print(f"Platform: {platform.system()} {platform.release()} ({platform.machine()})")
+ print(f"Skill: {skill_root()}")
+ print()
+ binary = resolve_binary()
+ if not binary:
+ print("ast-grep binary: NOT FOUND")
+ print(" -> run: bash install.sh (POSIX) or pwsh install.ps1 (Windows)")
+ return 1
+ print(f"ast-grep binary: {binary}")
+ proc = run_sg(binary, ["--version"], timeout=5)
+ if proc.returncode == 0:
+ print(f" version: {proc.stdout.strip()}")
+ else:
+ print(f" --version returned exit {proc.returncode}")
+ print(f" stderr: {proc.stderr.strip()}")
+ return 1
+ return 0
+
+
+def cmd_install(_args: argparse.Namespace) -> int:
+ """Delegate to install.sh / install.ps1 in the skill root."""
+ if os.name == "nt":
+ installer = skill_root() / "install.ps1"
+ cmd = ["pwsh", "-File", str(installer)]
+ else:
+ installer = skill_root() / "install.sh"
+ cmd = ["bash", str(installer)]
+ if not installer.is_file():
+ err(f"installer not found: {installer}")
+ return 1
+ trace(f"running installer: {' '.join(cmd)}")
+ return subprocess.run(cmd).returncode
+
+
+def cmd_validate(args: argparse.Namespace) -> int:
+ """Offline pattern validation. No sg call. Useful for CI / quick checks."""
+ lang = normalize_lang(args.lang) if args.lang else None
+ hints = validate_pattern(args.pattern, lang)
+ if hints:
+ for h in hints:
+ print(f"hint: {h}")
+ return 2
+ print("pattern looks plausible for ast-grep.")
+ return 0
+
+
+# ---------- output formatting ----------
+
+def parse_compact_json(text: str) -> list[dict]:
+ """Parse `sg --json=compact` output. Salvages partial output when truncated."""
+ if not text.strip():
+ return []
+ try:
+ data = json.loads(text)
+ if isinstance(data, list):
+ return data
+ return []
+ except json.JSONDecodeError:
+ # Try line-by-line salvage for truncated output.
+ results = []
+ for line in text.splitlines():
+ line = line.strip().rstrip(",")
+ if not line.startswith("{"):
+ continue
+ try:
+ obj = json.loads(line)
+ if isinstance(obj, dict):
+ results.append(obj)
+ except json.JSONDecodeError:
+ continue
+ return results
+
+
+def format_matches(matches: list[dict], *, show_replacement: bool = False) -> None:
+ if not matches:
+ print("(no matches)")
+ return
+ by_file: dict[str, list[dict]] = {}
+ for m in matches:
+ by_file.setdefault(m.get("file", "?"), []).append(m)
+ for path, items in sorted(by_file.items()):
+ print(f"{path} ({len(items)} match{'es' if len(items) != 1 else ''})")
+ for m in items:
+ r = m.get("range", {})
+ start = r.get("start", {})
+ line = start.get("line", "?")
+ col = start.get("column", "?")
+ text = (m.get("text") or "").splitlines()
+ preview = text[0] if text else ""
+ print(f" {path}:{line}:{col} {preview}")
+ if show_replacement and "replacement" in m:
+ rep = (m.get("replacement") or "").splitlines()
+ rep_preview = rep[0] if rep else ""
+ print(f" -> {rep_preview}")
+
+
+# ---------- argparse ----------
+
+def build_parser() -> argparse.ArgumentParser:
+ p = argparse.ArgumentParser(
+ prog="ast-grep-helper",
+ description="LLM-friendly wrapper around ast-grep (sg).",
+ formatter_class=argparse.RawDescriptionHelpFormatter,
+ )
+ p.add_argument("--version", action="version", version=f"ast-grep-helper {VERSION}")
+ p.add_argument("--quiet", "-q", action="store_true", help="Suppress trace lines on stderr.")
+ sub = p.add_subparsers(dest="cmd", required=True, metavar="COMMAND")
+
+ s = sub.add_parser("search", help="Search code by AST pattern.")
+ s.add_argument("pattern", help="AST pattern, e.g. 'console.log($MSG)'")
+ s.add_argument("paths", nargs="*", help="Paths to search (default: '.')")
+ s.add_argument("--lang", "-l", help="Language (e.g. ts, py, go, rust). See: langs subcommand.")
+ s.add_argument("--globs", action="append", help="Include/exclude glob (repeat; prefix '!' to exclude).")
+ s.add_argument("--context", "-C", type=int, help="Lines of context around each match.")
+ s.add_argument("--json-out", action="store_true", help="Emit raw JSON instead of human format.")
+ s.add_argument("--force", action="store_true", help="Skip pattern hint validation.")
+ s.set_defaults(func=cmd_search)
+
+ r = sub.add_parser("replace", help="Rewrite code by AST pattern (dry-run by default).")
+ r.add_argument("pattern", help="AST pattern.")
+ r.add_argument("rewrite", help="Replacement pattern (can reuse $VAR from pattern).")
+ r.add_argument("paths", nargs="*", help="Paths (default: '.')")
+ r.add_argument("--lang", "-l", help="Language.")
+ r.add_argument("--globs", action="append", help="Include/exclude glob.")
+ r.add_argument("--apply", action="store_true", help="Mutate files (default: dry-run preview).")
+ r.add_argument("--force", action="store_true", help="Skip pattern hint validation.")
+ r.set_defaults(func=cmd_replace)
+
+ sc = sub.add_parser("scan", help="Run YAML-rule-based scan.")
+ sc.add_argument("paths", nargs="*", help="Paths to scan.")
+ sc.add_argument("--config", "-c", help="Path to sgconfig.yml.")
+ sc.add_argument("--rule", "-r", help="Single rule file.")
+ sc.add_argument("--inline-rules", help="Inline YAML rule string.")
+ sc.add_argument("--report-style", choices=["rich", "medium", "short"], help="Report style.")
+ sc.add_argument("--apply", "-U", action="store_true", help="Apply fixes (default: report only).")
+ sc.set_defaults(func=cmd_scan)
+
+ t = sub.add_parser("test", help="Run ast-grep snapshot tests.")
+ t.add_argument("--config", "-c", help="Path to sgconfig.yml.")
+ t.add_argument("--test-dir", "-t", help="Test directory.")
+ t.add_argument("--update", "-U", action="store_true", help="Update snapshots.")
+ t.set_defaults(func=cmd_test)
+
+ n = sub.add_parser("new", help="Scaffold a new project / rule / test / util.")
+ n.add_argument("what", choices=["project", "rule", "test", "util"], help="What to create.")
+ n.add_argument("name", nargs="?", help="Name of the artifact.")
+ n.add_argument("--lang", "-l", help="Language.")
+ n.add_argument("--yes", "-y", action="store_true", help="Accept defaults.")
+ n.set_defaults(func=cmd_new)
+
+ sub.add_parser("langs", help="List supported languages.").set_defaults(func=cmd_langs)
+ sub.add_parser("doctor", help="Check ast-grep binary availability.").set_defaults(func=cmd_doctor)
+ sub.add_parser("install", help="Run the install script for this OS.").set_defaults(func=cmd_install)
+
+ v = sub.add_parser("validate", help="Validate a pattern offline (pattern hint check only).")
+ v.add_argument("pattern", help="AST pattern.")
+ v.add_argument("--lang", "-l", help="Language for language-specific hints.")
+ v.set_defaults(func=cmd_validate)
+
+ return p
+
+
+def main(argv: Optional[list[str]] = None) -> int:
+ global _QUIET
+ parser = build_parser()
+ # Accept `search PATTERN --lang js .` — plain parse_args greedily
+ # finalizes the nargs='*' paths list before the optional, then errors
+ # "unrecognized arguments: ." on the trailing path.
+ # parse_intermixed_args cannot be used with subparsers, so collect the
+ # leftover non-flag tokens and fold them into `paths` ourselves.
+ args, extras = parser.parse_known_args(argv)
+ bad = [tok for tok in extras if tok.startswith("-")]
+ if bad:
+ parser.error(f"unrecognized arguments: {' '.join(bad)}")
+ if extras:
+ if hasattr(args, "paths"):
+ args.paths = list(getattr(args, "paths") or []) + extras
+ else:
+ parser.error(f"unrecognized arguments: {' '.join(extras)}")
+ _QUIET = bool(getattr(args, "quiet", False))
+ return args.func(args)
+
+
+if __name__ == "__main__":
+ sys.exit(main())
diff --git a/src/skills/bundled/ast-grep/tests/smoke.ps1 b/src/skills/bundled/ast-grep/tests/smoke.ps1
new file mode 100644
index 0000000..b051d1c
--- /dev/null
+++ b/src/skills/bundled/ast-grep/tests/smoke.ps1
@@ -0,0 +1,123 @@
+#Requires -Version 5.1
+# Smoke test for the ast-grep skill on Windows (PowerShell 5.1+).
+$ErrorActionPreference = 'Stop'
+
+$ScriptDir = Split-Path -Parent $MyInvocation.MyCommand.Path
+$SkillDir = Split-Path -Parent $ScriptDir
+$Helper = Join-Path $SkillDir 'scripts/ast_grep_helper.py'
+$Python = if (Get-Command py -ErrorAction SilentlyContinue) { 'py' } else { 'python' }
+
+$Output = Join-Path $env:TEMP ("ast-grep-skill-smoke-" + [guid]::NewGuid().ToString('N').Substring(0,8))
+New-Item -ItemType Directory -Path $Output -Force | Out-Null
+
+function Pass([string]$msg) { Write-Host "PASS: $msg" }
+function Fail([string]$msg) { Write-Host "FAIL: $msg" -ForegroundColor Red; Remove-Item -Recurse -Force $Output -ErrorAction SilentlyContinue; exit 1 }
+
+function Run([string[]]$Args) {
+ $stdoutFile = Join-Path $Output ("out-" + [guid]::NewGuid().ToString('N').Substring(0,8) + ".txt")
+ $proc = Start-Process -FilePath $Python -ArgumentList (@($Helper) + $Args) -NoNewWindow -PassThru -Wait -RedirectStandardOutput $stdoutFile -RedirectStandardError "$stdoutFile.err"
+ $stdout = if (Test-Path $stdoutFile) { Get-Content $stdoutFile -Raw } else { '' }
+ $stderr = if (Test-Path "$stdoutFile.err") { Get-Content "$stdoutFile.err" -Raw } else { '' }
+ return [pscustomobject]@{
+ ExitCode = $proc.ExitCode
+ Stdout = $stdout
+ Stderr = $stderr
+ Combined = "$stdout`n$stderr"
+ }
+}
+
+try {
+ # 1. --version
+ $r = Run @('--version')
+ if ($r.Combined -notmatch 'ast-grep-helper') { Fail '--version output missing' }
+ Pass '--version'
+
+ # 2. langs (must list >=25)
+ $r = Run @('langs')
+ $langCount = ($r.Stdout -split "`n" | Where-Object { $_ -match '^ [a-z]' }).Count
+ if ($langCount -lt 25) { Fail "langs listed only $langCount (expected >=25)" }
+ Pass 'langs lists at least 25 languages'
+
+ # 3. regex misuse: \w+
+ $r = Run @('validate', '\w+', '--lang', 'ts')
+ if ($r.ExitCode -ne 2) { Fail "validate '\w+' should exit 2, got $($r.ExitCode)" }
+ if ($r.Combined -notmatch 'regex') { Fail "validate '\w+' should mention regex" }
+ Pass 'validate detects \w regex misuse'
+
+ # 4. valid pattern
+ $r = Run @('validate', 'console.log($MSG)', '--lang', 'ts')
+ if ($r.ExitCode -ne 0) { Fail "validate 'console.log(`$MSG)' should exit 0, got $($r.ExitCode)" }
+ Pass 'validate accepts plausible pattern'
+
+ # 5. Python trailing colon
+ $r = Run @('validate', 'def $F($$$):', '--lang', 'py')
+ if ($r.ExitCode -ne 2) { Fail "validate 'def `$F(`$`$`$):' should exit 2, got $($r.ExitCode)" }
+ if ($r.Combined -notmatch 'colon|trailing') { Fail 'validate should mention trailing colon' }
+ Pass 'validate detects Python trailing colon'
+
+ # 6. Incomplete TS function
+ $r = Run @('validate', 'function $N', '--lang', 'ts')
+ if ($r.ExitCode -ne 2) { Fail "validate 'function `$N' should exit 2, got $($r.ExitCode)" }
+ if ($r.Combined -notmatch 'incomplete|params|body') { Fail 'validate should hint about params/body' }
+ Pass 'validate detects incomplete TS function'
+
+ # 7. Alternation pipe
+ $r = Run @('validate', 'foo|bar', '--lang', 'ts')
+ if ($r.ExitCode -ne 2) { Fail "validate 'foo|bar' should exit 2, got $($r.ExitCode)" }
+ if ($r.Combined -notmatch 'alternation|regex') { Fail 'validate should mention alternation' }
+ Pass 'validate detects literal | alternation'
+
+ # 8. doctor
+ $r = Run @('doctor')
+ if ($r.Combined -notmatch 'ast-grep-helper') { Fail 'doctor missing helper version line' }
+ Pass 'doctor produces output'
+
+ # 9. search w/o binary
+ $r = Run @('-q', 'search', 'foo()', '--lang', 'ts', 'C:/nonexistent-path-xyzzy')
+ switch ($r.ExitCode) {
+ { $_ -in 0,1,4 } { Pass "search runs (rc=$($r.ExitCode), ast-grep available)" }
+ 3 {
+ if ($r.Combined -notmatch 'install') { Fail 'search rc=3 should print install hint' }
+ Pass 'search without binary prints install hint'
+ }
+ default { Fail "search returned unexpected rc=$($r.ExitCode): $($r.Combined)" }
+ }
+
+ # 10. install.ps1 syntax (parse-check via PowerShell tokenizer)
+ $tokens = $null
+ $errors = $null
+ [System.Management.Automation.Language.Parser]::ParseFile(
+ (Join-Path $SkillDir 'install.ps1'), [ref]$tokens, [ref]$errors) | Out-Null
+ if ($errors -and $errors.Count -gt 0) { Fail "install.ps1 has parse errors: $($errors -join '; ')" }
+ Pass 'install.ps1 parses cleanly'
+
+ # 11. SKILL.md frontmatter
+ $skill = Get-Content (Join-Path $SkillDir 'SKILL.md') -Raw
+ if (-not $skill.StartsWith("---`n") -and -not $skill.StartsWith("---`r`n")) {
+ Fail 'SKILL.md must start with YAML frontmatter'
+ }
+ $endIdx = $skill.IndexOf("`n---`n", 4)
+ if ($endIdx -lt 0) { $endIdx = $skill.IndexOf("`r`n---`r`n", 4) }
+ if ($endIdx -lt 0) { Fail 'SKILL.md missing closing ---' }
+ $fm = $skill.Substring(4, $endIdx - 4)
+ if ($fm -notmatch '(?m)^name:\s*ast-grep\s*$') { Fail 'frontmatter missing name: ast-grep' }
+ if ($fm -notmatch '(?m)^description:') { Fail 'frontmatter missing description' }
+ Pass 'SKILL.md frontmatter shape'
+
+ # 12. All required reference files exist
+ $required = @(
+ 'references/install.md', 'references/patterns.md', 'references/pitfalls.md',
+ 'references/recipes.md', 'references/cli.md', 'references/yaml-rules.md',
+ 'references/sgconfig.md'
+ )
+ foreach ($f in $required) {
+ if (-not (Test-Path (Join-Path $SkillDir $f))) { Fail "missing $f" }
+ }
+ Pass 'all references present'
+
+ Write-Host ''
+ Write-Host 'all smoke tests passed'
+}
+finally {
+ Remove-Item -Recurse -Force $Output -ErrorAction SilentlyContinue
+}
diff --git a/src/skills/bundled/ast-grep/tests/smoke.sh b/src/skills/bundled/ast-grep/tests/smoke.sh
new file mode 100644
index 0000000..0ab4b0e
--- /dev/null
+++ b/src/skills/bundled/ast-grep/tests/smoke.sh
@@ -0,0 +1,212 @@
+#!/usr/bin/env bash
+# Smoke test for the ast-grep skill on POSIX (macOS / Linux / WSL / Git Bash).
+#
+# Tests:
+# 1. helper --version
+# 2. helper langs (lists 25 languages)
+# 3. helper validate '\w+' --lang ts (must exit 2 with hint)
+# 4. helper validate 'console.log($MSG)' (must exit 0 plausible)
+# 5. helper validate 'def $F($$$):' --lang py (must exit 2 - trailing colon)
+# 6. helper validate 'function $N' --lang ts (must exit 2 - incomplete)
+# 7. helper validate 'foo|bar' --lang ts (must exit 2 - alternation)
+# 8. helper doctor (informational; tolerates no-binary)
+# 9. helper search w/o binary => exit 3 with install hint
+# 10. install.sh --help (parses)
+# 11. SKILL.md frontmatter shape (canonical)
+#
+# Runs against the helper using ONLY stdlib python3 - no ast-grep needed.
+
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
+SKILL_DIR="$(cd "$SCRIPT_DIR/.." && pwd)"
+HELPER="python3 $SKILL_DIR/scripts/ast_grep_helper.py"
+
+OUTPUT_DIR="$(mktemp -d -t ast-grep-skill-smoke-XXXXXX)"
+trap 'rm -rf "$OUTPUT_DIR"' EXIT
+
+fail() { echo "FAIL: $*" >&2; exit 1; }
+pass() { echo "PASS: $*"; }
+
+omo_runtime_slug() {
+ case "$(uname -s)" in
+ Darwin) local os_slug="darwin" ;;
+ MINGW*|MSYS*|CYGWIN*) local os_slug="win32" ;;
+ *) local os_slug="linux" ;;
+ esac
+
+ case "$(uname -m | tr '[:upper:]' '[:lower:]')" in
+ arm64|aarch64) local arch_slug="arm64" ;;
+ *) local arch_slug="x64" ;;
+ esac
+
+ printf '%s-%s' "$os_slug" "$arch_slug"
+}
+
+fake_sg() {
+ local target="$1"
+ mkdir -p "$(dirname "$target")"
+ cat > "$target" <<'SH'
+#!/usr/bin/env bash
+if [ "${1:-}" = "--version" ]; then
+ printf 'ast-grep 0.45.0 fake\n'
+else
+ printf 'fake ast-grep\n'
+fi
+SH
+ chmod +x "$target"
+}
+
+# 1. --version
+$HELPER --version | grep -q "ast-grep-helper" || fail "--version output missing"
+pass "--version"
+
+# 2. langs (must list 25)
+LANG_LINES=$($HELPER langs | grep -E '^ [a-z]' | wc -l | tr -d ' ')
+[ "$LANG_LINES" -ge 25 ] || fail "langs listed only $LANG_LINES (expected >=25)"
+pass "langs lists at least 25 languages"
+
+# 3. regex misuse: \w+
+set +e
+$HELPER validate '\w+' --lang ts > "$OUTPUT_DIR/v1.out" 2>&1
+RC=$?
+set -e
+[ $RC -eq 2 ] || fail "validate '\\w+' should exit 2, got $RC (output: $(cat $OUTPUT_DIR/v1.out))"
+grep -qi 'regex' "$OUTPUT_DIR/v1.out" || fail "validate '\\w+' should mention regex (output: $(cat $OUTPUT_DIR/v1.out))"
+pass "validate detects \\w regex misuse"
+
+# 4. valid pattern
+set +e
+$HELPER validate 'console.log($MSG)' --lang ts > "$OUTPUT_DIR/v2.out" 2>&1
+RC=$?
+set -e
+[ $RC -eq 0 ] || fail "validate 'console.log(\$MSG)' should exit 0, got $RC (output: $(cat $OUTPUT_DIR/v2.out))"
+pass "validate accepts plausible pattern"
+
+# 5. Python trailing colon
+set +e
+$HELPER validate 'def $F($$$):' --lang py > "$OUTPUT_DIR/v3.out" 2>&1
+RC=$?
+set -e
+[ $RC -eq 2 ] || fail "validate 'def \$F(\$\$\$):' should exit 2, got $RC"
+grep -qi 'colon\|trailing' "$OUTPUT_DIR/v3.out" || fail "validate should mention trailing colon"
+pass "validate detects Python trailing colon"
+
+# 6. Incomplete TS function
+set +e
+$HELPER validate 'function $N' --lang ts > "$OUTPUT_DIR/v4.out" 2>&1
+RC=$?
+set -e
+[ $RC -eq 2 ] || fail "validate 'function \$N' should exit 2, got $RC"
+grep -qi 'incomplete\|params\|body' "$OUTPUT_DIR/v4.out" || fail "validate should hint about params/body"
+pass "validate detects incomplete TS function"
+
+# 7. Alternation pipe
+set +e
+$HELPER validate 'foo|bar' --lang ts > "$OUTPUT_DIR/v5.out" 2>&1
+RC=$?
+set -e
+[ $RC -eq 2 ] || fail "validate 'foo|bar' should exit 2, got $RC"
+grep -qi 'alternation\|regex' "$OUTPUT_DIR/v5.out" || fail "validate should mention alternation"
+pass "validate detects literal | alternation"
+
+# 8. doctor (informational)
+$HELPER doctor > "$OUTPUT_DIR/doc.out" 2>&1 || true
+grep -q "ast-grep-helper" "$OUTPUT_DIR/doc.out" || fail "doctor missing helper version line"
+pass "doctor produces output"
+
+# Given OMO_AST_GREP_SG_PATH points at a fake executable.
+# When doctor resolves ast-grep, then it reports that exact path.
+FAKE_ENV_SG="$OUTPUT_DIR/fake-env/sg"
+fake_sg "$FAKE_ENV_SG"
+OMO_AST_GREP_SG_PATH="$FAKE_ENV_SG" $HELPER doctor > "$OUTPUT_DIR/omo-env.out" 2>&1
+grep -Fq "ast-grep binary: $FAKE_ENV_SG" "$OUTPUT_DIR/omo-env.out" || fail "OMO_AST_GREP_SG_PATH was not preferred: $(cat "$OUTPUT_DIR/omo-env.out")"
+pass "OMO_AST_GREP_SG_PATH resolves first"
+
+# Given HOME has an OMO runtime sg executable.
+# When doctor resolves ast-grep, then it reports the HOME runtime path.
+RUNTIME_HOME="$OUTPUT_DIR/home"
+RUNTIME_SLUG="$(omo_runtime_slug)"
+RUNTIME_BIN="sg"
+case "$RUNTIME_SLUG" in
+ win32-*) RUNTIME_BIN="sg.exe" ;;
+esac
+FAKE_RUNTIME_SG="$RUNTIME_HOME/.omo/runtime/ast-grep/$RUNTIME_SLUG/$RUNTIME_BIN"
+fake_sg "$FAKE_RUNTIME_SG"
+HOME="$RUNTIME_HOME" CODEX_HOME= OMO_AST_GREP_SG_PATH= $HELPER doctor > "$OUTPUT_DIR/omo-runtime.out" 2>&1
+grep -Fq "ast-grep binary: $FAKE_RUNTIME_SG" "$OUTPUT_DIR/omo-runtime.out" || fail "OMO HOME runtime was not resolved: $(cat "$OUTPUT_DIR/omo-runtime.out")"
+pass "OMO HOME runtime resolves before standalone fallback"
+
+# 9. search w/o binary => either runs successfully (binary found) OR exits 3 with hint
+# The CI may or may not have ast-grep installed; both are valid.
+set +e
+$HELPER -q search 'foo()' --lang ts /nonexistent-path-xyzzy > "$OUTPUT_DIR/s1.out" 2>&1
+RC=$?
+set -e
+case "$RC" in
+ 0|1|4)
+ # Binary present but no matches OR sg returned non-fatal error - both fine
+ pass "search runs (rc=$RC, ast-grep available)"
+ ;;
+ 3)
+ grep -qi 'install' "$OUTPUT_DIR/s1.out" || fail "search rc=3 should print install hint"
+ pass "search without binary prints install hint"
+ ;;
+ *)
+ fail "search returned unexpected rc=$RC: $(cat $OUTPUT_DIR/s1.out)"
+ ;;
+esac
+
+# 10. install.sh syntax check + --help
+bash -n "$SKILL_DIR/install.sh" || fail "install.sh has syntax errors"
+$SKILL_DIR/install.sh --help > "$OUTPUT_DIR/inst.out" 2>&1 || true
+grep -qi 'install' "$OUTPUT_DIR/inst.out" || fail "install.sh --help missing keyword 'install'"
+pass "install.sh syntax + --help"
+
+# 11. SKILL.md frontmatter
+python3 - < 0, "SKILL.md missing closing ---"
+fm = src[4:end]
+assert re.search(r"^name:\s*ast-grep\s*$", fm, re.M), "frontmatter missing name: ast-grep"
+assert re.search(r"^description:", fm, re.M), "frontmatter missing description"
+PY
+pass "SKILL.md frontmatter shape"
+
+# 12. All required reference files exist
+for f in references/install.md references/patterns.md references/pitfalls.md \
+ references/recipes.md references/cli.md references/yaml-rules.md \
+ references/sgconfig.md; do
+ test -f "$SKILL_DIR/$f" || fail "missing $f"
+done
+pass "all references present"
+
+python3 - "$SKILL_DIR" <<'PY' || fail "Korean characters found in skill content"
+import pathlib, re, sys
+root = pathlib.Path(sys.argv[1])
+hangul = re.compile(r"[\uac00-\ud7a3]")
+targets = [root / "SKILL.md", root / "README.md"]
+for d in ("references", "scripts", "tests", ".github"):
+ targets.extend((root / d).rglob("*"))
+targets.extend([root / "install.sh", root / "install.ps1"])
+hits = 0
+for p in targets:
+ if not p.is_file():
+ continue
+ try:
+ text = p.read_text(encoding="utf-8")
+ except (UnicodeDecodeError, OSError):
+ continue
+ for n, line in enumerate(text.splitlines(), 1):
+ if hangul.search(line):
+ print(f"{p}:{n}: {line.rstrip()}")
+ hits += 1
+sys.exit(1 if hits else 0)
+PY
+pass "no Korean in skill content"
+
+echo ""
+echo "all smoke tests passed"
diff --git a/src/skills/bundled/audiocraft-audio-generation/SKILL.md b/src/skills/bundled/audiocraft-audio-generation/SKILL.md
new file mode 100644
index 0000000..be5075c
--- /dev/null
+++ b/src/skills/bundled/audiocraft-audio-generation/SKILL.md
@@ -0,0 +1,569 @@
+---
+name: audiocraft-audio-generation
+description: "AudioCraft: MusicGen text-to-music, AudioGen text-to-sound."
+version: 1.0.0
+author: Orchestra Research
+license: MIT
+dependencies: [audiocraft, torch>=2.0.0, transformers>=4.30.0]
+platforms: [linux, macos]
+metadata:
+ hermes:
+ tags: [Multimodal, Audio Generation, Text-to-Music, Text-to-Audio, MusicGen]
+ related_skills: [heartmula, songwriting-and-ai-music]
+
+---
+
+# AudioCraft: Audio Generation
+
+Guide to using Meta's AudioCraft for text-to-music and text-to-audio generation with MusicGen, AudioGen, and EnCodec.
+
+## When to use AudioCraft
+
+**Use AudioCraft when:**
+- Need to generate music from text descriptions
+- Creating sound effects and environmental audio
+- Building music generation applications
+- Need melody-conditioned music generation
+- Want stereo audio output
+- Require controllable music generation with style transfer
+
+**Key features:**
+- **MusicGen**: Text-to-music generation with melody conditioning
+- **AudioGen**: Text-to-sound effects generation
+- **EnCodec**: High-fidelity neural audio codec
+- **Multiple model sizes**: Small (300M) to Large (3.3B)
+- **Stereo support**: Full stereo audio generation
+- **Style conditioning**: MusicGen-Style for reference-based generation
+
+**Use alternatives instead:**
+- **Stable Audio**: For longer commercial music generation
+- **Bark**: For text-to-speech with music/sound effects
+- **Riffusion**: For spectogram-based music generation
+- **OpenAI Jukebox**: For raw audio generation with lyrics
+
+## Quick start
+
+### Installation
+
+```bash
+# From PyPI
+pip install audiocraft
+
+# From GitHub (latest)
+pip install git+https://github.com/facebookresearch/audiocraft.git
+
+# Or use HuggingFace Transformers
+pip install transformers torch torchaudio
+```
+
+### Basic text-to-music (AudioCraft)
+
+```python
+import torchaudio
+from audiocraft.models import MusicGen
+
+# Load model
+model = MusicGen.get_pretrained('facebook/musicgen-small')
+
+# Set generation parameters
+model.set_generation_params(
+ duration=8, # seconds
+ top_k=250,
+ temperature=1.0
+)
+
+# Generate from text
+descriptions = ["happy upbeat electronic dance music with synths"]
+wav = model.generate(descriptions)
+
+# Save audio
+torchaudio.save("output.wav", wav[0].cpu(), sample_rate=32000)
+```
+
+### Using HuggingFace Transformers
+
+```python
+from transformers import AutoProcessor, MusicgenForConditionalGeneration
+import scipy
+
+# Load model and processor
+processor = AutoProcessor.from_pretrained("facebook/musicgen-small")
+model = MusicgenForConditionalGeneration.from_pretrained("facebook/musicgen-small")
+model.to("cuda")
+
+# Generate music
+inputs = processor(
+ text=["80s pop track with bassy drums and synth"],
+ padding=True,
+ return_tensors="pt"
+).to("cuda")
+
+audio_values = model.generate(
+ **inputs,
+ do_sample=True,
+ guidance_scale=3,
+ max_new_tokens=256
+)
+
+# Save
+sampling_rate = model.config.audio_encoder.sampling_rate
+scipy.io.wavfile.write("output.wav", rate=sampling_rate, data=audio_values[0, 0].cpu().numpy())
+```
+
+### Text-to-sound with AudioGen
+
+```python
+from audiocraft.models import AudioGen
+
+# Load AudioGen
+model = AudioGen.get_pretrained('facebook/audiogen-medium')
+
+model.set_generation_params(duration=5)
+
+# Generate sound effects
+descriptions = ["dog barking in a park with birds chirping"]
+wav = model.generate(descriptions)
+
+torchaudio.save("sound.wav", wav[0].cpu(), sample_rate=16000)
+```
+
+## Core concepts
+
+### Architecture overview
+
+```
+AudioCraft Architecture:
+┌──────────────────────────────────────────────────────────────┐
+│ Text Encoder (T5) │
+│ │ │
+│ Text Embeddings │
+└────────────────────────┬─────────────────────────────────────┘
+ │
+┌────────────────────────▼─────────────────────────────────────┐
+│ Transformer Decoder (LM) │
+│ Auto-regressively generates audio tokens │
+│ Using efficient token interleaving patterns │
+└────────────────────────┬─────────────────────────────────────┘
+ │
+┌────────────────────────▼─────────────────────────────────────┐
+│ EnCodec Audio Decoder │
+│ Converts tokens back to audio waveform │
+└──────────────────────────────────────────────────────────────┘
+```
+
+### Model variants
+
+| Model | Size | Description | Use Case |
+|-------|------|-------------|----------|
+| `musicgen-small` | 300M | Text-to-music | Quick generation |
+| `musicgen-medium` | 1.5B | Text-to-music | Balanced |
+| `musicgen-large` | 3.3B | Text-to-music | Best quality |
+| `musicgen-melody` | 1.5B | Text + melody | Melody conditioning |
+| `musicgen-melody-large` | 3.3B | Text + melody | Best melody |
+| `musicgen-stereo-*` | Varies | Stereo output | Stereo generation |
+| `musicgen-style` | 1.5B | Style transfer | Reference-based |
+| `audiogen-medium` | 1.5B | Text-to-sound | Sound effects |
+
+### Generation parameters
+
+| Parameter | Default | Description |
+|-----------|---------|-------------|
+| `duration` | 8.0 | Length in seconds (1-120) |
+| `top_k` | 250 | Top-k sampling |
+| `top_p` | 0.0 | Nucleus sampling (0 = disabled) |
+| `temperature` | 1.0 | Sampling temperature |
+| `cfg_coef` | 3.0 | Classifier-free guidance |
+
+## MusicGen usage
+
+### Text-to-music generation
+
+```python
+from audiocraft.models import MusicGen
+import torchaudio
+
+model = MusicGen.get_pretrained('facebook/musicgen-medium')
+
+# Configure generation
+model.set_generation_params(
+ duration=30, # Up to 30 seconds
+ top_k=250, # Sampling diversity
+ top_p=0.0, # 0 = use top_k only
+ temperature=1.0, # Creativity (higher = more varied)
+ cfg_coef=3.0 # Text adherence (higher = stricter)
+)
+
+# Generate multiple samples
+descriptions = [
+ "epic orchestral soundtrack with strings and brass",
+ "chill lo-fi hip hop beat with jazzy piano",
+ "energetic rock song with electric guitar"
+]
+
+# Generate (returns [batch, channels, samples])
+wav = model.generate(descriptions)
+
+# Save each
+for i, audio in enumerate(wav):
+ torchaudio.save(f"music_{i}.wav", audio.cpu(), sample_rate=32000)
+```
+
+### Melody-conditioned generation
+
+```python
+from audiocraft.models import MusicGen
+import torchaudio
+
+# Load melody model
+model = MusicGen.get_pretrained('facebook/musicgen-melody')
+model.set_generation_params(duration=30)
+
+# Load melody audio
+melody, sr = torchaudio.load("melody.wav")
+
+# Generate with melody conditioning
+descriptions = ["acoustic guitar folk song"]
+wav = model.generate_with_chroma(descriptions, melody, sr)
+
+torchaudio.save("melody_conditioned.wav", wav[0].cpu(), sample_rate=32000)
+```
+
+### Stereo generation
+
+```python
+from audiocraft.models import MusicGen
+
+# Load stereo model
+model = MusicGen.get_pretrained('facebook/musicgen-stereo-medium')
+model.set_generation_params(duration=15)
+
+descriptions = ["ambient electronic music with wide stereo panning"]
+wav = model.generate(descriptions)
+
+# wav shape: [batch, 2, samples] for stereo
+print(f"Stereo shape: {wav.shape}") # [1, 2, 480000]
+torchaudio.save("stereo.wav", wav[0].cpu(), sample_rate=32000)
+```
+
+### Audio continuation
+
+```python
+from transformers import AutoProcessor, MusicgenForConditionalGeneration
+
+processor = AutoProcessor.from_pretrained("facebook/musicgen-medium")
+model = MusicgenForConditionalGeneration.from_pretrained("facebook/musicgen-medium")
+
+# Load audio to continue
+import torchaudio
+audio, sr = torchaudio.load("intro.wav")
+
+# Process with text and audio
+inputs = processor(
+ audio=audio.squeeze().numpy(),
+ sampling_rate=sr,
+ text=["continue with a epic chorus"],
+ padding=True,
+ return_tensors="pt"
+)
+
+# Generate continuation
+audio_values = model.generate(**inputs, do_sample=True, guidance_scale=3, max_new_tokens=512)
+```
+
+## MusicGen-Style usage
+
+### Style-conditioned generation
+
+```python
+from audiocraft.models import MusicGen
+
+# Load style model
+model = MusicGen.get_pretrained('facebook/musicgen-style')
+
+# Configure generation with style
+model.set_generation_params(
+ duration=30,
+ cfg_coef=3.0,
+ cfg_coef_beta=5.0 # Style influence
+)
+
+# Configure style conditioner
+model.set_style_conditioner_params(
+ eval_q=3, # RVQ quantizers (1-6)
+ excerpt_length=3.0 # Style excerpt length
+)
+
+# Load style reference
+style_audio, sr = torchaudio.load("reference_style.wav")
+
+# Generate with text + style
+descriptions = ["upbeat dance track"]
+wav = model.generate_with_style(descriptions, style_audio, sr)
+```
+
+### Style-only generation (no text)
+
+```python
+# Generate matching style without text prompt
+model.set_generation_params(
+ duration=30,
+ cfg_coef=3.0,
+ cfg_coef_beta=None # Disable double CFG for style-only
+)
+
+wav = model.generate_with_style([None], style_audio, sr)
+```
+
+## AudioGen usage
+
+### Sound effect generation
+
+```python
+from audiocraft.models import AudioGen
+import torchaudio
+
+model = AudioGen.get_pretrained('facebook/audiogen-medium')
+model.set_generation_params(duration=10)
+
+# Generate various sounds
+descriptions = [
+ "thunderstorm with heavy rain and lightning",
+ "busy city traffic with car horns",
+ "ocean waves crashing on rocks",
+ "crackling campfire in forest"
+]
+
+wav = model.generate(descriptions)
+
+for i, audio in enumerate(wav):
+ torchaudio.save(f"sound_{i}.wav", audio.cpu(), sample_rate=16000)
+```
+
+## EnCodec usage
+
+### Audio compression
+
+```python
+from audiocraft.models import CompressionModel
+import torch
+import torchaudio
+
+# Load EnCodec
+model = CompressionModel.get_pretrained('facebook/encodec_32khz')
+
+# Load audio
+wav, sr = torchaudio.load("audio.wav")
+
+# Ensure correct sample rate
+if sr != 32000:
+ resampler = torchaudio.transforms.Resample(sr, 32000)
+ wav = resampler(wav)
+
+# Encode to tokens
+with torch.no_grad():
+ encoded = model.encode(wav.unsqueeze(0))
+ codes = encoded[0] # Audio codes
+
+# Decode back to audio
+with torch.no_grad():
+ decoded = model.decode(codes)
+
+torchaudio.save("reconstructed.wav", decoded[0].cpu(), sample_rate=32000)
+```
+
+## Common workflows
+
+### Workflow 1: Music generation pipeline
+
+```python
+import torch
+import torchaudio
+from audiocraft.models import MusicGen
+
+class MusicGenerator:
+ def __init__(self, model_name="facebook/musicgen-medium"):
+ self.model = MusicGen.get_pretrained(model_name)
+ self.sample_rate = 32000
+
+ def generate(self, prompt, duration=30, temperature=1.0, cfg=3.0):
+ self.model.set_generation_params(
+ duration=duration,
+ top_k=250,
+ temperature=temperature,
+ cfg_coef=cfg
+ )
+
+ with torch.no_grad():
+ wav = self.model.generate([prompt])
+
+ return wav[0].cpu()
+
+ def generate_batch(self, prompts, duration=30):
+ self.model.set_generation_params(duration=duration)
+
+ with torch.no_grad():
+ wav = self.model.generate(prompts)
+
+ return wav.cpu()
+
+ def save(self, audio, path):
+ torchaudio.save(path, audio, sample_rate=self.sample_rate)
+
+# Usage
+generator = MusicGenerator()
+audio = generator.generate(
+ "epic cinematic orchestral music",
+ duration=30,
+ temperature=1.0
+)
+generator.save(audio, "epic_music.wav")
+```
+
+### Workflow 2: Sound design batch processing
+
+```python
+import json
+from pathlib import Path
+from audiocraft.models import AudioGen
+import torchaudio
+
+def batch_generate_sounds(sound_specs, output_dir):
+ """
+ Generate multiple sounds from specifications.
+
+ Args:
+ sound_specs: list of {"name": str, "description": str, "duration": float}
+ output_dir: output directory path
+ """
+ model = AudioGen.get_pretrained('facebook/audiogen-medium')
+ output_dir = Path(output_dir)
+ output_dir.mkdir(exist_ok=True)
+
+ results = []
+
+ for spec in sound_specs:
+ model.set_generation_params(duration=spec.get("duration", 5))
+
+ wav = model.generate([spec["description"]])
+
+ output_path = output_dir / f"{spec['name']}.wav"
+ torchaudio.save(str(output_path), wav[0].cpu(), sample_rate=16000)
+
+ results.append({
+ "name": spec["name"],
+ "path": str(output_path),
+ "description": spec["description"]
+ })
+
+ return results
+
+# Usage
+sounds = [
+ {"name": "explosion", "description": "massive explosion with debris", "duration": 3},
+ {"name": "footsteps", "description": "footsteps on wooden floor", "duration": 5},
+ {"name": "door", "description": "wooden door creaking and closing", "duration": 2}
+]
+
+results = batch_generate_sounds(sounds, "sound_effects/")
+```
+
+### Workflow 3: Gradio demo
+
+```python
+import gradio as gr
+import torch
+import torchaudio
+from audiocraft.models import MusicGen
+
+model = MusicGen.get_pretrained('facebook/musicgen-small')
+
+def generate_music(prompt, duration, temperature, cfg_coef):
+ model.set_generation_params(
+ duration=duration,
+ temperature=temperature,
+ cfg_coef=cfg_coef
+ )
+
+ with torch.no_grad():
+ wav = model.generate([prompt])
+
+ # Save to temp file
+ path = "temp_output.wav"
+ torchaudio.save(path, wav[0].cpu(), sample_rate=32000)
+ return path
+
+demo = gr.Interface(
+ fn=generate_music,
+ inputs=[
+ gr.Textbox(label="Music Description", placeholder="upbeat electronic dance music"),
+ gr.Slider(1, 30, value=8, label="Duration (seconds)"),
+ gr.Slider(0.5, 2.0, value=1.0, label="Temperature"),
+ gr.Slider(1.0, 10.0, value=3.0, label="CFG Coefficient")
+ ],
+ outputs=gr.Audio(label="Generated Music"),
+ title="MusicGen Demo"
+)
+
+demo.launch()
+```
+
+## Performance optimization
+
+### Memory optimization
+
+```python
+# Use smaller model
+model = MusicGen.get_pretrained('facebook/musicgen-small')
+
+# Clear cache between generations
+torch.cuda.empty_cache()
+
+# Generate shorter durations
+model.set_generation_params(duration=10) # Instead of 30
+
+# Use half precision
+model = model.half()
+```
+
+### Batch processing efficiency
+
+```python
+# Process multiple prompts at once (more efficient)
+descriptions = ["prompt1", "prompt2", "prompt3", "prompt4"]
+wav = model.generate(descriptions) # Single batch
+
+# Instead of
+for desc in descriptions:
+ wav = model.generate([desc]) # Multiple batches (slower)
+```
+
+### GPU memory requirements
+
+| Model | FP32 VRAM | FP16 VRAM |
+|-------|-----------|-----------|
+| musicgen-small | ~4GB | ~2GB |
+| musicgen-medium | ~8GB | ~4GB |
+| musicgen-large | ~16GB | ~8GB |
+
+## Common issues
+
+| Issue | Solution |
+|-------|----------|
+| CUDA OOM | Use smaller model, reduce duration |
+| Poor quality | Increase cfg_coef, better prompts |
+| Generation too short | Check max duration setting |
+| Audio artifacts | Try different temperature |
+| Stereo not working | Use stereo model variant |
+
+## References
+
+- **[Advanced Usage](references/advanced-usage.md)** - Training, fine-tuning, deployment
+- **[Troubleshooting](references/troubleshooting.md)** - Common issues and solutions
+
+## Resources
+
+- **GitHub**: https://github.com/facebookresearch/audiocraft
+- **Paper (MusicGen)**: https://arxiv.org/abs/2306.05284
+- **Paper (AudioGen)**: https://arxiv.org/abs/2209.15352
+- **HuggingFace**: https://huggingface.co/facebook/musicgen-small
+- **Demo**: https://huggingface.co/spaces/facebook/MusicGen
diff --git a/src/skills/bundled/audiocraft-audio-generation/references/advanced-usage.md b/src/skills/bundled/audiocraft-audio-generation/references/advanced-usage.md
new file mode 100644
index 0000000..953be2b
--- /dev/null
+++ b/src/skills/bundled/audiocraft-audio-generation/references/advanced-usage.md
@@ -0,0 +1,666 @@
+# AudioCraft Advanced Usage Guide
+
+## Fine-tuning MusicGen
+
+### Custom dataset preparation
+
+```python
+import os
+import json
+from pathlib import Path
+import torchaudio
+
+def prepare_dataset(audio_dir, output_dir, metadata_file):
+ """
+ Prepare dataset for MusicGen fine-tuning.
+
+ Directory structure:
+ output_dir/
+ ├── audio/
+ │ ├── 0001.wav
+ │ ├── 0002.wav
+ │ └── ...
+ └── metadata.json
+ """
+ output_dir = Path(output_dir)
+ audio_output = output_dir / "audio"
+ audio_output.mkdir(parents=True, exist_ok=True)
+
+ # Load metadata (format: {"path": "...", "description": "..."})
+ with open(metadata_file) as f:
+ metadata = json.load(f)
+
+ processed = []
+
+ for idx, item in enumerate(metadata):
+ audio_path = Path(audio_dir) / item["path"]
+
+ # Load and resample to 32kHz
+ wav, sr = torchaudio.load(str(audio_path))
+ if sr != 32000:
+ resampler = torchaudio.transforms.Resample(sr, 32000)
+ wav = resampler(wav)
+
+ # Convert to mono if stereo
+ if wav.shape[0] > 1:
+ wav = wav.mean(dim=0, keepdim=True)
+
+ # Save processed audio
+ output_path = audio_output / f"{idx:04d}.wav"
+ torchaudio.save(str(output_path), wav, sample_rate=32000)
+
+ processed.append({
+ "path": str(output_path.relative_to(output_dir)),
+ "description": item["description"],
+ "duration": wav.shape[1] / 32000
+ })
+
+ # Save processed metadata
+ with open(output_dir / "metadata.json", "w") as f:
+ json.dump(processed, f, indent=2)
+
+ print(f"Processed {len(processed)} samples")
+ return processed
+```
+
+### Fine-tuning with dora
+
+```bash
+# AudioCraft uses dora for experiment management
+# Install dora
+pip install dora-search
+
+# Clone AudioCraft
+git clone https://github.com/facebookresearch/audiocraft.git
+cd audiocraft
+
+# Create config for fine-tuning
+cat > config/solver/musicgen/finetune.yaml << 'EOF'
+defaults:
+ - musicgen/musicgen_base
+ - /model: lm/musicgen_lm
+ - /conditioner: cond_base
+
+solver: musicgen
+autocast: true
+autocast_dtype: float16
+
+optim:
+ epochs: 100
+ batch_size: 4
+ lr: 1e-4
+ ema: 0.999
+ optimizer: adamw
+
+dataset:
+ batch_size: 4
+ num_workers: 4
+ train:
+ - dset: your_dataset
+ root: /path/to/dataset
+ valid:
+ - dset: your_dataset
+ root: /path/to/dataset
+
+checkpoint:
+ save_every: 10
+ keep_every_states: null
+EOF
+
+# Run fine-tuning
+dora run solver=musicgen/finetune
+```
+
+### LoRA fine-tuning
+
+```python
+from peft import LoraConfig, get_peft_model
+from audiocraft.models import MusicGen
+import torch
+
+# Load base model
+model = MusicGen.get_pretrained('facebook/musicgen-small')
+
+# Get the language model component
+lm = model.lm
+
+# Configure LoRA
+lora_config = LoraConfig(
+ r=8,
+ lora_alpha=16,
+ target_modules=["q_proj", "v_proj", "k_proj", "out_proj"],
+ lora_dropout=0.05,
+ bias="none"
+)
+
+# Apply LoRA
+lm = get_peft_model(lm, lora_config)
+lm.print_trainable_parameters()
+```
+
+## Multi-GPU Training
+
+### DataParallel
+
+```python
+import torch
+import torch.nn as nn
+from audiocraft.models import MusicGen
+
+model = MusicGen.get_pretrained('facebook/musicgen-small')
+
+# Wrap LM with DataParallel
+if torch.cuda.device_count() > 1:
+ model.lm = nn.DataParallel(model.lm)
+
+model.to("cuda")
+```
+
+### DistributedDataParallel
+
+```python
+import torch.distributed as dist
+from torch.nn.parallel import DistributedDataParallel as DDP
+
+def setup(rank, world_size):
+ dist.init_process_group("nccl", rank=rank, world_size=world_size)
+ torch.cuda.set_device(rank)
+
+def train(rank, world_size):
+ setup(rank, world_size)
+
+ model = MusicGen.get_pretrained('facebook/musicgen-small')
+ model.lm = model.lm.to(rank)
+ model.lm = DDP(model.lm, device_ids=[rank])
+
+ # Training loop
+ # ...
+
+ dist.destroy_process_group()
+```
+
+## Custom Conditioning
+
+### Adding new conditioners
+
+```python
+from audiocraft.modules.conditioners import BaseConditioner
+import torch
+
+class CustomConditioner(BaseConditioner):
+ """Custom conditioner for additional control signals."""
+
+ def __init__(self, dim, output_dim):
+ super().__init__(dim, output_dim)
+ self.embed = torch.nn.Linear(dim, output_dim)
+
+ def forward(self, x):
+ return self.embed(x)
+
+ def tokenize(self, x):
+ # Tokenize input for conditioning
+ return x
+
+# Use with MusicGen
+from audiocraft.models.builders import get_lm_model
+
+# Modify model config to include custom conditioner
+# This requires editing the model configuration
+```
+
+### Melody conditioning internals
+
+```python
+from audiocraft.models import MusicGen
+from audiocraft.modules.codebooks_patterns import DelayedPatternProvider
+import torch
+
+model = MusicGen.get_pretrained('facebook/musicgen-melody')
+
+# Access chroma extractor
+chroma_extractor = model.lm.condition_provider.conditioners.get('chroma')
+
+# Manual chroma extraction
+def extract_chroma(audio, sr):
+ """Extract chroma features from audio."""
+ import librosa
+
+ # Compute chroma
+ chroma = librosa.feature.chroma_cqt(y=audio.numpy(), sr=sr)
+
+ return torch.from_numpy(chroma).float()
+
+# Use extracted chroma for conditioning
+chroma = extract_chroma(melody_audio, sample_rate)
+```
+
+## EnCodec Deep Dive
+
+### Custom compression settings
+
+```python
+from audiocraft.models import CompressionModel
+import torch
+
+# Load EnCodec
+encodec = CompressionModel.get_pretrained('facebook/encodec_32khz')
+
+# Access codec parameters
+print(f"Sample rate: {encodec.sample_rate}")
+print(f"Channels: {encodec.channels}")
+print(f"Cardinality: {encodec.cardinality}") # Codebook size
+print(f"Num codebooks: {encodec.num_codebooks}")
+print(f"Frame rate: {encodec.frame_rate}")
+
+# Encode with specific bandwidth
+# Lower bandwidth = more compression, lower quality
+encodec.set_target_bandwidth(6.0) # 6 kbps
+
+audio = torch.randn(1, 1, 32000) # 1 second
+encoded = encodec.encode(audio)
+decoded = encodec.decode(encoded[0])
+```
+
+### Streaming encoding
+
+```python
+import torch
+from audiocraft.models import CompressionModel
+
+encodec = CompressionModel.get_pretrained('facebook/encodec_32khz')
+
+def encode_streaming(audio_stream, chunk_size=32000):
+ """Encode audio in streaming fashion."""
+ all_codes = []
+
+ for chunk in audio_stream:
+ # Ensure chunk is right shape
+ if chunk.dim() == 1:
+ chunk = chunk.unsqueeze(0).unsqueeze(0)
+
+ with torch.no_grad():
+ codes = encodec.encode(chunk)[0]
+ all_codes.append(codes)
+
+ return torch.cat(all_codes, dim=-1)
+
+def decode_streaming(codes_stream, output_stream):
+ """Decode codes in streaming fashion."""
+ for codes in codes_stream:
+ with torch.no_grad():
+ audio = encodec.decode(codes)
+ output_stream.write(audio.cpu().numpy())
+```
+
+## MultiBand Diffusion
+
+### Using MBD for enhanced quality
+
+```python
+from audiocraft.models import MusicGen, MultiBandDiffusion
+
+# Load MusicGen
+model = MusicGen.get_pretrained('facebook/musicgen-medium')
+
+# Load MultiBand Diffusion
+mbd = MultiBandDiffusion.get_mbd_musicgen()
+
+model.set_generation_params(duration=10)
+
+# Generate with standard decoder
+descriptions = ["epic orchestral music"]
+wav_standard = model.generate(descriptions)
+
+# Generate tokens and use MBD decoder
+with torch.no_grad():
+ # Get tokens
+ gen_tokens = model.generate_tokens(descriptions)
+
+ # Decode with MBD
+ wav_mbd = mbd.tokens_to_wav(gen_tokens)
+
+# Compare quality
+print(f"Standard shape: {wav_standard.shape}")
+print(f"MBD shape: {wav_mbd.shape}")
+```
+
+## API Server Deployment
+
+### FastAPI server
+
+```python
+from fastapi import FastAPI, HTTPException
+from pydantic import BaseModel
+import torch
+import torchaudio
+from audiocraft.models import MusicGen
+import io
+import base64
+
+app = FastAPI()
+
+# Load model at startup
+model = None
+
+@app.on_event("startup")
+async def load_model():
+ global model
+ model = MusicGen.get_pretrained('facebook/musicgen-small')
+ model.set_generation_params(duration=10)
+
+class GenerateRequest(BaseModel):
+ prompt: str
+ duration: float = 10.0
+ temperature: float = 1.0
+ cfg_coef: float = 3.0
+
+class GenerateResponse(BaseModel):
+ audio_base64: str
+ sample_rate: int
+ duration: float
+
+@app.post("/generate", response_model=GenerateResponse)
+async def generate(request: GenerateRequest):
+ if model is None:
+ raise HTTPException(status_code=500, detail="Model not loaded")
+
+ try:
+ model.set_generation_params(
+ duration=min(request.duration, 30),
+ temperature=request.temperature,
+ cfg_coef=request.cfg_coef
+ )
+
+ with torch.no_grad():
+ wav = model.generate([request.prompt])
+
+ # Convert to bytes
+ buffer = io.BytesIO()
+ torchaudio.save(buffer, wav[0].cpu(), sample_rate=32000, format="wav")
+ buffer.seek(0)
+
+ audio_base64 = base64.b64encode(buffer.read()).decode()
+
+ return GenerateResponse(
+ audio_base64=audio_base64,
+ sample_rate=32000,
+ duration=wav.shape[-1] / 32000
+ )
+
+ except Exception as e:
+ raise HTTPException(status_code=500, detail=str(e))
+
+@app.get("/health")
+async def health():
+ return {"status": "ok", "model_loaded": model is not None}
+
+# Run: uvicorn server:app --host 0.0.0.0 --port 8000
+```
+
+### Batch processing service
+
+```python
+import asyncio
+from concurrent.futures import ThreadPoolExecutor
+import torch
+from audiocraft.models import MusicGen
+
+class MusicGenService:
+ def __init__(self, model_name='facebook/musicgen-small', max_workers=2):
+ self.model = MusicGen.get_pretrained(model_name)
+ self.executor = ThreadPoolExecutor(max_workers=max_workers)
+ self.lock = asyncio.Lock()
+
+ async def generate_async(self, prompt, duration=10):
+ """Async generation with thread pool."""
+ loop = asyncio.get_event_loop()
+
+ def _generate():
+ with torch.no_grad():
+ self.model.set_generation_params(duration=duration)
+ return self.model.generate([prompt])
+
+ # Run in thread pool
+ wav = await loop.run_in_executor(self.executor, _generate)
+ return wav[0].cpu()
+
+ async def generate_batch_async(self, prompts, duration=10):
+ """Process multiple prompts concurrently."""
+ tasks = [self.generate_async(p, duration) for p in prompts]
+ return await asyncio.gather(*tasks)
+
+# Usage
+service = MusicGenService()
+
+async def main():
+ prompts = ["jazz piano", "rock guitar", "electronic beats"]
+ results = await service.generate_batch_async(prompts)
+ return results
+```
+
+## Integration Patterns
+
+### LangChain tool
+
+```python
+from langchain.tools import BaseTool
+import torch
+import torchaudio
+from audiocraft.models import MusicGen
+import tempfile
+
+class MusicGeneratorTool(BaseTool):
+ name = "music_generator"
+ description = "Generate music from a text description. Input should be a detailed description of the music style, mood, and instruments."
+
+ def __init__(self):
+ super().__init__()
+ self.model = MusicGen.get_pretrained('facebook/musicgen-small')
+ self.model.set_generation_params(duration=15)
+
+ def _run(self, description: str) -> str:
+ with torch.no_grad():
+ wav = self.model.generate([description])
+
+ # Save to temp file
+ with tempfile.NamedTemporaryFile(suffix=".wav", delete=False) as f:
+ torchaudio.save(f.name, wav[0].cpu(), sample_rate=32000)
+ return f"Generated music saved to: {f.name}"
+
+ async def _arun(self, description: str) -> str:
+ return self._run(description)
+```
+
+### Gradio with advanced controls
+
+```python
+import gradio as gr
+import torch
+import torchaudio
+from audiocraft.models import MusicGen
+
+models = {}
+
+def load_model(model_size):
+ if model_size not in models:
+ model_name = f"facebook/musicgen-{model_size}"
+ models[model_size] = MusicGen.get_pretrained(model_name)
+ return models[model_size]
+
+def generate(prompt, duration, temperature, cfg_coef, top_k, model_size):
+ model = load_model(model_size)
+
+ model.set_generation_params(
+ duration=duration,
+ temperature=temperature,
+ cfg_coef=cfg_coef,
+ top_k=top_k
+ )
+
+ with torch.no_grad():
+ wav = model.generate([prompt])
+
+ # Save
+ path = "output.wav"
+ torchaudio.save(path, wav[0].cpu(), sample_rate=32000)
+ return path
+
+demo = gr.Interface(
+ fn=generate,
+ inputs=[
+ gr.Textbox(label="Prompt", lines=3),
+ gr.Slider(1, 30, value=10, label="Duration (s)"),
+ gr.Slider(0.1, 2.0, value=1.0, label="Temperature"),
+ gr.Slider(0.5, 10.0, value=3.0, label="CFG Coefficient"),
+ gr.Slider(50, 500, value=250, step=50, label="Top-K"),
+ gr.Dropdown(["small", "medium", "large"], value="small", label="Model Size")
+ ],
+ outputs=gr.Audio(label="Generated Music"),
+ title="MusicGen Advanced",
+ allow_flagging="never"
+)
+
+demo.launch(share=True)
+```
+
+## Audio Processing Pipeline
+
+### Post-processing chain
+
+```python
+import torch
+import torchaudio
+import torchaudio.transforms as T
+import numpy as np
+
+class AudioPostProcessor:
+ def __init__(self, sample_rate=32000):
+ self.sample_rate = sample_rate
+
+ def normalize(self, audio, target_db=-14.0):
+ """Normalize audio to target loudness."""
+ rms = torch.sqrt(torch.mean(audio ** 2))
+ target_rms = 10 ** (target_db / 20)
+ gain = target_rms / (rms + 1e-8)
+ return audio * gain
+
+ def fade_in_out(self, audio, fade_duration=0.1):
+ """Apply fade in/out."""
+ fade_samples = int(fade_duration * self.sample_rate)
+
+ # Create fade curves
+ fade_in = torch.linspace(0, 1, fade_samples)
+ fade_out = torch.linspace(1, 0, fade_samples)
+
+ # Apply fades
+ audio[..., :fade_samples] *= fade_in
+ audio[..., -fade_samples:] *= fade_out
+
+ return audio
+
+ def apply_reverb(self, audio, decay=0.5):
+ """Apply simple reverb effect."""
+ impulse = torch.zeros(int(self.sample_rate * 0.5))
+ impulse[0] = 1.0
+ impulse[int(self.sample_rate * 0.1)] = decay * 0.5
+ impulse[int(self.sample_rate * 0.2)] = decay * 0.25
+
+ # Convolve
+ audio = torch.nn.functional.conv1d(
+ audio.unsqueeze(0),
+ impulse.unsqueeze(0).unsqueeze(0),
+ padding=len(impulse) // 2
+ ).squeeze(0)
+
+ return audio
+
+ def process(self, audio):
+ """Full processing pipeline."""
+ audio = self.normalize(audio)
+ audio = self.fade_in_out(audio)
+ return audio
+
+# Usage with MusicGen
+from audiocraft.models import MusicGen
+
+model = MusicGen.get_pretrained('facebook/musicgen-small')
+model.set_generation_params(duration=10)
+
+wav = model.generate(["chill ambient music"])
+processor = AudioPostProcessor()
+wav_processed = processor.process(wav[0].cpu())
+
+torchaudio.save("processed.wav", wav_processed, sample_rate=32000)
+```
+
+## Evaluation
+
+### Audio quality metrics
+
+```python
+import torch
+from audiocraft.metrics import CLAPTextConsistencyMetric
+from audiocraft.data.audio import audio_read
+
+def evaluate_generation(audio_path, text_prompt):
+ """Evaluate generated audio quality."""
+ # Load audio
+ wav, sr = audio_read(audio_path)
+
+ # CLAP consistency (text-audio alignment)
+ clap_metric = CLAPTextConsistencyMetric()
+ clap_score = clap_metric.compute(wav, [text_prompt])
+
+ return {
+ "clap_score": clap_score,
+ "duration": wav.shape[-1] / sr
+ }
+
+# Batch evaluation
+def evaluate_batch(generations):
+ """Evaluate multiple generations."""
+ results = []
+ for gen in generations:
+ result = evaluate_generation(gen["path"], gen["prompt"])
+ result["prompt"] = gen["prompt"]
+ results.append(result)
+
+ # Aggregate
+ avg_clap = sum(r["clap_score"] for r in results) / len(results)
+ return {
+ "individual": results,
+ "average_clap": avg_clap
+ }
+```
+
+## Model Comparison
+
+### MusicGen variants benchmark
+
+| Model | CLAP Score | Generation Time (10s) | VRAM |
+|-------|------------|----------------------|------|
+| musicgen-small | 0.35 | ~5s | 2GB |
+| musicgen-medium | 0.42 | ~15s | 4GB |
+| musicgen-large | 0.48 | ~30s | 8GB |
+| musicgen-melody | 0.45 | ~15s | 4GB |
+| musicgen-stereo-medium | 0.41 | ~18s | 5GB |
+
+### Prompt engineering tips
+
+```python
+# Good prompts - specific and descriptive
+good_prompts = [
+ "upbeat electronic dance music with synthesizer leads and punchy drums at 128 bpm",
+ "melancholic piano ballad with strings, slow tempo, emotional and cinematic",
+ "funky disco groove with slap bass, brass section, and rhythmic guitar"
+]
+
+# Bad prompts - too vague
+bad_prompts = [
+ "nice music",
+ "song",
+ "good beat"
+]
+
+# Structure: [mood] [genre] with [instruments] at [tempo/style]
+```
diff --git a/src/skills/bundled/audiocraft-audio-generation/references/troubleshooting.md b/src/skills/bundled/audiocraft-audio-generation/references/troubleshooting.md
new file mode 100644
index 0000000..7b83e86
--- /dev/null
+++ b/src/skills/bundled/audiocraft-audio-generation/references/troubleshooting.md
@@ -0,0 +1,504 @@
+# AudioCraft Troubleshooting Guide
+
+## Installation Issues
+
+### Import errors
+
+**Error**: `ModuleNotFoundError: No module named 'audiocraft'`
+
+**Solutions**:
+```bash
+# Install from PyPI
+pip install audiocraft
+
+# Or from GitHub
+pip install git+https://github.com/facebookresearch/audiocraft.git
+
+# Verify installation
+python -c "from audiocraft.models import MusicGen; print('OK')"
+```
+
+### FFmpeg not found
+
+**Error**: `RuntimeError: ffmpeg not found`
+
+**Solutions**:
+```bash
+# Ubuntu/Debian
+sudo apt-get install ffmpeg
+
+# macOS
+brew install ffmpeg
+
+# Windows (using conda)
+conda install -c conda-forge ffmpeg
+
+# Verify
+ffmpeg -version
+```
+
+### PyTorch CUDA mismatch
+
+**Error**: `RuntimeError: CUDA error: no kernel image is available`
+
+**Solutions**:
+```bash
+# Check CUDA version
+nvcc --version
+python -c "import torch; print(torch.version.cuda)"
+
+# Install matching PyTorch
+pip install torch torchaudio --index-url https://download.pytorch.org/whl/cu121
+
+# For CUDA 11.8
+pip install torch torchaudio --index-url https://download.pytorch.org/whl/cu118
+```
+
+### xformers issues
+
+**Error**: `ImportError: xformers` related errors
+
+**Solutions**:
+```bash
+# Install xformers for memory efficiency
+pip install xformers
+
+# Or disable xformers
+export AUDIOCRAFT_USE_XFORMERS=0
+
+# In Python
+import os
+os.environ["AUDIOCRAFT_USE_XFORMERS"] = "0"
+from audiocraft.models import MusicGen
+```
+
+## Model Loading Issues
+
+### Out of memory during load
+
+**Error**: `torch.cuda.OutOfMemoryError` during model loading
+
+**Solutions**:
+```python
+# Use smaller model
+model = MusicGen.get_pretrained('facebook/musicgen-small')
+
+# Force CPU loading first
+import torch
+device = "cpu"
+model = MusicGen.get_pretrained('facebook/musicgen-small', device=device)
+model = model.to("cuda")
+
+# Use HuggingFace with device_map
+from transformers import MusicgenForConditionalGeneration
+model = MusicgenForConditionalGeneration.from_pretrained(
+ "facebook/musicgen-small",
+ device_map="auto"
+)
+```
+
+### Download failures
+
+**Error**: Connection errors or incomplete downloads
+
+**Solutions**:
+```python
+# Set cache directory
+import os
+os.environ["AUDIOCRAFT_CACHE_DIR"] = "/path/to/cache"
+
+# Or for HuggingFace
+os.environ["HF_HOME"] = "/path/to/hf_cache"
+
+# Resume download
+from huggingface_hub import snapshot_download
+snapshot_download("facebook/musicgen-small", resume_download=True)
+
+# Use local files
+model = MusicGen.get_pretrained('/local/path/to/model')
+```
+
+### Wrong model type
+
+**Error**: Loading wrong model for task
+
+**Solutions**:
+```python
+# For text-to-music: use MusicGen
+from audiocraft.models import MusicGen
+model = MusicGen.get_pretrained('facebook/musicgen-medium')
+
+# For text-to-sound: use AudioGen
+from audiocraft.models import AudioGen
+model = AudioGen.get_pretrained('facebook/audiogen-medium')
+
+# For melody conditioning: use melody variant
+model = MusicGen.get_pretrained('facebook/musicgen-melody')
+
+# For stereo: use stereo variant
+model = MusicGen.get_pretrained('facebook/musicgen-stereo-medium')
+```
+
+## Generation Issues
+
+### Empty or silent output
+
+**Problem**: Generated audio is silent or very quiet
+
+**Solutions**:
+```python
+import torch
+
+# Check output
+wav = model.generate(["upbeat music"])
+print(f"Shape: {wav.shape}")
+print(f"Max amplitude: {wav.abs().max().item()}")
+print(f"Mean amplitude: {wav.abs().mean().item()}")
+
+# If too quiet, normalize
+def normalize_audio(audio, target_db=-14.0):
+ rms = torch.sqrt(torch.mean(audio ** 2))
+ target_rms = 10 ** (target_db / 20)
+ gain = target_rms / (rms + 1e-8)
+ return audio * gain
+
+wav_normalized = normalize_audio(wav)
+```
+
+### Poor quality output
+
+**Problem**: Generated music sounds bad or noisy
+
+**Solutions**:
+```python
+# Use larger model
+model = MusicGen.get_pretrained('facebook/musicgen-large')
+
+# Adjust generation parameters
+model.set_generation_params(
+ duration=15,
+ top_k=250, # Increase for more diversity
+ temperature=0.8, # Lower for more focused output
+ cfg_coef=4.0 # Increase for better text adherence
+)
+
+# Use better prompts
+# Bad: "music"
+# Good: "upbeat electronic dance music with synthesizers and punchy drums"
+
+# Try MultiBand Diffusion
+from audiocraft.models import MultiBandDiffusion
+mbd = MultiBandDiffusion.get_mbd_musicgen()
+tokens = model.generate_tokens(["prompt"])
+wav = mbd.tokens_to_wav(tokens)
+```
+
+### Generation too short
+
+**Problem**: Audio shorter than expected
+
+**Solutions**:
+```python
+# Check duration setting
+model.set_generation_params(duration=30) # Set before generate
+
+# Verify in generation
+print(f"Duration setting: {model.generation_params}")
+
+# Check output shape
+wav = model.generate(["prompt"])
+actual_duration = wav.shape[-1] / 32000
+print(f"Actual duration: {actual_duration}s")
+
+# Note: max duration is typically 30s
+```
+
+### Melody conditioning fails
+
+**Error**: Issues with melody-conditioned generation
+
+**Solutions**:
+```python
+import torchaudio
+from audiocraft.models import MusicGen
+
+# Load melody model (not base model)
+model = MusicGen.get_pretrained('facebook/musicgen-melody')
+
+# Load and prepare melody
+melody, sr = torchaudio.load("melody.wav")
+
+# Resample to model sample rate if needed
+if sr != 32000:
+ resampler = torchaudio.transforms.Resample(sr, 32000)
+ melody = resampler(melody)
+
+# Ensure correct shape [batch, channels, samples]
+if melody.dim() == 1:
+ melody = melody.unsqueeze(0).unsqueeze(0)
+elif melody.dim() == 2:
+ melody = melody.unsqueeze(0)
+
+# Convert stereo to mono
+if melody.shape[1] > 1:
+ melody = melody.mean(dim=1, keepdim=True)
+
+# Generate with melody
+model.set_generation_params(duration=min(melody.shape[-1] / 32000, 30))
+wav = model.generate_with_chroma(["piano cover"], melody, 32000)
+```
+
+## Memory Issues
+
+### CUDA out of memory
+
+**Error**: `torch.cuda.OutOfMemoryError: CUDA out of memory`
+
+**Solutions**:
+```python
+import torch
+
+# Clear cache before generation
+torch.cuda.empty_cache()
+
+# Use smaller model
+model = MusicGen.get_pretrained('facebook/musicgen-small')
+
+# Reduce duration
+model.set_generation_params(duration=10) # Instead of 30
+
+# Generate one at a time
+for prompt in prompts:
+ wav = model.generate([prompt])
+ save_audio(wav)
+ torch.cuda.empty_cache()
+
+# Use CPU for very large generations
+model = MusicGen.get_pretrained('facebook/musicgen-small', device="cpu")
+```
+
+### Memory leak during batch processing
+
+**Problem**: Memory grows over time
+
+**Solutions**:
+```python
+import gc
+import torch
+
+def generate_with_cleanup(model, prompts):
+ results = []
+
+ for prompt in prompts:
+ with torch.no_grad():
+ wav = model.generate([prompt])
+ results.append(wav.cpu())
+
+ # Cleanup
+ del wav
+ gc.collect()
+ torch.cuda.empty_cache()
+
+ return results
+
+# Use context manager
+with torch.inference_mode():
+ wav = model.generate(["prompt"])
+```
+
+## Audio Format Issues
+
+### Wrong sample rate
+
+**Problem**: Audio plays at wrong speed
+
+**Solutions**:
+```python
+import torchaudio
+
+# MusicGen outputs at 32kHz
+sample_rate = 32000
+
+# AudioGen outputs at 16kHz
+sample_rate = 16000
+
+# Always use correct rate when saving
+torchaudio.save("output.wav", wav[0].cpu(), sample_rate=sample_rate)
+
+# Resample if needed
+resampler = torchaudio.transforms.Resample(32000, 44100)
+wav_resampled = resampler(wav)
+```
+
+### Stereo/mono mismatch
+
+**Problem**: Wrong number of channels
+
+**Solutions**:
+```python
+# Check model type
+print(f"Audio channels: {wav.shape}")
+# Mono: [batch, 1, samples]
+# Stereo: [batch, 2, samples]
+
+# Convert mono to stereo
+if wav.shape[1] == 1:
+ wav_stereo = wav.repeat(1, 2, 1)
+
+# Convert stereo to mono
+if wav.shape[1] == 2:
+ wav_mono = wav.mean(dim=1, keepdim=True)
+
+# Use stereo model for stereo output
+model = MusicGen.get_pretrained('facebook/musicgen-stereo-medium')
+```
+
+### Clipping and distortion
+
+**Problem**: Audio has clipping or distortion
+
+**Solutions**:
+```python
+import torch
+
+# Check for clipping
+max_val = wav.abs().max().item()
+print(f"Max amplitude: {max_val}")
+
+# Normalize to prevent clipping
+if max_val > 1.0:
+ wav = wav / max_val
+
+# Apply soft clipping
+def soft_clip(x, threshold=0.9):
+ return torch.tanh(x / threshold) * threshold
+
+wav_clipped = soft_clip(wav)
+
+# Lower temperature during generation
+model.set_generation_params(temperature=0.7) # More controlled
+```
+
+## HuggingFace Transformers Issues
+
+### Processor errors
+
+**Error**: Issues with MusicgenProcessor
+
+**Solutions**:
+```python
+from transformers import AutoProcessor, MusicgenForConditionalGeneration
+
+# Load matching processor and model
+processor = AutoProcessor.from_pretrained("facebook/musicgen-small")
+model = MusicgenForConditionalGeneration.from_pretrained("facebook/musicgen-small")
+
+# Ensure inputs are on same device
+inputs = processor(
+ text=["prompt"],
+ padding=True,
+ return_tensors="pt"
+).to("cuda")
+
+# Check processor configuration
+print(processor.tokenizer)
+print(processor.feature_extractor)
+```
+
+### Generation parameter errors
+
+**Error**: Invalid generation parameters
+
+**Solutions**:
+```python
+# HuggingFace uses different parameter names
+audio_values = model.generate(
+ **inputs,
+ do_sample=True, # Enable sampling
+ guidance_scale=3.0, # CFG (not cfg_coef)
+ max_new_tokens=256, # Token limit (not duration)
+ temperature=1.0
+)
+
+# Calculate tokens from duration
+# ~50 tokens per second
+duration_seconds = 10
+max_tokens = duration_seconds * 50
+audio_values = model.generate(**inputs, max_new_tokens=max_tokens)
+```
+
+## Performance Issues
+
+### Slow generation
+
+**Problem**: Generation takes too long
+
+**Solutions**:
+```python
+# Use smaller model
+model = MusicGen.get_pretrained('facebook/musicgen-small')
+
+# Reduce duration
+model.set_generation_params(duration=10)
+
+# Use GPU
+model.to("cuda")
+
+# Enable flash attention if available
+# (requires compatible hardware)
+
+# Batch multiple prompts
+prompts = ["prompt1", "prompt2", "prompt3"]
+wav = model.generate(prompts) # Single batch is faster than loop
+
+# Use compile (PyTorch 2.0+)
+model.lm = torch.compile(model.lm)
+```
+
+### CPU fallback
+
+**Problem**: Generation running on CPU instead of GPU
+
+**Solutions**:
+```python
+import torch
+
+# Check CUDA availability
+print(f"CUDA available: {torch.cuda.is_available()}")
+print(f"CUDA device: {torch.cuda.get_device_name(0)}")
+
+# Explicitly move to GPU
+model = MusicGen.get_pretrained('facebook/musicgen-small')
+model.to("cuda")
+
+# Verify model device
+print(f"Model device: {next(model.lm.parameters()).device}")
+```
+
+## Common Error Messages
+
+| Error | Cause | Solution |
+|-------|-------|----------|
+| `CUDA out of memory` | Model too large | Use smaller model, reduce duration |
+| `ffmpeg not found` | FFmpeg not installed | Install FFmpeg |
+| `No module named 'audiocraft'` | Not installed | `pip install audiocraft` |
+| `RuntimeError: Expected 3D tensor` | Wrong input shape | Check tensor dimensions |
+| `KeyError: 'melody'` | Wrong model for melody | Use musicgen-melody |
+| `Sample rate mismatch` | Wrong audio format | Resample to model rate |
+
+## Getting Help
+
+1. **GitHub Issues**: https://github.com/facebookresearch/audiocraft/issues
+2. **HuggingFace Forums**: https://discuss.huggingface.co
+3. **Paper**: https://arxiv.org/abs/2306.05284
+
+### Reporting Issues
+
+Include:
+- Python version
+- PyTorch version
+- CUDA version
+- AudioCraft version: `pip show audiocraft`
+- Full error traceback
+- Minimal reproducible code
+- Hardware (GPU model, VRAM)
diff --git a/src/skills/bundled/axolotl/SKILL.md b/src/skills/bundled/axolotl/SKILL.md
new file mode 100644
index 0000000..513dea4
--- /dev/null
+++ b/src/skills/bundled/axolotl/SKILL.md
@@ -0,0 +1,166 @@
+---
+name: axolotl
+description: "Axolotl: YAML LLM fine-tuning (LoRA, DPO, GRPO)."
+version: 1.0.0
+author: Orchestra Research
+license: MIT
+dependencies: [axolotl, torch, transformers, datasets, peft, accelerate, deepspeed]
+platforms: [linux, macos]
+metadata:
+ hermes:
+ tags: [Fine-Tuning, Axolotl, LLM, LoRA, QLoRA, DPO, KTO, ORPO, GRPO, YAML, HuggingFace, DeepSpeed, Multimodal]
+
+---
+
+# Axolotl Skill
+
+## What's inside
+
+Expert guidance for fine-tuning LLMs with Axolotl — YAML configs, 100+ models, LoRA/QLoRA, DPO/KTO/ORPO/GRPO, multimodal support.
+
+Assistance with axolotl development, generated from official documentation.
+
+## When to Use This Skill
+
+This skill should be triggered when:
+- Working with axolotl
+- Asking about axolotl features or APIs
+- Implementing axolotl solutions
+- Debugging axolotl code
+- Learning axolotl best practices
+
+## Quick Reference
+
+### Common Patterns
+
+**Pattern 1:** To validate that acceptable data transfer speeds exist for your training job, running NCCL Tests can help pinpoint bottlenecks, for example:
+
+```
+./build/all_reduce_perf -b 8 -e 128M -f 2 -g 3
+```
+
+**Pattern 2:** Configure your model to use FSDP in the Axolotl yaml. For example:
+
+```
+fsdp_version: 2
+fsdp_config:
+ offload_params: true
+ state_dict_type: FULL_STATE_DICT
+ auto_wrap_policy: TRANSFORMER_BASED_WRAP
+ transformer_layer_cls_to_wrap: LlamaDecoderLayer
+ reshard_after_forward: true
+```
+
+**Pattern 3:** The context_parallel_size should be a divisor of the total number of GPUs. For example:
+
+```
+context_parallel_size
+```
+
+**Pattern 4:** For example: - With 8 GPUs and no sequence parallelism: 8 different batches processed per step - With 8 GPUs and context_parallel_size=4: Only 2 different batches processed per step (each split across 4 GPUs) - If your per-GPU micro_batch_size is 2, the global batch size decreases from 16 to 4
+
+```
+context_parallel_size=4
+```
+
+**Pattern 5:** Setting save_compressed: true in your configuration enables saving models in a compressed format, which: - Reduces disk space usage by approximately 40% - Maintains compatibility with vLLM for accelerated inference - Maintains compatibility with llmcompressor for further optimization (example: quantization)
+
+```
+save_compressed: true
+```
+
+**Pattern 6:** Note It is not necessary to place your integration in the integrations folder. It can be in any location, so long as it’s installed in a package in your python env. See this repo for an example: https://github.com/axolotl-ai-cloud/diff-transformer
+
+```
+integrations
+```
+
+**Pattern 7:** Handle both single-example and batched data. - single example: sample[‘input_ids’] is a list[int] - batched data: sample[‘input_ids’] is a list[list[int]]
+
+```
+utils.trainer.drop_long_seq(sample, sequence_len=2048, min_sequence_len=2)
+```
+
+### Example Code Patterns
+
+**Example 1** (python):
+```python
+cli.cloud.modal_.ModalCloud(config, app=None)
+```
+
+**Example 2** (python):
+```python
+cli.cloud.modal_.run_cmd(cmd, run_folder, volumes=None)
+```
+
+**Example 3** (python):
+```python
+core.trainers.base.AxolotlTrainer(
+ *_args,
+ bench_data_collator=None,
+ eval_data_collator=None,
+ dataset_tags=None,
+ **kwargs,
+)
+```
+
+**Example 4** (python):
+```python
+core.trainers.base.AxolotlTrainer.log(logs, start_time=None)
+```
+
+**Example 5** (python):
+```python
+prompt_strategies.input_output.RawInputOutputPrompter()
+```
+
+## Reference Files
+
+This skill includes comprehensive documentation in `references/`:
+
+- **api.md** - Api documentation
+- **dataset-formats.md** - Dataset-Formats documentation
+- **other.md** - Other documentation
+
+Use `view` to read specific reference files when detailed information is needed.
+
+## Working with This Skill
+
+### For Beginners
+Start with the getting_started or tutorials reference files for foundational concepts.
+
+### For Specific Features
+Use the appropriate category reference file (api, guides, etc.) for detailed information.
+
+### For Code Examples
+The quick reference section above contains common patterns extracted from the official docs.
+
+## Resources
+
+### references/
+Organized documentation extracted from official sources. These files contain:
+- Detailed explanations
+- Code examples with language annotations
+- Links to original documentation
+- Table of contents for quick navigation
+
+### scripts/
+Add helper scripts here for common automation tasks.
+
+### assets/
+Add templates, boilerplate, or example projects here.
+
+## Notes
+
+- This skill was automatically generated from official documentation
+- Reference files preserve the structure and examples from source docs
+- Code examples include language detection for better syntax highlighting
+- Quick reference patterns are extracted from common usage examples in the docs
+
+## Updating
+
+To refresh this skill with updated documentation:
+1. Re-run the scraper with the same configuration
+2. The skill will be rebuilt with the latest information
+
+
diff --git a/src/skills/bundled/axolotl/references/api.md b/src/skills/bundled/axolotl/references/api.md
new file mode 100644
index 0000000..2f94b53
--- /dev/null
+++ b/src/skills/bundled/axolotl/references/api.md
@@ -0,0 +1,5548 @@
+# Axolotl - Api
+
+**Pages:** 150
+
+---
+
+## cli.cloud.modal_
+
+**URL:** https://docs.axolotl.ai/docs/api/cli.cloud.modal_.html
+
+**Contents:**
+- cli.cloud.modal_
+- Classes
+ - ModalCloud
+- Functions
+ - run_cmd
+
+Modal Cloud support from CLI
+
+Modal Cloud implementation.
+
+Run a command inside a folder, with Modal Volume reloading before and commit on success.
+
+**Examples:**
+
+Example 1 (python):
+```python
+cli.cloud.modal_.ModalCloud(config, app=None)
+```
+
+Example 2 (python):
+```python
+cli.cloud.modal_.run_cmd(cmd, run_folder, volumes=None)
+```
+
+---
+
+## core.trainers.base
+
+**URL:** https://docs.axolotl.ai/docs/api/core.trainers.base.html
+
+**Contents:**
+- core.trainers.base
+- Classes
+ - AxolotlTrainer
+ - Methods
+ - log
+ - Parameters
+ - push_to_hub
+ - store_metrics
+ - Parameters
+
+Module for customized trainers
+
+Extend the base Trainer for axolotl helpers
+
+Log logs on the various objects watching training, including stored metrics.
+
+Overwrite the push_to_hub method in order to force-add the tags when pushing the model on the Hub. Please refer to ~transformers.Trainer.push_to_hub for more details.
+
+Store metrics with specified reduction type.
+
+**Examples:**
+
+Example 1 (python):
+```python
+core.trainers.base.AxolotlTrainer(
+ *_args,
+ bench_data_collator=None,
+ eval_data_collator=None,
+ dataset_tags=None,
+ **kwargs,
+)
+```
+
+Example 2 (python):
+```python
+core.trainers.base.AxolotlTrainer.log(logs, start_time=None)
+```
+
+Example 3 (python):
+```python
+core.trainers.base.AxolotlTrainer.push_to_hub(*args, **kwargs)
+```
+
+Example 4 (python):
+```python
+core.trainers.base.AxolotlTrainer.store_metrics(
+ metrics,
+ train_eval='train',
+ reduction='mean',
+)
+```
+
+---
+
+## prompt_strategies.input_output
+
+**URL:** https://docs.axolotl.ai/docs/api/prompt_strategies.input_output.html
+
+**Contents:**
+- prompt_strategies.input_output
+- Classes
+ - RawInputOutputPrompter
+ - RawInputOutputStrategy
+
+prompt_strategies.input_output
+
+Module for plain input/output prompt pairs
+
+prompter for raw i/o data
+
+Prompt Strategy class for input/output pairs
+
+**Examples:**
+
+Example 1 (python):
+```python
+prompt_strategies.input_output.RawInputOutputPrompter()
+```
+
+Example 2 (python):
+```python
+prompt_strategies.input_output.RawInputOutputStrategy(
+ *args,
+ eos_token=None,
+ **kwargs,
+)
+```
+
+---
+
+## prompt_strategies.completion
+
+**URL:** https://docs.axolotl.ai/docs/api/prompt_strategies.completion.html
+
+**Contents:**
+- prompt_strategies.completion
+- Classes
+ - CompletionPromptTokenizingStrategy
+ - CompletionPrompter
+
+prompt_strategies.completion
+
+Basic completion text
+
+Tokenizing strategy for Completion prompts.
+
+Prompter for completion
+
+**Examples:**
+
+Example 1 (python):
+```python
+prompt_strategies.completion.CompletionPromptTokenizingStrategy(
+ *args,
+ max_length=None,
+ **kwargs,
+)
+```
+
+Example 2 (python):
+```python
+prompt_strategies.completion.CompletionPrompter()
+```
+
+---
+
+## utils.collators.core
+
+**URL:** https://docs.axolotl.ai/docs/api/utils.collators.core.html
+
+**Contents:**
+- utils.collators.core
+
+basic shared collator constants
+
+---
+
+## monkeypatch.data.batch_dataset_fetcher
+
+**URL:** https://docs.axolotl.ai/docs/api/monkeypatch.data.batch_dataset_fetcher.html
+
+**Contents:**
+- monkeypatch.data.batch_dataset_fetcher
+- Functions
+ - apply_multipack_dataloader_patch
+ - patch_fetchers
+ - patched_worker_loop
+ - remove_multipack_dataloader_patch
+
+monkeypatch.data.batch_dataset_fetcher
+
+Monkey patches for the dataset fetcher to handle batches of packed indexes.
+
+This patch allows DataLoader to correctly process batches that contain multiple bins of packed sequences.
+
+Apply patches to PyTorch’s DataLoader components.
+
+Worker loop that ensures patches are applied in worker processes.
+
+Remove the monkeypatch and restore original PyTorch DataLoader behavior.
+
+**Examples:**
+
+Example 1 (python):
+```python
+monkeypatch.data.batch_dataset_fetcher.apply_multipack_dataloader_patch()
+```
+
+Example 2 (python):
+```python
+monkeypatch.data.batch_dataset_fetcher.patch_fetchers()
+```
+
+Example 3 (python):
+```python
+monkeypatch.data.batch_dataset_fetcher.patched_worker_loop(*args, **kwargs)
+```
+
+Example 4 (python):
+```python
+monkeypatch.data.batch_dataset_fetcher.remove_multipack_dataloader_patch()
+```
+
+---
+
+## core.datasets.chat
+
+**URL:** https://docs.axolotl.ai/docs/api/core.datasets.chat.html
+
+**Contents:**
+- core.datasets.chat
+- Classes
+ - TokenizedChatDataset
+
+Tokenized chat dataset
+
+**Examples:**
+
+Example 1 (python):
+```python
+core.datasets.chat.TokenizedChatDataset(
+ data,
+ model_transform,
+ *args,
+ message_transform=None,
+ formatter=None,
+ process_count=None,
+ keep_in_memory=False,
+ **kwargs,
+)
+```
+
+---
+
+## utils.freeze
+
+**URL:** https://docs.axolotl.ai/docs/api/utils.freeze.html
+
+**Contents:**
+- utils.freeze
+- Classes
+ - LayerNamePattern
+ - Methods
+ - match
+- Functions
+ - freeze_layers_except
+
+module to freeze/unfreeze parameters by name
+
+Represents a regex pattern for layer names, potentially including a parameter index range.
+
+Checks if the given layer name matches the regex pattern.
+
+Parameters: - name (str): The layer name to check.
+
+Returns: - bool: True if the layer name matches the pattern, False otherwise.
+
+Freezes all layers of the given model except for the layers that match given regex patterns. Periods in the patterns are treated as literal periods, not as wildcard characters.
+
+Parameters: - model (nn.Module): The PyTorch model to be modified. - regex_patterns (list of str): List of regex patterns to match layer names to keep unfrozen. Note that you cannot use a dot as a wildcard character in the patterns since it is reserved for separating layer names. Also, to match the entire layer name, the pattern should start with “^” and end with “\(", otherwise it will match any part of the layer name. The range pattern part is optional and it is not compiled as a regex pattern which means you must put "\)” before the range pattern if you want to match the entire layer name. E.g., [“^model.embed_tokens.weight\([:32000]", "layers.2[0-9]+.block_sparse_moe.gate.[a-z]+\)”]
+
+Returns: None; the model is modified in place.
+
+**Examples:**
+
+Example 1 (python):
+```python
+utils.freeze.LayerNamePattern(pattern)
+```
+
+Example 2 (python):
+```python
+utils.freeze.LayerNamePattern.match(name)
+```
+
+Example 3 (python):
+```python
+utils.freeze.freeze_layers_except(model, regex_patterns)
+```
+
+---
+
+## monkeypatch.unsloth_
+
+**URL:** https://docs.axolotl.ai/docs/api/monkeypatch.unsloth_.html
+
+**Contents:**
+- monkeypatch.unsloth_
+
+module for patching with unsloth optimizations
+
+---
+
+## utils.schemas.datasets
+
+**URL:** https://docs.axolotl.ai/docs/api/utils.schemas.datasets.html
+
+**Contents:**
+- utils.schemas.datasets
+- Classes
+ - DPODataset
+ - KTODataset
+ - PretrainingDataset
+ - SFTDataset
+ - Methods
+ - handle_legacy_message_fields
+ - StepwiseSupervisedDataset
+ - UserDefinedDPOType
+
+utils.schemas.datasets
+
+Pydantic models for datasets-related configuration
+
+DPO configuration subset
+
+KTO configuration subset
+
+Pretraining dataset configuration subset
+
+SFT configuration subset
+
+Handle backwards compatibility between legacy message field mapping and new property mapping system.
+
+Stepwise supervised dataset configuration subset
+
+User defined typing for DPO
+
+User defined typing for KTO
+
+Structure for user defined prompt types
+
+**Examples:**
+
+Example 1 (python):
+```python
+utils.schemas.datasets.DPODataset()
+```
+
+Example 2 (python):
+```python
+utils.schemas.datasets.KTODataset()
+```
+
+Example 3 (python):
+```python
+utils.schemas.datasets.PretrainingDataset()
+```
+
+Example 4 (python):
+```python
+utils.schemas.datasets.SFTDataset()
+```
+
+---
+
+## core.chat.format.llama3x
+
+**URL:** https://docs.axolotl.ai/docs/api/core.chat.format.llama3x.html
+
+**Contents:**
+- core.chat.format.llama3x
+
+core.chat.format.llama3x
+
+Llama 3.x chat formatting functions for MessageContents
+
+---
+
+## datasets
+
+**URL:** https://docs.axolotl.ai/docs/api/datasets.html
+
+**Contents:**
+- datasets
+- Classes
+ - TokenizedPromptDataset
+ - Parameters
+
+Module containing dataset functionality.
+
+We want this to be a wrapper for an existing dataset that we have loaded. Lets use the concept of middlewares to wrap each dataset. We’ll use the collators later on to pad the datasets.
+
+Dataset that returns tokenized prompts from a stream of text files.
+
+**Examples:**
+
+Example 1 (python):
+```python
+datasets.TokenizedPromptDataset(
+ prompt_tokenizer,
+ dataset,
+ process_count=None,
+ keep_in_memory=False,
+ **kwargs,
+)
+```
+
+---
+
+## prompt_strategies.bradley_terry.llama3
+
+**URL:** https://docs.axolotl.ai/docs/api/prompt_strategies.bradley_terry.llama3.html
+
+**Contents:**
+- prompt_strategies.bradley_terry.llama3
+- Functions
+ - icr
+
+prompt_strategies.bradley_terry.llama3
+
+chatml transforms for datasets with system, input, chosen, rejected to match llama3 chat template
+
+chatml transforms for datasets with system, input, chosen, rejected ex. https://huggingface.co/datasets/argilla/distilabel-intel-orca-dpo-pairs
+
+**Examples:**
+
+Example 1 (python):
+```python
+prompt_strategies.bradley_terry.llama3.icr(cfg, **kwargs)
+```
+
+---
+
+## common.datasets
+
+**URL:** https://docs.axolotl.ai/docs/api/common.datasets.html
+
+**Contents:**
+- common.datasets
+- Classes
+ - TrainDatasetMeta
+- Functions
+ - load_datasets
+ - Parameters
+ - Returns
+ - load_preference_datasets
+ - Parameters
+ - Returns
+
+Dataset loading utilities.
+
+Dataclass with fields for training and validation datasets and metadata.
+
+Loads one or more training or evaluation datasets, calling axolotl.utils.data.prepare_datasets. Optionally, logs out debug information.
+
+Loads one or more training or evaluation datasets for RL training using paired preference data, calling axolotl.utils.data.rl.prepare_preference_datasets. Optionally, logs out debug information.
+
+Randomly sample num_samples samples with replacement from dataset.
+
+**Examples:**
+
+Example 1 (python):
+```python
+common.datasets.TrainDatasetMeta(
+ train_dataset,
+ eval_dataset=None,
+ total_num_steps=None,
+)
+```
+
+Example 2 (python):
+```python
+common.datasets.load_datasets(cfg, cli_args=None, debug=False)
+```
+
+Example 3 (python):
+```python
+common.datasets.load_preference_datasets(cfg, cli_args=None)
+```
+
+Example 4 (python):
+```python
+common.datasets.sample_dataset(dataset, num_samples)
+```
+
+---
+
+## cli.train
+
+**URL:** https://docs.axolotl.ai/docs/api/cli.train.html
+
+**Contents:**
+- cli.train
+- Functions
+ - do_cli
+ - Parameters
+ - do_train
+ - Parameters
+
+CLI to run training on a model.
+
+Parses axolotl config, CLI args, and calls do_train.
+
+Trains a transformers model by first loading the dataset(s) specified in the axolotl config, and then calling axolotl.train.train. Also runs the plugin manager’s post_train_unload once training completes.
+
+**Examples:**
+
+Example 1 (python):
+```python
+cli.train.do_cli(config=Path('examples/'), **kwargs)
+```
+
+Example 2 (python):
+```python
+cli.train.do_train(cfg, cli_args)
+```
+
+---
+
+## cli.utils.fetch
+
+**URL:** https://docs.axolotl.ai/docs/api/cli.utils.fetch.html
+
+**Contents:**
+- cli.utils.fetch
+- Functions
+ - fetch_from_github
+ - Parameters
+
+Utilities for axolotl fetch CLI command.
+
+Sync files from a specific directory in the GitHub repository. Only downloads files that don’t exist locally or have changed.
+
+**Examples:**
+
+Example 1 (python):
+```python
+cli.utils.fetch.fetch_from_github(dir_prefix, dest_dir=None, max_workers=5)
+```
+
+---
+
+## utils.tokenization
+
+**URL:** https://docs.axolotl.ai/docs/api/utils.tokenization.html
+
+**Contents:**
+- utils.tokenization
+- Functions
+ - color_token_for_rl_debug
+ - process_tokens_for_rl_debug
+
+Module for tokenization utilities
+
+Helper function to color tokens based on their type.
+
+Helper function to process and color tokens.
+
+**Examples:**
+
+Example 1 (python):
+```python
+utils.tokenization.color_token_for_rl_debug(
+ decoded_token,
+ encoded_token,
+ color,
+ text_only,
+)
+```
+
+Example 2 (python):
+```python
+utils.tokenization.process_tokens_for_rl_debug(
+ tokens,
+ color,
+ tokenizer,
+ text_only,
+)
+```
+
+---
+
+## core.trainers.grpo.sampler
+
+**URL:** https://docs.axolotl.ai/docs/api/core.trainers.grpo.sampler.html
+
+**Contents:**
+- core.trainers.grpo.sampler
+- Classes
+ - SequenceParallelRepeatRandomSampler
+ - Parameters
+ - Methods
+ - set_epoch
+ - Parameters
+
+core.trainers.grpo.sampler
+
+Repeat random sampler (similar to the one implemented in https://github.com/huggingface/trl/blob/main/trl/trainer/grpo_trainer.py) that adds sequence parallelism functionality; i.e., duplicating data across ranks in the same sequence parallel group.
+
+Sampler for GRPO training with sequence parallelism.
+
+This sampler ensures: - Ranks in the same sequence parallel (SP) group receive identical data. - Each index is repeated multiple times for sampling different completions. - Entire batches are repeated for reuse in multiple updates. - Data is properly distributed across SP groups.
+
+In the table below, the values represent dataset indices. Each SP group has context_parallel_size = 2 GPUs working together on the same data. There are 2 SP groups (SP0 and SP1), with world_size = 4 total GPUs.
+
+grad_accum=2 ▲ ▲ 0 0 [0 0 0 1 1 1] [2 2 2 3 3 3] <- SP groups get different data ▼ | 0 1 [0 0 0 1 1 1] [2 2 2 3 3 3] <- Same data for each SP group GPU | | 1 2 [0 0 0 1 1 1] [2 2 2 3 3 3] <- Repeat same indices for iterations num_iterations=2 ▼ 1 3 [0 0 0 1 1 1] [2 2 2 3 3 3] <- When using gradient accumulation
+
+Sets the epoch for this sampler.
+
+**Examples:**
+
+Example 1 (python):
+```python
+core.trainers.grpo.sampler.SequenceParallelRepeatRandomSampler(
+ dataset,
+ mini_repeat_count,
+ world_size,
+ rank,
+ batch_size=1,
+ repeat_count=1,
+ context_parallel_size=1,
+ shuffle=True,
+ seed=0,
+ drop_last=False,
+)
+```
+
+Example 2 (unknown):
+```unknown
+Sequence Parallel Groups
+ | SP0 | SP1 |
+ | GPU 0 | GPU 1 | GPU 2 | GPU 3 |
+ global_step step <---> mini_repeat_count=3
+ <----------> batch_size=2 per SP group
+```
+
+Example 3 (unknown):
+```unknown
+2 4 [4 4 4 5 5 5] [6 6 6 7 7 7] <- New batch of data indices
+ 2 5 [4 4 4 5 5 5] [6 6 6 7 7 7]
+ ...
+```
+
+Example 4 (python):
+```python
+core.trainers.grpo.sampler.SequenceParallelRepeatRandomSampler.set_epoch(epoch)
+```
+
+---
+
+## evaluate
+
+**URL:** https://docs.axolotl.ai/docs/api/evaluate.html
+
+**Contents:**
+- evaluate
+- Functions
+ - evaluate
+ - Parameters
+ - Returns
+ - evaluate_dataset
+ - Parameters
+ - Returns
+
+Module for evaluating models.
+
+Evaluate a model on training and validation datasets.
+
+Helper function to evaluate a single dataset.
+
+**Examples:**
+
+Example 1 (python):
+```python
+evaluate.evaluate(cfg, dataset_meta)
+```
+
+Example 2 (python):
+```python
+evaluate.evaluate_dataset(trainer, dataset, dataset_type, flash_optimum=False)
+```
+
+---
+
+## utils.optimizers.adopt
+
+**URL:** https://docs.axolotl.ai/docs/api/utils.optimizers.adopt.html
+
+**Contents:**
+- utils.optimizers.adopt
+- Functions
+ - adopt
+
+utils.optimizers.adopt
+
+Copied from https://github.com/iShohei220/adopt
+
+ADOPT: Modified Adam Can Converge with Any β2 with the Optimal Rate (2024) Taniguchi, Shohei and Harada, Keno and Minegishi, Gouki and Oshima, Yuta and Jeong, Seong Cheol and Nagahara, Go and Iiyama, Tomoshi and Suzuki, Masahiro and Iwasawa, Yusuke and Matsuo, Yutaka
+
+Functional API that performs ADOPT algorithm computation.
+
+**Examples:**
+
+Example 1 (python):
+```python
+utils.optimizers.adopt.adopt(
+ params,
+ grads,
+ exp_avgs,
+ exp_avg_sqs,
+ state_steps,
+ foreach=None,
+ capturable=False,
+ differentiable=False,
+ fused=None,
+ grad_scale=None,
+ found_inf=None,
+ has_complex=False,
+ *,
+ beta1,
+ beta2,
+ lr,
+ clip_lambda,
+ weight_decay,
+ decouple,
+ eps,
+ maximize,
+)
+```
+
+---
+
+## prompt_tokenizers
+
+**URL:** https://docs.axolotl.ai/docs/api/prompt_tokenizers.html
+
+**Contents:**
+- prompt_tokenizers
+- Classes
+ - AlpacaMultipleChoicePromptTokenizingStrategy
+ - AlpacaPromptTokenizingStrategy
+ - AlpacaReflectionPTStrategy
+ - DatasetWrappingStrategy
+ - GPTeacherPromptTokenizingStrategy
+ - InstructionPromptTokenizingStrategy
+ - InvalidDataException
+ - JeopardyPromptTokenizingStrategy
+
+Module containing PromptTokenizingStrategy and Prompter classes
+
+Tokenizing strategy for Alpaca Multiple Choice prompts.
+
+Tokenizing strategy for Alpaca prompts.
+
+Tokenizing strategy for Alpaca Reflection prompts.
+
+Abstract class for wrapping datasets for Chat Messages
+
+Tokenizing strategy for GPTeacher prompts.
+
+Tokenizing strategy for instruction-based prompts.
+
+Exception raised when the data is invalid
+
+Tokenizing strategy for Jeopardy prompts.
+
+Tokenizing strategy for NomicGPT4All prompts.
+
+Tokenizing strategy for OpenAssistant prompts.
+
+Abstract class for tokenizing strategies
+
+Tokenizing strategy for Reflection prompts.
+
+Tokenizing strategy for SummarizeTLDR prompts.
+
+Parses the tokenized prompt and append the tokenized input_ids, attention_mask and labels to the result
+
+Returns the default values for the tokenize prompt function
+
+**Examples:**
+
+Example 1 (python):
+```python
+prompt_tokenizers.AlpacaMultipleChoicePromptTokenizingStrategy(
+ prompter,
+ tokenizer,
+ train_on_inputs=False,
+ sequence_len=2048,
+)
+```
+
+Example 2 (python):
+```python
+prompt_tokenizers.AlpacaPromptTokenizingStrategy(
+ prompter,
+ tokenizer,
+ train_on_inputs=False,
+ sequence_len=2048,
+)
+```
+
+Example 3 (python):
+```python
+prompt_tokenizers.AlpacaReflectionPTStrategy(
+ prompter,
+ tokenizer,
+ train_on_inputs=False,
+ sequence_len=2048,
+)
+```
+
+Example 4 (python):
+```python
+prompt_tokenizers.DatasetWrappingStrategy()
+```
+
+---
+
+## cli.art
+
+**URL:** https://docs.axolotl.ai/docs/api/cli.art.html
+
+**Contents:**
+- cli.art
+- Functions
+ - print_axolotl_text_art
+
+Axolotl ASCII logo utils.
+
+Prints axolotl ASCII art.
+
+**Examples:**
+
+Example 1 (python):
+```python
+cli.art.print_axolotl_text_art()
+```
+
+---
+
+## utils.callbacks.perplexity
+
+**URL:** https://docs.axolotl.ai/docs/api/utils.callbacks.perplexity.html
+
+**Contents:**
+- utils.callbacks.perplexity
+- Classes
+ - Perplexity
+ - Methods
+ - compute
+
+utils.callbacks.perplexity
+
+callback to calculate perplexity as an evaluation metric.
+
+Calculate perplexity as defined in https://huggingface.co/docs/transformers/en/perplexity. This is a custom variant that doesn’t re-tokenize the input or re-load the model.
+
+Compute perplexity in a fixed length sliding window across the sequence.
+
+**Examples:**
+
+Example 1 (python):
+```python
+utils.callbacks.perplexity.Perplexity(tokenizer, max_seq_len, stride=512)
+```
+
+Example 2 (python):
+```python
+utils.callbacks.perplexity.Perplexity.compute(model, references=None)
+```
+
+---
+
+## cli.utils.train
+
+**URL:** https://docs.axolotl.ai/docs/api/cli.utils.train.html
+
+**Contents:**
+- cli.utils.train
+- Functions
+ - build_command
+ - Parameters
+ - Returns
+ - generate_config_files
+ - Parameters
+ - launch_training
+
+Utilities for axolotl train CLI command.
+
+Build command list from base command and options.
+
+Generate list of configuration files to process. Yields a tuple of the configuration file name and a boolean indicating whether this is a group of configurations (i.e., a sweep).
+
+Execute training with the given configuration.
+
+**Examples:**
+
+Example 1 (python):
+```python
+cli.utils.train.build_command(base_cmd, options)
+```
+
+Example 2 (python):
+```python
+cli.utils.train.generate_config_files(config, sweep)
+```
+
+Example 3 (python):
+```python
+cli.utils.train.launch_training(
+ cfg_file,
+ launcher,
+ cloud,
+ kwargs,
+ launcher_args=None,
+ use_exec=False,
+)
+```
+
+---
+
+## cli.vllm_serve
+
+**URL:** https://docs.axolotl.ai/docs/api/cli.vllm_serve.html
+
+**Contents:**
+- cli.vllm_serve
+- Classes
+ - AxolotlScriptArguments
+- Functions
+ - do_vllm_serve
+ - Returns
+
+CLI to start the vllm server for online RL
+
+Additional arguments for the VLLM server
+
+Starts the VLLM server for serving LLM models used for online RL
+
+Args :param cfg: Parsed doct of the YAML config :param cli_args: dict of additional command-line arguments of type VllmServeCliArgs
+
+**Examples:**
+
+Example 1 (python):
+```python
+cli.vllm_serve.AxolotlScriptArguments(
+ reasoning_parser='',
+ enable_reasoning=None,
+)
+```
+
+Example 2 (python):
+```python
+cli.vllm_serve.do_vllm_serve(config, cli_args)
+```
+
+---
+
+## convert
+
+**URL:** https://docs.axolotl.ai/docs/api/convert.html
+
+**Contents:**
+- convert
+- Classes
+ - FileReader
+ - FileWriter
+ - JsonParser
+ - JsonToJsonlConverter
+ - JsonlSerializer
+ - StdoutWriter
+
+Module containing File Reader, File Writer, Json Parser, and Jsonl Serializer classes
+
+Reads a file and returns its contents as a string
+
+Writes a string to a file
+
+Parses a string as JSON and returns the result
+
+Converts a JSON file to JSONL
+
+Serializes a list of JSON objects into a JSONL string
+
+Writes a string to stdout
+
+**Examples:**
+
+Example 1 (python):
+```python
+convert.FileReader()
+```
+
+Example 2 (python):
+```python
+convert.FileWriter(file_path)
+```
+
+Example 3 (python):
+```python
+convert.JsonParser()
+```
+
+Example 4 (python):
+```python
+convert.JsonToJsonlConverter(
+ file_reader,
+ file_writer,
+ json_parser,
+ jsonl_serializer,
+)
+```
+
+---
+
+## monkeypatch.utils
+
+**URL:** https://docs.axolotl.ai/docs/api/monkeypatch.utils.html
+
+**Contents:**
+- monkeypatch.utils
+- Functions
+ - get_cu_seqlens
+ - get_cu_seqlens_from_pos_ids
+ - mask_2d_to_4d
+
+Shared utils for the monkeypatches
+
+generate a cumulative sequence length mask for flash attention using attn mask
+
+generate a cumulative sequence length mask for flash attention using pos ids
+
+Expands attention_mask from [bsz, seq_len] to [bsz, 1, tgt_seq_len, src_seq_len]. This expansion handles packed sequences so that sequences share the same attention mask integer value when they attend to each other within that sequence. This expansion transforms the mask to lower triangular form to prevent future peeking.
+
+**Examples:**
+
+Example 1 (python):
+```python
+monkeypatch.utils.get_cu_seqlens(attn_mask)
+```
+
+Example 2 (python):
+```python
+monkeypatch.utils.get_cu_seqlens_from_pos_ids(position_ids)
+```
+
+Example 3 (python):
+```python
+monkeypatch.utils.mask_2d_to_4d(mask, dtype, tgt_len=None)
+```
+
+---
+
+## prompt_strategies.pygmalion
+
+**URL:** https://docs.axolotl.ai/docs/api/prompt_strategies.pygmalion.html
+
+**Contents:**
+- prompt_strategies.pygmalion
+- Classes
+ - PygmalionPromptTokenizingStrategy
+ - PygmalionPrompter
+
+prompt_strategies.pygmalion
+
+Module containing the PygmalionPromptTokenizingStrategy and PygmalionPrompter class
+
+Tokenizing strategy for Pygmalion.
+
+Prompter for Pygmalion.
+
+**Examples:**
+
+Example 1 (python):
+```python
+prompt_strategies.pygmalion.PygmalionPromptTokenizingStrategy(
+ prompter,
+ tokenizer,
+ *args,
+ **kwargs,
+)
+```
+
+Example 2 (python):
+```python
+prompt_strategies.pygmalion.PygmalionPrompter(*args, **kwargs)
+```
+
+---
+
+## utils.callbacks.mlflow_
+
+**URL:** https://docs.axolotl.ai/docs/api/utils.callbacks.mlflow_.html
+
+**Contents:**
+- utils.callbacks.mlflow_
+- Classes
+ - SaveAxolotlConfigtoMlflowCallback
+
+utils.callbacks.mlflow_
+
+MLFlow module for trainer callbacks
+
+Callback to save axolotl config to mlflow
+
+**Examples:**
+
+Example 1 (python):
+```python
+utils.callbacks.mlflow_.SaveAxolotlConfigtoMlflowCallback(axolotl_config_path)
+```
+
+---
+
+## loaders.adapter
+
+**URL:** https://docs.axolotl.ai/docs/api/loaders.adapter.html
+
+**Contents:**
+- loaders.adapter
+- Functions
+ - setup_quantized_meta_for_peft
+ - setup_quantized_peft_meta_for_training
+
+Adapter loading functionality, including LoRA / QLoRA and associated utils
+
+Replaces quant_state.to with a dummy function to prevent PEFT from moving quant_state to meta device
+
+Replaces dummy quant_state.to method with the original function to allow training to continue
+
+**Examples:**
+
+Example 1 (python):
+```python
+loaders.adapter.setup_quantized_meta_for_peft(model)
+```
+
+Example 2 (python):
+```python
+loaders.adapter.setup_quantized_peft_meta_for_training(model)
+```
+
+---
+
+## cli.cloud.base
+
+**URL:** https://docs.axolotl.ai/docs/api/cli.cloud.base.html
+
+**Contents:**
+- cli.cloud.base
+- Classes
+ - Cloud
+
+base class for cloud platforms from cli
+
+Abstract base class for cloud platforms.
+
+**Examples:**
+
+Example 1 (python):
+```python
+cli.cloud.base.Cloud()
+```
+
+---
+
+## monkeypatch.llama_attn_hijack_flash
+
+**URL:** https://docs.axolotl.ai/docs/api/monkeypatch.llama_attn_hijack_flash.html
+
+**Contents:**
+- monkeypatch.llama_attn_hijack_flash
+- Functions
+ - flashattn_forward_with_s2attn
+
+monkeypatch.llama_attn_hijack_flash
+
+Flash attention monkey patch for llama model
+
+Input shape: Batch x Time x Channel
+
+From: https://github.com/dvlab-research/LongLoRA/blob/main/llama_attn_replace.py
+
+attention_mask: [bsz, q_len]
+
+cu_seqlens will be ignored if provided max_seqlen will be ignored if provided
+
+**Examples:**
+
+Example 1 (python):
+```python
+monkeypatch.llama_attn_hijack_flash.flashattn_forward_with_s2attn(
+ self,
+ hidden_states,
+ attention_mask=None,
+ position_ids=None,
+ past_key_value=None,
+ output_attentions=False,
+ use_cache=False,
+ padding_mask=None,
+ cu_seqlens=None,
+ max_seqlen=None,
+)
+```
+
+---
+
+## monkeypatch.llama_patch_multipack
+
+**URL:** https://docs.axolotl.ai/docs/api/monkeypatch.llama_patch_multipack.html
+
+**Contents:**
+- monkeypatch.llama_patch_multipack
+
+monkeypatch.llama_patch_multipack
+
+Patched LlamaAttention to use torch.nn.functional.scaled_dot_product_attention
+
+---
+
+## cli.inference
+
+**URL:** https://docs.axolotl.ai/docs/api/cli.inference.html
+
+**Contents:**
+- cli.inference
+- Functions
+ - do_cli
+ - Parameters
+ - do_inference
+ - Parameters
+ - do_inference_gradio
+ - Parameters
+ - get_multi_line_input
+ - Returns
+
+CLI to run inference on a trained model.
+
+Parses axolotl config, CLI args, and calls do_inference or do_inference_gradio.
+
+Runs inference on the command line in a loop. User input is accepted, a chat template is (optionally) applied, and the model specified in the axolotl config is used to generate completions according to a default generation config.
+
+Runs inference in a Gradio interface. User input is accepted, a chat template is (optionally) applied, and the model specified in the axolotl config is used to generate completions according to a default generation config.
+
+Gets multi-line input from terminal.
+
+**Examples:**
+
+Example 1 (python):
+```python
+cli.inference.do_cli(config=Path('examples/'), gradio=False, **kwargs)
+```
+
+Example 2 (python):
+```python
+cli.inference.do_inference(cfg, cli_args)
+```
+
+Example 3 (python):
+```python
+cli.inference.do_inference_gradio(cfg, cli_args)
+```
+
+Example 4 (python):
+```python
+cli.inference.get_multi_line_input()
+```
+
+---
+
+## loaders.tokenizer
+
+**URL:** https://docs.axolotl.ai/docs/api/loaders.tokenizer.html
+
+**Contents:**
+- loaders.tokenizer
+- Functions
+ - load_tokenizer
+ - modify_tokenizer_files
+ - Parameters
+ - Returns
+
+Tokenizer loading functionality and associated utils
+
+Load and configure the tokenizer based on the provided config.
+
+Modify tokenizer files to replace added_tokens strings, save to output directory, and return the path to the modified tokenizer.
+
+This only works with reserved tokens that were added to the tokenizer, not tokens already part of the vocab.
+
+Ref: https://github.com/huggingface/transformers/issues/27974#issuecomment-1854188941
+
+**Examples:**
+
+Example 1 (python):
+```python
+loaders.tokenizer.load_tokenizer(cfg)
+```
+
+Example 2 (python):
+```python
+loaders.tokenizer.modify_tokenizer_files(
+ tokenizer_path,
+ token_mappings,
+ output_dir,
+)
+```
+
+---
+
+## cli.utils.sweeps
+
+**URL:** https://docs.axolotl.ai/docs/api/cli.utils.sweeps.html
+
+**Contents:**
+- cli.utils.sweeps
+- Functions
+ - generate_sweep_configs
+ - Parameters
+ - Returns
+ - Example
+
+Utilities for handling sweeps over configs for axolotl train CLI command
+
+Recursively generates all possible configurations by applying sweeps to the base config.
+
+sweeps_config = { ‘learning_rate’: [0.1, 0.01], ’_’: [ {‘load_in_8bit’: True, ‘adapter’: ‘lora’}, {‘load_in_4bit’: True, ‘adapter’: ‘qlora’} ] }
+
+**Examples:**
+
+Example 1 (python):
+```python
+cli.utils.sweeps.generate_sweep_configs(base_config, sweeps_config)
+```
+
+---
+
+## prompt_strategies.dpo.chatml
+
+**URL:** https://docs.axolotl.ai/docs/api/prompt_strategies.dpo.chatml.html
+
+**Contents:**
+- prompt_strategies.dpo.chatml
+- Functions
+ - argilla_chat
+ - icr
+ - intel
+ - ultra
+
+prompt_strategies.dpo.chatml
+
+DPO strategies for chatml
+
+for argilla/dpo-mix-7k conversations
+
+chatml transforms for datasets with system, input, chosen, rejected ex. https://huggingface.co/datasets/argilla/distilabel-intel-orca-dpo-pairs
+
+For Intel Orca DPO Pairs
+
+for ultrafeedback binarized conversations
+
+**Examples:**
+
+Example 1 (python):
+```python
+prompt_strategies.dpo.chatml.argilla_chat(cfg, **kwargs)
+```
+
+Example 2 (python):
+```python
+prompt_strategies.dpo.chatml.icr(cfg, **kwargs)
+```
+
+Example 3 (python):
+```python
+prompt_strategies.dpo.chatml.intel(cfg, **kwargs)
+```
+
+Example 4 (python):
+```python
+prompt_strategies.dpo.chatml.ultra(cfg, **kwargs)
+```
+
+---
+
+## cli.quantize
+
+**URL:** https://docs.axolotl.ai/docs/api/cli.quantize.html
+
+**Contents:**
+- cli.quantize
+- Functions
+ - do_quantize
+ - Parameters
+
+CLI to post-training quantize a model using torchao
+
+Quantizes a model’s model’s weights
+
+**Examples:**
+
+Example 1 (python):
+```python
+cli.quantize.do_quantize(config, cli_args)
+```
+
+---
+
+## utils.dict
+
+**URL:** https://docs.axolotl.ai/docs/api/utils.dict.html
+
+**Contents:**
+- utils.dict
+- Classes
+ - DictDefault
+- Functions
+ - remove_none_values
+
+Module containing the DictDefault class
+
+A Dict that returns None instead of returning empty Dict for missing keys.
+
+Remove null from a dictionary-like obj or list. These can appear due to Dataset loading causing schema merge. See https://github.com/axolotl-ai-cloud/axolotl/pull/2909
+
+**Examples:**
+
+Example 1 (python):
+```python
+utils.dict.DictDefault()
+```
+
+Example 2 (python):
+```python
+utils.dict.remove_none_values(obj)
+```
+
+---
+
+## API Reference
+
+**URL:** https://docs.axolotl.ai/docs/api/
+
+**Contents:**
+- API Reference
+- Core
+- CLI
+- Trainers
+- Model Loading
+- Mixins
+- Context Managers
+- Prompt Strategies
+- Kernels
+- Monkey Patches
+
+Core functionality for training
+
+Command-line interface
+
+Training implementations
+
+Functionality for loading and patching models, tokenizers, etc.
+
+Mixin classes for augmenting trainers
+
+Context managers for altering trainer behaviors
+
+Prompt formatting strategies
+
+Low-level performance optimizations
+
+Runtime patches for model optimizations
+
+Pydantic data models for Axolotl config
+
+Third-party integrations and extensions
+
+Common utilities and shared functionality
+
+Custom model implementations
+
+Data processing utilities
+
+---
+
+## monkeypatch.lora_kernels
+
+**URL:** https://docs.axolotl.ai/docs/api/monkeypatch.lora_kernels.html
+
+**Contents:**
+- monkeypatch.lora_kernels
+- Classes
+ - FakeMLP
+- Functions
+ - apply_lora_kernel_patches
+ - Parameters
+ - Returns
+ - Raises
+ - Note
+ - get_attention_cls_from_config
+
+monkeypatch.lora_kernels
+
+Module for patching custom LoRA Triton kernels and torch.autograd functions.
+
+placeholder MLP for triton patching
+
+Applies optimized Triton kernel patches to a PEFT model.
+
+Patches a PEFT model with optimized implementations for MLP and attention computations. The optimizations include custom Triton kernels for activation functions and specialized autograd functions for LoRA computations.
+
+The optimizations require LoRA adapters with no dropout and no bias terms. The function will skip patching if these conditions aren’t met.
+
+Get the appropriate attention class by inspecting the model config. Uses dynamic import to support any model architecture that follows the standard transformers naming convention.
+
+Get the layers of the model. Handles text-only and multimodal models.
+
+Original implementation of output projection without optimizations.
+
+Original implementation of QKV projection without optimizations.
+
+Given an axolotl config, this method patches the inferred attention class forward pass with optimized LoRA implementations.
+
+It modifies the attention class to use optimized QKV and output projections. The original implementation is preserved and can be restored if needed.
+
+**Examples:**
+
+Example 1 (python):
+```python
+monkeypatch.lora_kernels.FakeMLP(gate_proj, up_proj, down_proj)
+```
+
+Example 2 (python):
+```python
+monkeypatch.lora_kernels.apply_lora_kernel_patches(model, cfg)
+```
+
+Example 3 (python):
+```python
+monkeypatch.lora_kernels.get_attention_cls_from_config(cfg)
+```
+
+Example 4 (python):
+```python
+monkeypatch.lora_kernels.get_layers(model)
+```
+
+---
+
+## monkeypatch.stablelm_attn_hijack_flash
+
+**URL:** https://docs.axolotl.ai/docs/api/monkeypatch.stablelm_attn_hijack_flash.html
+
+**Contents:**
+- monkeypatch.stablelm_attn_hijack_flash
+- Functions
+ - repeat_kv
+ - rotate_half
+
+monkeypatch.stablelm_attn_hijack_flash
+
+PyTorch StableLM Epoch model.
+
+This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim)
+
+Rotates half the hidden dims of the input.
+
+**Examples:**
+
+Example 1 (python):
+```python
+monkeypatch.stablelm_attn_hijack_flash.repeat_kv(hidden_states, n_rep)
+```
+
+Example 2 (python):
+```python
+monkeypatch.stablelm_attn_hijack_flash.rotate_half(x)
+```
+
+---
+
+## core.trainers.mixins.rng_state_loader
+
+**URL:** https://docs.axolotl.ai/docs/api/core.trainers.mixins.rng_state_loader.html
+
+**Contents:**
+- core.trainers.mixins.rng_state_loader
+- Classes
+ - RngLoaderMixin
+
+core.trainers.mixins.rng_state_loader
+
+Temporary fix/override for bug in resume from checkpoint
+
+See https://github.com/huggingface/transformers/pull/37162
+
+TODO: Remove when upstream added PR to release
+
+mixin for method override to load RNG states from a checkpoint
+
+**Examples:**
+
+Example 1 (python):
+```python
+core.trainers.mixins.rng_state_loader.RngLoaderMixin()
+```
+
+---
+
+## core.trainers.utils
+
+**URL:** https://docs.axolotl.ai/docs/api/core.trainers.utils.html
+
+**Contents:**
+- core.trainers.utils
+
+Utils for Axolotl trainers
+
+---
+
+## core.training_args
+
+**URL:** https://docs.axolotl.ai/docs/api/core.training_args.html
+
+**Contents:**
+- core.training_args
+- Classes
+ - AxolotlCPOConfig
+ - AxolotlKTOConfig
+ - AxolotlORPOConfig
+ - AxolotlPRMConfig
+ - AxolotlRewardConfig
+ - AxolotlTrainingArguments
+
+extra axolotl specific training args
+
+CPO config for CPO training
+
+KTO config for KTO training
+
+ORPO config for ORPO training
+
+PRM config for PRM training
+
+Reward config for Reward training
+
+Training arguments for Causal trainer
+
+This code is duplicated due to HF TrainingArguments not setting output_dir with a default value so it can’t be used as a mixin.
+
+**Examples:**
+
+Example 1 (python):
+```python
+core.training_args.AxolotlCPOConfig(simpo_gamma=None)
+```
+
+Example 2 (python):
+```python
+core.training_args.AxolotlKTOConfig()
+```
+
+Example 3 (python):
+```python
+core.training_args.AxolotlORPOConfig()
+```
+
+Example 4 (python):
+```python
+core.training_args.AxolotlPRMConfig()
+```
+
+---
+
+## monkeypatch.btlm_attn_hijack_flash
+
+**URL:** https://docs.axolotl.ai/docs/api/monkeypatch.btlm_attn_hijack_flash.html
+
+**Contents:**
+- monkeypatch.btlm_attn_hijack_flash
+
+monkeypatch.btlm_attn_hijack_flash
+
+Flash attention monkey patch for cerebras btlm model
+
+---
+
+## prompt_strategies.dpo.passthrough
+
+**URL:** https://docs.axolotl.ai/docs/api/prompt_strategies.dpo.passthrough.html
+
+**Contents:**
+- prompt_strategies.dpo.passthrough
+
+prompt_strategies.dpo.passthrough
+
+DPO prompt strategies passthrough/zero-processing strategy
+
+---
+
+## kernels.swiglu
+
+**URL:** https://docs.axolotl.ai/docs/api/kernels.swiglu.html
+
+**Contents:**
+- kernels.swiglu
+- Functions
+ - swiglu_backward
+ - Parameters
+ - Returns
+ - swiglu_forward
+ - Parameters
+ - Returns
+
+Module for definition of SwiGLU Triton kernels.
+
+See “GLU Variants Improve Transformer” (https://arxiv.org/abs/2002.05202).
+
+Credit to unsloth (https://unsloth.ai/) for inspiration for this implementation.
+
+SwiGLU backward pass using in-place operations.
+
+SwiGLU forward pass. Computes SwiGLU activation: x * sigmoid(x) * up, where x is the gate tensor.
+
+**Examples:**
+
+Example 1 (python):
+```python
+kernels.swiglu.swiglu_backward(grad_output, gate, up)
+```
+
+Example 2 (python):
+```python
+kernels.swiglu.swiglu_forward(gate, up)
+```
+
+---
+
+## core.trainers.grpo.trainer
+
+**URL:** https://docs.axolotl.ai/docs/api/core.trainers.grpo.trainer.html
+
+**Contents:**
+- core.trainers.grpo.trainer
+- Classes
+ - AxolotlGRPOSequenceParallelTrainer
+ - Methods
+ - get_train_dataloader
+ - AxolotlGRPOTrainer
+
+core.trainers.grpo.trainer
+
+Axolotl GRPO trainers (with and without sequence parallelism handling)
+
+Extend the base GRPOTrainer for sequence parallelism handling
+
+Get dataloader for training
+
+Extend the base GRPOTrainer for axolotl helpers
+
+**Examples:**
+
+Example 1 (python):
+```python
+core.trainers.grpo.trainer.AxolotlGRPOSequenceParallelTrainer(
+ model,
+ reward_funcs,
+ args=None,
+ train_dataset=None,
+ eval_dataset=None,
+ processing_class=None,
+ reward_processing_classes=None,
+ callbacks=None,
+ optimizers=(None, None),
+ peft_config=None,
+ optimizer_cls_and_kwargs=None,
+)
+```
+
+Example 2 (python):
+```python
+core.trainers.grpo.trainer.AxolotlGRPOSequenceParallelTrainer.get_train_dataloader(
+)
+```
+
+Example 3 (python):
+```python
+core.trainers.grpo.trainer.AxolotlGRPOTrainer(*args, **kwargs)
+```
+
+---
+
+## prompt_strategies.user_defined
+
+**URL:** https://docs.axolotl.ai/docs/api/prompt_strategies.user_defined.html
+
+**Contents:**
+- prompt_strategies.user_defined
+- Classes
+ - UserDefinedDatasetConfig
+ - UserDefinedPromptTokenizationStrategy
+
+prompt_strategies.user_defined
+
+User Defined prompts with configuration from the YML config
+
+dataclass configuration representing a userdefined dataset type
+
+Prompt Tokenization Strategy for user defined prompts
+
+**Examples:**
+
+Example 1 (python):
+```python
+prompt_strategies.user_defined.UserDefinedDatasetConfig(
+ system_prompt='',
+ field_system='system',
+ field_instruction='instruction',
+ field_input='input',
+ field_output='output',
+ format='{instruction} {input} ',
+ no_input_format='{instruction} ',
+ system_format='{system}',
+)
+```
+
+Example 2 (python):
+```python
+prompt_strategies.user_defined.UserDefinedPromptTokenizationStrategy(
+ prompter,
+ tokenizer,
+ train_on_inputs=False,
+ sequence_len=2048,
+)
+```
+
+---
+
+## utils.schemas.training
+
+**URL:** https://docs.axolotl.ai/docs/api/utils.schemas.training.html
+
+**Contents:**
+- utils.schemas.training
+- Classes
+ - HyperparametersConfig
+ - JaggedLRConfig
+ - LrGroup
+
+utils.schemas.training
+
+Pydantic models for training hyperparameters
+
+Training hyperparams configuration subset
+
+JaggedLR configuration subset, can be used w/ ReLoRA training
+
+Custom learning rate group configuration
+
+**Examples:**
+
+Example 1 (python):
+```python
+utils.schemas.training.HyperparametersConfig()
+```
+
+Example 2 (python):
+```python
+utils.schemas.training.JaggedLRConfig()
+```
+
+Example 3 (python):
+```python
+utils.schemas.training.LrGroup()
+```
+
+---
+
+## utils.quantization
+
+**URL:** https://docs.axolotl.ai/docs/api/utils.quantization.html
+
+**Contents:**
+- utils.quantization
+- Functions
+ - convert_qat_model
+ - get_quantization_config
+ - Parameters
+ - Returns
+ - Raises
+ - prepare_model_for_qat
+ - Parameters
+ - Raises
+
+Utilities for quantization including QAT and PTQ using torchao.
+
+This function converts a QAT model which has fake quantized layers back to the original model.
+
+This function is used to build a post-training quantization config.
+
+This function is used to prepare a model for QAT by swapping the model’s linear layers with fake quantized linear layers, and optionally the embedding weights with fake quantized embedding weights.
+
+This function is used to quantize a model.
+
+**Examples:**
+
+Example 1 (python):
+```python
+utils.quantization.convert_qat_model(model, quantize_embedding=False)
+```
+
+Example 2 (python):
+```python
+utils.quantization.get_quantization_config(
+ weight_dtype,
+ activation_dtype=None,
+ group_size=None,
+)
+```
+
+Example 3 (python):
+```python
+utils.quantization.prepare_model_for_qat(
+ model,
+ weight_dtype,
+ group_size=None,
+ activation_dtype=None,
+ quantize_embedding=False,
+)
+```
+
+Example 4 (python):
+```python
+utils.quantization.quantize_model(
+ model,
+ weight_dtype,
+ group_size=None,
+ activation_dtype=None,
+ quantize_embedding=None,
+)
+```
+
+---
+
+## logging_config
+
+**URL:** https://docs.axolotl.ai/docs/api/logging_config.html
+
+**Contents:**
+- logging_config
+- Classes
+ - AxolotlLogger
+ - AxolotlOrWarnErrorFilter
+ - ColorfulFormatter
+- Functions
+ - configure_logging
+
+Common logging module for axolotl.
+
+Logger that applies filtering to non-axolotl loggers.
+
+Allows ANY WARNING or higher (unless overridden by LOG_LEVEL). Allows axolotl.* at INFO or higher (unless overridden by AXOLOTL_LOG_LEVEL). Drops all other records (i.e. non-axolotl.INFO, DEBUG, etc. by default).
+
+Formatter to add coloring to log messages by log type
+
+Configure with default logging
+
+**Examples:**
+
+Example 1 (python):
+```python
+logging_config.AxolotlLogger(name, level=logging.NOTSET)
+```
+
+Example 2 (python):
+```python
+logging_config.AxolotlOrWarnErrorFilter(**kwargs)
+```
+
+Example 3 (python):
+```python
+logging_config.ColorfulFormatter()
+```
+
+Example 4 (python):
+```python
+logging_config.configure_logging()
+```
+
+---
+
+## prompt_strategies.stepwise_supervised
+
+**URL:** https://docs.axolotl.ai/docs/api/prompt_strategies.stepwise_supervised.html
+
+**Contents:**
+- prompt_strategies.stepwise_supervised
+- Classes
+ - StepwiseSupervisedPromptTokenizingStrategy
+
+prompt_strategies.stepwise_supervised
+
+Module for stepwise datasets, typically including a prompt and reasoning traces, and (optionally) per-step, or per-prompt-trace labels for reward modelling.
+
+Tokenizing strategy for supervised stepwise datasets, typically used for COT-reasoning. These datasets should include the following columns: - prompt: the prompt text - completions: a list of n completion steps - labels: a list of n labels indicating the “correctness” of each step
+
+**Examples:**
+
+Example 1 (python):
+```python
+prompt_strategies.stepwise_supervised.StepwiseSupervisedPromptTokenizingStrategy(
+ tokenizer,
+ sequence_len=2048,
+ step_separator='\n',
+ max_completion_length=None,
+ train_on_last_step_only=False,
+)
+```
+
+---
+
+## utils.schemas.model
+
+**URL:** https://docs.axolotl.ai/docs/api/utils.schemas.model.html
+
+**Contents:**
+- utils.schemas.model
+- Classes
+ - ModelInputConfig
+ - ModelOutputConfig
+ - SpecialTokensConfig
+
+Pydantic models for model input / output, etc. configuration
+
+Model configuration subset
+
+model save configuration subset
+
+Special tokens configuration subset
+
+**Examples:**
+
+Example 1 (python):
+```python
+utils.schemas.model.ModelInputConfig()
+```
+
+Example 2 (python):
+```python
+utils.schemas.model.ModelOutputConfig()
+```
+
+Example 3 (python):
+```python
+utils.schemas.model.SpecialTokensConfig()
+```
+
+---
+
+## utils.schemas.enums
+
+**URL:** https://docs.axolotl.ai/docs/api/utils.schemas.enums.html
+
+**Contents:**
+- utils.schemas.enums
+- Classes
+ - ChatTemplate
+ - CustomSupportedOptimizers
+ - RLType
+ - RingAttnFunc
+
+Enums for Axolotl input config
+
+Chat templates configuration subset
+
+Custom supported optimizers
+
+RL trainer type configuration subset
+
+Enum class for supported ring-flash-attn implementations
+
+**Examples:**
+
+Example 1 (python):
+```python
+utils.schemas.enums.ChatTemplate()
+```
+
+Example 2 (python):
+```python
+utils.schemas.enums.CustomSupportedOptimizers()
+```
+
+Example 3 (python):
+```python
+utils.schemas.enums.RLType()
+```
+
+Example 4 (python):
+```python
+utils.schemas.enums.RingAttnFunc()
+```
+
+---
+
+## core.trainers.trl
+
+**URL:** https://docs.axolotl.ai/docs/api/core.trainers.trl.html
+
+**Contents:**
+- core.trainers.trl
+- Classes
+ - AxolotlCPOTrainer
+ - AxolotlKTOTrainer
+ - AxolotlORPOTrainer
+ - AxolotlPRMTrainer
+ - AxolotlRewardTrainer
+
+Module for TRL RL trainers
+
+Extend the base CPOTrainer for axolotl helpers
+
+Extend the base KTOTrainer for axolotl helpers
+
+Extend the base ORPOTrainer for axolotl helpers
+
+Extend the base trl.PRMTrainer for axolotl helpers
+
+Extend the base RewardTrainer for axolotl helpers
+
+**Examples:**
+
+Example 1 (python):
+```python
+core.trainers.trl.AxolotlCPOTrainer(*args, **kwargs)
+```
+
+Example 2 (python):
+```python
+core.trainers.trl.AxolotlKTOTrainer(*args, **kwargs)
+```
+
+Example 3 (python):
+```python
+core.trainers.trl.AxolotlORPOTrainer(*args, **kwargs)
+```
+
+Example 4 (python):
+```python
+core.trainers.trl.AxolotlPRMTrainer(*args, **kwargs)
+```
+
+---
+
+## utils.schedulers
+
+**URL:** https://docs.axolotl.ai/docs/api/utils.schedulers.html
+
+**Contents:**
+- utils.schedulers
+- Classes
+ - InterpolatingLogScheduler
+ - JaggedLRRestartScheduler
+ - RexLR
+ - Parameters
+- Functions
+ - get_cosine_schedule_with_min_lr
+ - Create a learning rate schedule which has
+ - get_cosine_schedule_with_quadratic_warmup
+
+Module for custom LRScheduler class
+
+A scheduler that interpolates learning rates in a logarithmic fashion
+
+Wraps another scheduler to apply per-lora-restart learning rate warmups.
+
+Reflected Exponential (REX) learning rate scheduler.
+
+Create a schedule with a learning rate that decreases following the values of the cosine function between the initial lr set in the optimizer to 0, after a warmup period during which it increases linearly between 0 and the initial lr set in the optimizer.
+
+torch.optim.lr_scheduler.LambdaLR with the appropriate schedule.
+
+Implementation of Continual Pre-Training of Large Language Models: How to (re)warm your model? (https://arxiv.org/pdf/2308.04014.pdf) Create a schedule with a learning rate that decreases following the values of the cosine function between the initial lr set in the optimizer to min_lr_ratio until num_training_steps * constant_lr_ratio, after constant_rate returns constant value of min_rate , after a warmup period during which it increases linearly between 0 and the initial lr set in the optimizer.
+
+torch.optim.lr_scheduler.LambdaLR with the appropriate schedule.
+
+**Examples:**
+
+Example 1 (python):
+```python
+utils.schedulers.InterpolatingLogScheduler(
+ optimizer,
+ num_steps,
+ min_lr,
+ max_lr,
+ last_epoch=-1,
+)
+```
+
+Example 2 (python):
+```python
+utils.schedulers.JaggedLRRestartScheduler(
+ optimizer,
+ inner_schedule,
+ jagged_restart_steps,
+ jagged_restart_warmup_steps,
+ jagged_restart_anneal_steps=1,
+ min_lr_scale=0.001,
+)
+```
+
+Example 3 (python):
+```python
+utils.schedulers.RexLR(
+ optimizer,
+ max_lr,
+ min_lr,
+ total_steps=0,
+ num_warmup_steps=0,
+ last_step=0,
+)
+```
+
+Example 4 (python):
+```python
+utils.schedulers.get_cosine_schedule_with_min_lr(
+ optimizer,
+ num_warmup_steps,
+ num_training_steps,
+ min_lr_ratio=0.0,
+)
+```
+
+---
+
+## cli.merge_lora
+
+**URL:** https://docs.axolotl.ai/docs/api/cli.merge_lora.html
+
+**Contents:**
+- cli.merge_lora
+- Functions
+ - do_cli
+ - Parameters
+ - Raises
+ - do_merge_lora
+ - Parameters
+
+CLI to merge a trained LoRA into a base model.
+
+Parses axolotl config, CLI args, and calls do_merge_lora. Note that various config values will be overwritten to allow the LoRA merge logic to work as expected (load_in_8bit=False, load_in4bit=False, flash_attention=False, etc.).
+
+Calls transformers’ merge_and_unload on the model given in the axolotl config along with the LoRA adapters to combine them into a single base model.
+
+**Examples:**
+
+Example 1 (python):
+```python
+cli.merge_lora.do_cli(config=Path('examples/'), **kwargs)
+```
+
+Example 2 (python):
+```python
+cli.merge_lora.do_merge_lora(cfg)
+```
+
+---
+
+## prompt_strategies.alpaca_w_system
+
+**URL:** https://docs.axolotl.ai/docs/api/prompt_strategies.alpaca_w_system.html
+
+**Contents:**
+- prompt_strategies.alpaca_w_system
+- Classes
+ - InstructionWSystemPromptTokenizingStrategy
+ - OpenOrcaPromptTokenizingStrategy
+ - OpenOrcaSystemDataPrompter
+ - SystemDataPrompter
+
+prompt_strategies.alpaca_w_system
+
+Prompt strategies loader for alpaca instruction datasets with system prompts
+
+Tokenizing strategy for instruction-based prompts.
+
+Tokenizing strategy for OpenOrca datasets
+
+Alpaca Style Prompter that uses system prompts from the dataset, with OpenOrca prompts
+
+Alpaca Style Prompter that uses system prompts from the dataset
+
+**Examples:**
+
+Example 1 (python):
+```python
+prompt_strategies.alpaca_w_system.InstructionWSystemPromptTokenizingStrategy(
+ prompter,
+ tokenizer,
+ train_on_inputs=False,
+ sequence_len=2048,
+)
+```
+
+Example 2 (python):
+```python
+prompt_strategies.alpaca_w_system.OpenOrcaPromptTokenizingStrategy(
+ prompter,
+ tokenizer,
+ train_on_inputs=False,
+ sequence_len=2048,
+)
+```
+
+Example 3 (python):
+```python
+prompt_strategies.alpaca_w_system.OpenOrcaSystemDataPrompter(
+ prompt_style=PromptStyle.INSTRUCT.value,
+)
+```
+
+Example 4 (python):
+```python
+prompt_strategies.alpaca_w_system.SystemDataPrompter(
+ prompt_style=PromptStyle.INSTRUCT.value,
+)
+```
+
+---
+
+## loaders.patch_manager
+
+**URL:** https://docs.axolotl.ai/docs/api/loaders.patch_manager.html
+
+**Contents:**
+- loaders.patch_manager
+- Classes
+ - PatchManager
+ - Attributes
+ - Methods
+ - apply_post_model_load_patches
+ - apply_post_plugin_pre_model_load_patches
+ - apply_pre_model_load_patches
+
+loaders.patch_manager
+
+Patch manager class implementation to complement axolotl.loaders.ModelLoader.
+
+Applies pre- and post-model load patches for various fixes and optimizations.
+
+Manages the application of patches during the model loading process.
+
+Apply patches that require the model instance.
+
+Apply post plugin-pre_model_load load patches based on config.
+
+Apply pre-model load patches based on config.
+
+**Examples:**
+
+Example 1 (python):
+```python
+loaders.patch_manager.PatchManager(cfg, model_config, inference=False)
+```
+
+Example 2 (python):
+```python
+loaders.patch_manager.PatchManager.apply_post_model_load_patches(model)
+```
+
+Example 3 (python):
+```python
+loaders.patch_manager.PatchManager.apply_post_plugin_pre_model_load_patches()
+```
+
+Example 4 (python):
+```python
+loaders.patch_manager.PatchManager.apply_pre_model_load_patches()
+```
+
+---
+
+## utils.schemas.peft
+
+**URL:** https://docs.axolotl.ai/docs/api/utils.schemas.peft.html
+
+**Contents:**
+- utils.schemas.peft
+- Classes
+ - LoftQConfig
+ - LoraConfig
+ - PeftConfig
+ - ReLoRAConfig
+
+Pydantic models for PEFT-related configuration
+
+LoftQ configuration subset
+
+Peft / LoRA configuration subset
+
+peftq configuration subset
+
+ReLoRA configuration subset
+
+**Examples:**
+
+Example 1 (python):
+```python
+utils.schemas.peft.LoftQConfig()
+```
+
+Example 2 (python):
+```python
+utils.schemas.peft.LoraConfig()
+```
+
+Example 3 (python):
+```python
+utils.schemas.peft.PeftConfig()
+```
+
+Example 4 (python):
+```python
+utils.schemas.peft.ReLoRAConfig()
+```
+
+---
+
+## common.const
+
+**URL:** https://docs.axolotl.ai/docs/api/common.const.html
+
+**Contents:**
+- common.const
+
+Various shared constants
+
+---
+
+## prompt_strategies.kto.user_defined
+
+**URL:** https://docs.axolotl.ai/docs/api/prompt_strategies.kto.user_defined.html
+
+**Contents:**
+- prompt_strategies.kto.user_defined
+
+prompt_strategies.kto.user_defined
+
+User-defined KTO strategies
+
+---
+
+## prompt_strategies.base
+
+**URL:** https://docs.axolotl.ai/docs/api/prompt_strategies.base.html
+
+**Contents:**
+- prompt_strategies.base
+
+prompt_strategies.base
+
+module for base dataset transform strategies
+
+---
+
+## cli.delinearize_llama4
+
+**URL:** https://docs.axolotl.ai/docs/api/cli.delinearize_llama4.html
+
+**Contents:**
+- cli.delinearize_llama4
+- Functions
+ - do_cli
+ - Parameters
+
+cli.delinearize_llama4
+
+CLI tool to delinearize quantized/Linearized Llama-4 models.
+
+Convert a patched HF format Llama4 model (with separated projections) back to the original HF format (with fused projections).
+
+**Examples:**
+
+Example 1 (python):
+```python
+cli.delinearize_llama4.do_cli(model, output)
+```
+
+---
+
+## integrations.base
+
+**URL:** https://docs.axolotl.ai/docs/api/integrations.base.html
+
+**Contents:**
+- integrations.base
+- Classes
+ - BaseOptimizerFactory
+ - Methods
+ - get_decay_parameter_names
+ - BasePlugin
+ - Note
+ - Methods
+ - add_callbacks_post_trainer
+ - Parameters
+
+Base class for all plugins.
+
+A plugin is a reusable, modular, and self-contained piece of code that extends the functionality of Axolotl. Plugins can be used to integrate third-party models, modify the training process, or add new features.
+
+To create a new plugin, you need to inherit from the BasePlugin class and implement the required methods.
+
+Base class for factories to create custom optimizers
+
+Get all parameter names that weight decay will be applied to.
+
+This function filters out parameters in two ways: 1. By layer type (instances of layers specified in ALL_LAYERNORM_LAYERS) 2. By parameter name patterns (containing ‘bias’, or variation of ‘norm’)
+
+Base class for all plugins. Defines the interface for plugin methods.
+
+A plugin is a reusable, modular, and self-contained piece of code that extends the functionality of Axolotl. Plugins can be used to integrate third-party models, modify the training process, or add new features.
+
+To create a new plugin, you need to inherit from the BasePlugin class and implement the required methods.
+
+Plugin methods include: - register(cfg): Registers the plugin with the given configuration. - load_datasets(cfg): Loads and preprocesses the dataset for training. - pre_model_load(cfg): Performs actions before the model is loaded. - post_model_build(cfg, model): Performs actions after the model is loaded, but before LoRA adapters are applied. - pre_lora_load(cfg, model): Performs actions before LoRA weights are loaded. - post_lora_load(cfg, model): Performs actions after LoRA weights are loaded. - post_model_load(cfg, model): Performs actions after the model is loaded, inclusive of any adapters. - post_trainer_create(cfg, trainer): Performs actions after the trainer is created. - create_optimizer(cfg, trainer): Creates and returns an optimizer for training. - create_lr_scheduler(cfg, trainer, optimizer, num_training_steps): Creates and returns a learning rate scheduler. - add_callbacks_pre_trainer(cfg, model): Adds callbacks to the trainer before training. - add_callbacks_post_trainer(cfg, trainer): Adds callbacks to the trainer after training.
+
+Adds callbacks to the trainer after creating the trainer. This is useful for callbacks that require access to the model or trainer.
+
+Set up callbacks before creating the trainer.
+
+Creates and returns a learning rate scheduler.
+
+Creates and returns an optimizer for training.
+
+Returns a custom class for the collator.
+
+Returns a pydantic model for the plugin’s input arguments.
+
+Returns a custom class for the trainer.
+
+Returns custom training arguments to set on TrainingArgs.
+
+Returns a dataclass model for the plugin’s training arguments.
+
+Loads and preprocesses the dataset for training.
+
+Performs actions after LoRA weights are loaded.
+
+Performs actions after the model is built/loaded, but before any adapters are applied.
+
+Performs actions after the model is loaded.
+
+Performs actions after training is complete.
+
+Performs actions after training is complete and the model is unloaded.
+
+Performs actions after the trainer is created.
+
+Performs actions before LoRA weights are loaded.
+
+Performs actions before the model is loaded.
+
+Registers the plugin with the given configuration as an unparsed dict.
+
+The PluginManager class is responsible for loading and managing plugins. It should be a singleton so it can be accessed from anywhere in the codebase.
+
+Key methods include: - get_instance(): Static method to get the singleton instance of PluginManager. - register(plugin_name: str): Registers a new plugin by its name. - pre_model_load(cfg): Calls the pre_model_load method of all registered plugins.
+
+Calls the add_callbacks_post_trainer method of all registered plugins.
+
+Calls the add_callbacks_pre_trainer method of all registered plugins.
+
+Calls the create_lr_scheduler method of all registered plugins and returns the first non-None scheduler.
+
+Calls the create_optimizer method of all registered plugins and returns the first non-None optimizer.
+
+Calls the get_collator_cls_and_kwargs method of all registered plugins and returns the first non-None collator class.
+
+Parameters: cfg (dict): The configuration for the plugins. is_eval (bool): Whether this is an eval split.
+
+Returns: object: The collator class, or None if none was found.
+
+Returns a list of Pydantic classes for all registered plugins’ input arguments.’
+
+Returns the singleton instance of PluginManager. If the instance doesn’t exist, it creates a new one.
+
+Calls the get_trainer_cls method of all registered plugins and returns the first non-None trainer class.
+
+Calls the get_training_args method of all registered plugins and returns the combined training arguments.
+
+Parameters: cfg (dict): The configuration for the plugins.
+
+Returns: object: The training arguments
+
+Returns a list of dataclasses for all registered plugins’ training args mixins’
+
+Returns: list[str]: A list of dataclsses
+
+Calls the load_datasets method of each registered plugin.
+
+Calls the post_lora_load method of all registered plugins.
+
+Calls the post_model_build method of all registered plugins after the model has been built / loaded, but before any adapters have been applied.
+
+Calls the post_model_load method of all registered plugins after the model has been loaded inclusive of any adapters.
+
+Calls the post_train method of all registered plugins.
+
+Calls the post_train_unload method of all registered plugins.
+
+Calls the post_trainer_create method of all registered plugins.
+
+Calls the pre_lora_load method of all registered plugins.
+
+Calls the pre_model_load method of all registered plugins.
+
+Registers a new plugin by its name.
+
+Loads a plugin based on the given plugin name.
+
+The plugin name should be in the format “module_name.class_name”. This function splits the plugin name into module and class, imports the module, retrieves the class from the module, and creates an instance of the class.
+
+**Examples:**
+
+Example 1 (python):
+```python
+integrations.base.BaseOptimizerFactory()
+```
+
+Example 2 (python):
+```python
+integrations.base.BaseOptimizerFactory.get_decay_parameter_names(model)
+```
+
+Example 3 (python):
+```python
+integrations.base.BasePlugin()
+```
+
+Example 4 (python):
+```python
+integrations.base.BasePlugin.add_callbacks_post_trainer(cfg, trainer)
+```
+
+---
+
+## prompt_strategies.chat_template
+
+**URL:** https://docs.axolotl.ai/docs/api/prompt_strategies.chat_template.html
+
+**Contents:**
+- prompt_strategies.chat_template
+- Classes
+ - ChatTemplatePrompter
+ - Methods
+ - build_prompt
+ - Parameters
+ - ChatTemplateStrategy
+ - Methods
+ - find_first_eot_token
+ - find_turn
+
+prompt_strategies.chat_template
+
+HF Chat Templates prompt strategy
+
+Prompter for HF chat templates
+
+Build a prompt from a conversation.
+
+Tokenizing strategy for instruction-based prompts.
+
+Find the first EOT token in the input_ids starting from start_idx.
+
+Locate the starting and ending indices of the specified turn in a conversation.
+
+Public method that can handle either a single prompt or a batch of prompts.
+
+Mistral prompter for chat template.
+
+Mistral strategy for chat template.
+
+Find the first EOT token in the input_ids starting from start_idx.
+
+Load chat template strategy based on configuration.
+
+**Examples:**
+
+Example 1 (python):
+```python
+prompt_strategies.chat_template.ChatTemplatePrompter(
+ tokenizer,
+ chat_template,
+ processor=None,
+ max_length=2048,
+ message_property_mappings=None,
+ message_field_training=None,
+ message_field_training_detail=None,
+ field_messages='messages',
+ field_system='system',
+ field_tools='tools',
+ field_thinking='reasoning_content',
+ roles=None,
+ template_thinking_key='reasoning_content',
+ chat_template_kwargs=None,
+ drop_system_message=False,
+)
+```
+
+Example 2 (python):
+```python
+prompt_strategies.chat_template.ChatTemplatePrompter.build_prompt(
+ conversation,
+ add_generation_prompt=False,
+ images=None,
+ tools=None,
+)
+```
+
+Example 3 (python):
+```python
+prompt_strategies.chat_template.ChatTemplateStrategy(
+ prompter,
+ tokenizer,
+ train_on_inputs,
+ sequence_len,
+ roles_to_train=None,
+ train_on_eos=None,
+ train_on_eot=None,
+ eot_tokens=None,
+ split_thinking=False,
+)
+```
+
+Example 4 (python):
+```python
+prompt_strategies.chat_template.ChatTemplateStrategy.find_first_eot_token(
+ input_ids,
+ start_idx,
+)
+```
+
+---
+
+## kernels.quantize
+
+**URL:** https://docs.axolotl.ai/docs/api/kernels.quantize.html
+
+**Contents:**
+- kernels.quantize
+- Functions
+ - dequantize
+ - Parameters
+ - Returns
+ - Raises
+ - Note
+
+Dequantization utilities for bitsandbytes integration.
+
+Fast NF4 dequantization using bitsandbytes CUDA kernels.
+
+Performs efficient dequantization of weights from NF4 format using bitsandbytes’ optimized CUDA implementations. Supports both legacy list and new QuantState formats.
+
+Uses CUDA streams for better performance when available in newer bitsandbytes versions (>0.43.3).
+
+**Examples:**
+
+Example 1 (python):
+```python
+kernels.quantize.dequantize(W, quant_state=None, out=None)
+```
+
+---
+
+## integrations.spectrum.args
+
+**URL:** https://docs.axolotl.ai/docs/api/integrations.spectrum.args.html
+
+**Contents:**
+- integrations.spectrum.args
+- Classes
+ - SpectrumArgs
+
+integrations.spectrum.args
+
+Module for handling Spectrum input arguments.
+
+Input args for Spectrum.
+
+**Examples:**
+
+Example 1 (python):
+```python
+integrations.spectrum.args.SpectrumArgs()
+```
+
+---
+
+## prompt_strategies.alpaca_chat
+
+**URL:** https://docs.axolotl.ai/docs/api/prompt_strategies.alpaca_chat.html
+
+**Contents:**
+- prompt_strategies.alpaca_chat
+- Classes
+ - AlpacaChatPrompter
+ - AlpacaConcisePrompter
+ - AlpacaQAPromptTokenizingStrategy
+ - CamelAIPromptTokenizingStrategy
+ - NoSystemPrompter
+
+prompt_strategies.alpaca_chat
+
+Module for Alpaca prompt strategy classes
+
+Alpaca Chat Prompter extending the system prompt to for chat-instruct answers
+
+Alpaca Prompter extending the system prompt to ask for concise chat-instruct answers
+
+Tokenizing strategy for AlpacaQA
+
+Tokenizing strategy for CamelAI datasets
+
+Null Prompter with no system prompts
+
+**Examples:**
+
+Example 1 (python):
+```python
+prompt_strategies.alpaca_chat.AlpacaChatPrompter()
+```
+
+Example 2 (python):
+```python
+prompt_strategies.alpaca_chat.AlpacaConcisePrompter(
+ prompt_style=PromptStyle.INSTRUCT.value,
+)
+```
+
+Example 3 (python):
+```python
+prompt_strategies.alpaca_chat.AlpacaQAPromptTokenizingStrategy(
+ prompter,
+ tokenizer,
+ train_on_inputs=False,
+ sequence_len=2048,
+)
+```
+
+Example 4 (python):
+```python
+prompt_strategies.alpaca_chat.CamelAIPromptTokenizingStrategy(
+ prompter,
+ tokenizer,
+ train_on_inputs=False,
+ sequence_len=2048,
+)
+```
+
+---
+
+## utils.collators.mamba
+
+**URL:** https://docs.axolotl.ai/docs/api/utils.collators.mamba.html
+
+**Contents:**
+- utils.collators.mamba
+- Classes
+ - MambaDataCollator
+
+utils.collators.mamba
+
+Collator for State Space Models (Mamba)
+
+**Examples:**
+
+Example 1 (python):
+```python
+utils.collators.mamba.MambaDataCollator(tokenizer)
+```
+
+---
+
+## prompt_strategies.messages.chat
+
+**URL:** https://docs.axolotl.ai/docs/api/prompt_strategies.messages.chat.html
+
+**Contents:**
+- prompt_strategies.messages.chat
+- Classes
+ - ChatMessageDatasetWrappingStrategy
+
+prompt_strategies.messages.chat
+
+Chat dataset wrapping strategy for new internal messages representations
+
+Chat dataset wrapping strategy for new internal messages representations
+
+**Examples:**
+
+Example 1 (python):
+```python
+prompt_strategies.messages.chat.ChatMessageDatasetWrappingStrategy(
+ processor,
+ message_transform=None,
+ formatter=None,
+ **kwargs,
+)
+```
+
+---
+
+## train
+
+**URL:** https://docs.axolotl.ai/docs/api/train.html
+
+**Contents:**
+- train
+- Functions
+ - create_model_card
+ - Parameters
+ - execute_training
+ - Parameters
+ - handle_untrained_tokens_fix
+ - Parameters
+ - save_initial_configs
+ - Parameters
+
+Prepare and train a model on a dataset. Can also infer from a model or merge lora
+
+Create a model card for the trained model if needed.
+
+Execute the training process with appropriate SDP kernel configurations.
+
+Apply fixes for untrained tokens if configured.
+
+Save initial configurations before training.
+
+Save the trained model according to configuration and training setup.
+
+Load the tokenizer, processor (for multimodal models), and model based on configuration.
+
+Load model, tokenizer, trainer, etc. Helper function to encapsulate the full trainer setup.
+
+Set up the Axolotl badge and add the Axolotl config to the model card if available.
+
+Set up the reference model for RL training if needed.
+
+Set up signal handler for graceful termination.
+
+Train a model on the given dataset.
+
+**Examples:**
+
+Example 1 (python):
+```python
+train.create_model_card(cfg, trainer)
+```
+
+Example 2 (python):
+```python
+train.execute_training(cfg, trainer, resume_from_checkpoint)
+```
+
+Example 3 (python):
+```python
+train.handle_untrained_tokens_fix(
+ cfg,
+ model,
+ tokenizer,
+ train_dataset,
+ safe_serialization,
+)
+```
+
+Example 4 (python):
+```python
+train.save_initial_configs(cfg, tokenizer, model, peft_config, processor)
+```
+
+---
+
+## cli.utils.load
+
+**URL:** https://docs.axolotl.ai/docs/api/cli.utils.load.html
+
+**Contents:**
+- cli.utils.load
+- Functions
+ - load_model_and_tokenizer
+ - Parameters
+ - Returns
+
+Utilities for model, tokenizer, etc. loading.
+
+Helper function for loading a model, tokenizer, and processor specified in the given axolotl config.
+
+**Examples:**
+
+Example 1 (python):
+```python
+cli.utils.load.load_model_and_tokenizer(cfg, inference=False)
+```
+
+---
+
+## loaders.model
+
+**URL:** https://docs.axolotl.ai/docs/api/loaders.model.html
+
+**Contents:**
+- loaders.model
+- Classes
+ - ModelLoader
+ - The loading process includes
+ - Attributes
+ - Methods
+ - load
+ - Returns
+
+Model loader class implementation for loading, configuring, and patching various models.
+
+Manages model configuration, initialization and application of patches during model loading.
+
+This class orchestrates the entire process of loading a model from configuration to final preparation. It handles device mapping, quantization, attention mechanisms, adapter integration, and various optimizations.
+
+Load and prepare the model with all configurations and patches.
+
+**Examples:**
+
+Example 1 (python):
+```python
+loaders.model.ModelLoader(
+ cfg,
+ tokenizer,
+ *,
+ inference=False,
+ reference_model=False,
+ **kwargs,
+)
+```
+
+Example 2 (python):
+```python
+loaders.model.ModelLoader.load()
+```
+
+---
+
+## utils.distributed
+
+**URL:** https://docs.axolotl.ai/docs/api/utils.distributed.html
+
+**Contents:**
+- utils.distributed
+- Functions
+ - barrier
+ - cleanup_distributed
+ - compute_and_broadcast
+ - gather_from_all_ranks
+ - gather_scalar_from_all_ranks
+ - is_distributed
+ - is_main_process
+ - Returns
+
+Utilities for distributed functionality.
+
+Acts as a barrier to wait for all processes. This ensures that all processes reach the barrier before proceeding further.
+
+Destroy process group if torch distributed is initialized. Called in training early termination or when training successfully completes.
+
+Compute a value using the function ‘fn’ only on the specified rank (default is 0). The value is then broadcasted to all other ranks.
+
+Args: - fn (callable): A function that computes the value. This should not have any side effects. - rank (int, optional): The rank that computes the value. Default is 0.
+
+Returns: - The computed value (int or float).
+
+Run a callable ‘fn’ on all ranks and gather the results on the specified rank.
+
+Args: - fn (callable): A function that computes the value. This should not have any side effects. - rank (int, optional): The rank that gathers the values. Default is 0. - world_size (int, optional): Total number of processes in the current distributed setup.
+
+Returns: - A list of computed values from all ranks if on the gathering rank, otherwise None.
+
+Run a callable ‘fn’ on all ranks and gather the results on the specified rank.
+
+Args: - fn (callable): A function that computes the value. This should not have any side effects. - rank (int, optional): The rank that gathers the values. Default is 0. - world_size (int, optional): Total number of processes in the current distributed setup.
+
+Returns: - A list of computed values from all ranks if on the gathering rank, otherwise None.
+
+Check if distributed training is initialized.
+
+Check if the current process is the main process. If not in distributed mode, always return True.
+
+We use a simpler logic when the distributed state is not initialized: we just log on the 0-th local rank.
+
+Run a callable ‘fn1’ on all ranks, gather the results, reduce them using ‘fn2’, and then broadcast the reduced result to all ranks.
+
+Args: - fn1 (callable): A function that computes the value on each rank. - fn2 (callable): A reduction function that takes a list of values and returns a single value. - world_size (int, optional): Total number of processes in the current distributed setup.
+
+Returns: - The reduced and broadcasted value.
+
+runs the wrapped context so that rank 0 runs first before other ranks
+
+**Examples:**
+
+Example 1 (python):
+```python
+utils.distributed.barrier()
+```
+
+Example 2 (python):
+```python
+utils.distributed.cleanup_distributed()
+```
+
+Example 3 (python):
+```python
+utils.distributed.compute_and_broadcast(fn)
+```
+
+Example 4 (python):
+```python
+utils.distributed.gather_from_all_ranks(fn, world_size=1)
+```
+
+---
+
+## cli.config
+
+**URL:** https://docs.axolotl.ai/docs/api/cli.config.html
+
+**Contents:**
+- cli.config
+- Functions
+ - check_remote_config
+ - Parameters
+ - Returns
+ - Raises
+ - choose_config
+ - Parameters
+ - Returns
+ - Raises
+
+Configuration loading and processing.
+
+First, determines if the passed config is a valid HTTPS URL. Then, attempts to query for it and parse its content, first as JSON, then as YAML (YAML is preferred). Finally, the parsed content is written to a local file and its path is returned.
+
+Helper method for choosing a axolotl config YAML file (considering only files ending with .yml or .yaml). If more than one config file exists in the passed path, the user is prompted to choose one.
+
+Loads the axolotl configuration stored at config, validates it, and performs various setup.
+
+Registers the plugins for the given configuration.
+
+**Examples:**
+
+Example 1 (python):
+```python
+cli.config.check_remote_config(config)
+```
+
+Example 2 (python):
+```python
+cli.config.choose_config(path)
+```
+
+Example 3 (python):
+```python
+cli.config.load_cfg(config=Path('examples/'), **kwargs)
+```
+
+Example 4 (python):
+```python
+cli.config.prepare_plugins(cfg)
+```
+
+---
+
+## cli.checks
+
+**URL:** https://docs.axolotl.ai/docs/api/cli.checks.html
+
+**Contents:**
+- cli.checks
+- Functions
+ - check_accelerate_default_config
+ - check_user_token
+ - Returns
+ - Raises
+
+Various checks for Axolotl CLI.
+
+Logs at warning level if no accelerate config file is found.
+
+Checks for HF user info. Check is skipped if HF_HUB_OFFLINE=1.
+
+**Examples:**
+
+Example 1 (python):
+```python
+cli.checks.check_accelerate_default_config()
+```
+
+Example 2 (python):
+```python
+cli.checks.check_user_token()
+```
+
+---
+
+## prompt_strategies.llama2_chat
+
+**URL:** https://docs.axolotl.ai/docs/api/prompt_strategies.llama2_chat.html
+
+**Contents:**
+- prompt_strategies.llama2_chat
+- Classes
+ - LLama2ChatTokenizingStrategy
+ - Llama2ChatConversation
+ - Methods
+ - append_message
+ - get_prompt
+ - Llama2ChatPrompter
+
+prompt_strategies.llama2_chat
+
+Prompt Strategy for finetuning Llama2 chat models see also https://github.com/facebookresearch/llama/blob/6c7fe276574e78057f917549435a2554000a876d/llama/generation.py#L213 for ma reference implementation.
+
+This implementation is based on the Vicuna PR and the fastchat repo, see also: https://github.com/lm-sys/FastChat/blob/cdd7730686cb1bf9ae2b768ee171bdf7d1ff04f3/fastchat/conversation.py#L847
+
+Use dataset type: “llama2_chat” in config.yml to use this prompt style.
+
+E.g. in the config.yml:
+
+The dataset itself should look like this:
+
+in a jsonl file. The first message should be from the human, the second from gpt. For a custom system message, the first “from” can be “system” (followed by alternating “human” and “gpt” turns).
+
+Important: Don’t use “special_tokens:” in your config.yml if you are not sure what you are doing!
+
+Tokenizing strategy for Llama2 prompts. adapted from https://github.com/lm-sys/FastChat/blob/main/fastchat/train/train.py
+
+A class that manages prompt templates and keeps all conversation history. copied from https://github.com/lm-sys/FastChat/blob/main/fastchat/conversation.py
+
+Append a new message.
+
+Get the prompt for generation.
+
+A prompter that generates prompts for Llama2 models.
+
+**Examples:**
+
+Example 1 (unknown):
+```unknown
+datasets:
+ - path: llama_finetune_train.jsonl
+ type: llama2_chat
+```
+
+Example 2 (unknown):
+```unknown
+{'conversations':[{"from": "human", "value": "Who are you?"}, {"from": "gpt", "value": "I am Vicuna"},...]}
+```
+
+Example 3 (python):
+```python
+prompt_strategies.llama2_chat.LLama2ChatTokenizingStrategy(*args, **kwargs)
+```
+
+Example 4 (python):
+```python
+prompt_strategies.llama2_chat.Llama2ChatConversation(
+ name='llama2',
+ system="[INST] <>\nYou are a helpful, respectful and honest assistant. Always answer as helpfully as possible, while being safe. Your answers should not include any harmful, unethical, racist, sexist, toxic, dangerous, or illegal content. Please ensure that your responses are socially unbiased and positive in nature.\n\nIf a question does not make any sense, or is not factually coherent, explain why instead of answering something not correct. If you don't know the answer to a question, please don't share false information.\n<>\n\n",
+ roles=('[INST]', '[/INST]'),
+ messages=list(),
+ offset=0,
+)
+```
+
+---
+
+## cli.utils
+
+**URL:** https://docs.axolotl.ai/docs/api/cli.utils.html
+
+**Contents:**
+- cli.utils
+
+Init for axolotl.cli.utils module.
+
+---
+
+## cli.utils.args
+
+**URL:** https://docs.axolotl.ai/docs/api/cli.utils.args.html
+
+**Contents:**
+- cli.utils.args
+- Functions
+ - add_options_from_config
+ - Parameters
+ - Returns
+ - add_options_from_dataclass
+ - Parameters
+ - Returns
+ - filter_none_kwargs
+ - Parameters
+
+Utilities for axolotl CLI args.
+
+Create Click options from the fields of a Pydantic model.
+
+Create Click options from the fields of a dataclass.
+
+Wraps function to remove None-valued kwargs.
+
+**Examples:**
+
+Example 1 (python):
+```python
+cli.utils.args.add_options_from_config(config_class)
+```
+
+Example 2 (python):
+```python
+cli.utils.args.add_options_from_dataclass(config_class)
+```
+
+Example 3 (python):
+```python
+cli.utils.args.filter_none_kwargs(func)
+```
+
+---
+
+## integrations.grokfast.optimizer
+
+**URL:** https://docs.axolotl.ai/docs/api/integrations.grokfast.optimizer.html
+
+**Contents:**
+- integrations.grokfast.optimizer
+
+integrations.grokfast.optimizer
+
+---
+
+## core.builders.causal
+
+**URL:** https://docs.axolotl.ai/docs/api/core.builders.causal.html
+
+**Contents:**
+- core.builders.causal
+- Classes
+ - HFCausalTrainerBuilder
+
+Builder for causal trainers
+
+Build the HuggingFace training args/trainer for causal models and reward modeling using TRL.
+
+**Examples:**
+
+Example 1 (python):
+```python
+core.builders.causal.HFCausalTrainerBuilder(
+ cfg,
+ model,
+ tokenizer,
+ processor=None,
+)
+```
+
+---
+
+## prompt_strategies.dpo.user_defined
+
+**URL:** https://docs.axolotl.ai/docs/api/prompt_strategies.dpo.user_defined.html
+
+**Contents:**
+- prompt_strategies.dpo.user_defined
+
+prompt_strategies.dpo.user_defined
+
+User-defined DPO strategies
+
+---
+
+## cli.evaluate
+
+**URL:** https://docs.axolotl.ai/docs/api/cli.evaluate.html
+
+**Contents:**
+- cli.evaluate
+- Functions
+ - do_cli
+ - Parameters
+ - do_evaluate
+ - Parameters
+
+CLI to run evaluation on a model.
+
+Parses axolotl config, CLI args, and calls do_evaluate.
+
+Evaluates a transformers model by first loading the dataset(s) specified in the axolotl config, and then calling axolotl.evaluate.evaluate, which computes evaluation metrics on the given dataset(s) and writes them to disk.
+
+**Examples:**
+
+Example 1 (python):
+```python
+cli.evaluate.do_cli(config=Path('examples/'), **kwargs)
+```
+
+Example 2 (python):
+```python
+cli.evaluate.do_evaluate(cfg, cli_args)
+```
+
+---
+
+## utils.schemas.utils
+
+**URL:** https://docs.axolotl.ai/docs/api/utils.schemas.utils.html
+
+**Contents:**
+- utils.schemas.utils
+- Functions
+ - handle_legacy_message_fields_logic
+ - Parameters
+ - Returns
+ - Raises
+
+Utilities for Axolotl Pydantic models
+
+Handle backwards compatibility between legacy message field mapping and new property mapping system.
+
+Previously, the config only supported mapping ‘role’ and ‘content’ fields via dedicated config options: - message_field_role: Mapped to the role field - message_field_content: Mapped to the content field
+
+The new system uses message_property_mappings to support arbitrary field mappings: message_property_mappings: role: source_role_field content: source_content_field additional_field: source_field
+
+**Examples:**
+
+Example 1 (python):
+```python
+utils.schemas.utils.handle_legacy_message_fields_logic(data)
+```
+
+---
+
+## prompt_strategies.alpaca_instruct
+
+**URL:** https://docs.axolotl.ai/docs/api/prompt_strategies.alpaca_instruct.html
+
+**Contents:**
+- prompt_strategies.alpaca_instruct
+
+prompt_strategies.alpaca_instruct
+
+Module loading the AlpacaInstructPromptTokenizingStrategy class
+
+---
+
+## utils.callbacks.lisa
+
+**URL:** https://docs.axolotl.ai/docs/api/utils.callbacks.lisa.html
+
+**Contents:**
+- utils.callbacks.lisa
+
+Adapted from https://github.com/OptimalScale/LMFlow/pull/701 for HF transformers & Axolotl Arxiv: https://arxiv.org/abs/2403.17919 License: Apache 2.0
+
+---
+
+## models.mamba.modeling_mamba
+
+**URL:** https://docs.axolotl.ai/docs/api/models.mamba.modeling_mamba.html
+
+**Contents:**
+- models.mamba.modeling_mamba
+
+models.mamba.modeling_mamba
+
+---
+
+## prompt_strategies.metharme
+
+**URL:** https://docs.axolotl.ai/docs/api/prompt_strategies.metharme.html
+
+**Contents:**
+- prompt_strategies.metharme
+- Classes
+ - MetharmePromptTokenizingStrategy
+ - MetharmePrompter
+
+prompt_strategies.metharme
+
+Module containing the MetharmenPromptTokenizingStrategy and MetharmePrompter class
+
+Tokenizing strategy for the Metharme models
+
+Prompter for the Metharme models.
+
+**Examples:**
+
+Example 1 (python):
+```python
+prompt_strategies.metharme.MetharmePromptTokenizingStrategy(
+ prompter,
+ tokenizer,
+ train_on_inputs=False,
+ sequence_len=2048,
+)
+```
+
+Example 2 (python):
+```python
+prompt_strategies.metharme.MetharmePrompter(*args, **kwargs)
+```
+
+---
+
+## core.trainers.mamba
+
+**URL:** https://docs.axolotl.ai/docs/api/core.trainers.mamba.html
+
+**Contents:**
+- core.trainers.mamba
+- Classes
+ - AxolotlMambaTrainer
+
+Module for mamba trainer
+
+Mamba specific trainer to handle loss calculation
+
+**Examples:**
+
+Example 1 (python):
+```python
+core.trainers.mamba.AxolotlMambaTrainer(
+ *_args,
+ bench_data_collator=None,
+ eval_data_collator=None,
+ dataset_tags=None,
+ **kwargs,
+)
+```
+
+---
+
+## utils.ctx_managers.sequence_parallel
+
+**URL:** https://docs.axolotl.ai/docs/api/utils.ctx_managers.sequence_parallel.html
+
+**Contents:**
+- utils.ctx_managers.sequence_parallel
+- Classes
+ - AllGatherWithGrad
+ - Methods
+ - backward
+ - Parameters
+ - Returns
+ - forward
+ - Parameters
+ - Returns
+
+utils.ctx_managers.sequence_parallel
+
+Module for Axolotl trainer sequence parallelism manager and utilities
+
+Custom autograd function for all-gather to preserve gradients.
+
+Backward pass for all-gather operation.
+
+Extracts the gradient slice corresponding to this rank’s original input from the full gradient tensor.
+
+Forward pass of all-gather of data with sequence dimension.
+
+Context manager for sequence parallelism operations.
+
+This class provides a context that will automatically apply sequence parallelism during model forward passes using a pre-forward hook, and gather outputs from across the sequence parallelism group using a post-forward hook.
+
+Apply sequence parallelism slicing to a batch.
+
+Special handling is implemented for integer logits_to_keep, which indicates to only keep the last N tokens in the sequence during generation.
+
+**Examples:**
+
+Example 1 (python):
+```python
+utils.ctx_managers.sequence_parallel.AllGatherWithGrad()
+```
+
+Example 2 (python):
+```python
+utils.ctx_managers.sequence_parallel.AllGatherWithGrad.backward(
+ ctx,
+ grad_output,
+)
+```
+
+Example 3 (python):
+```python
+utils.ctx_managers.sequence_parallel.AllGatherWithGrad.forward(
+ ctx,
+ input_tensor,
+ group,
+)
+```
+
+Example 4 (python):
+```python
+utils.ctx_managers.sequence_parallel.SequenceParallelContextManager(
+ models,
+ context_parallel_size,
+ gradient_accumulation_steps,
+ ring_attn_func,
+ heads_k_stride,
+ gather_outputs,
+ device_mesh=None,
+)
+```
+
+---
+
+## utils.callbacks.qat
+
+**URL:** https://docs.axolotl.ai/docs/api/utils.callbacks.qat.html
+
+**Contents:**
+- utils.callbacks.qat
+- Classes
+ - QATCallback
+- Functions
+ - toggle_fake_quant
+ - Parameters
+
+QAT Callback for HF Causal Trainer
+
+Callback to toggle fake quantization for the model.
+
+Toggle fake quantization for any fake quantized linear or embedding layers in the model.
+
+**Examples:**
+
+Example 1 (python):
+```python
+utils.callbacks.qat.QATCallback(cfg)
+```
+
+Example 2 (python):
+```python
+utils.callbacks.qat.toggle_fake_quant(mod, enable)
+```
+
+---
+
+## prompt_strategies.dpo.zephyr
+
+**URL:** https://docs.axolotl.ai/docs/api/prompt_strategies.dpo.zephyr.html
+
+**Contents:**
+- prompt_strategies.dpo.zephyr
+
+prompt_strategies.dpo.zephyr
+
+DPO strategies for zephyr
+
+---
+
+## kernels.utils
+
+**URL:** https://docs.axolotl.ai/docs/api/kernels.utils.html
+
+**Contents:**
+- kernels.utils
+
+Utilities for axolotl.kernels submodules.
+
+---
+
+## monkeypatch.multipack
+
+**URL:** https://docs.axolotl.ai/docs/api/monkeypatch.multipack.html
+
+**Contents:**
+- monkeypatch.multipack
+
+monkeypatch.multipack
+
+multipack patching for v2 of sample packing
+
+---
+
+## cli.main
+
+**URL:** https://docs.axolotl.ai/docs/api/cli.main.html
+
+**Contents:**
+- cli.main
+- Functions
+ - cli
+ - evaluate
+ - Parameters
+ - fetch
+ - Parameters
+ - inference
+ - Parameters
+ - merge_lora
+
+Click CLI definitions for various axolotl commands.
+
+Axolotl CLI - Train and fine-tune large language models
+
+Fetch example configs or other resources.
+
+Available directories: - examples: Example configuration files - deepspeed_configs: DeepSpeed configuration files
+
+Run inference with a trained model.
+
+Merge trained LoRA adapters into a base model.
+
+Merge sharded FSDP model weights.
+
+Preprocess datasets before training.
+
+Train or fine-tune a model.
+
+**Examples:**
+
+Example 1 (python):
+```python
+cli.main.cli()
+```
+
+Example 2 (python):
+```python
+cli.main.evaluate(ctx, config, launcher, **kwargs)
+```
+
+Example 3 (python):
+```python
+cli.main.fetch(directory, dest)
+```
+
+Example 4 (python):
+```python
+cli.main.inference(ctx, config, launcher, gradio, **kwargs)
+```
+
+---
+
+## core.trainers.mixins.optimizer
+
+**URL:** https://docs.axolotl.ai/docs/api/core.trainers.mixins.optimizer.html
+
+**Contents:**
+- core.trainers.mixins.optimizer
+- Classes
+ - OptimizerInitMixin
+ - OptimizerMixin
+
+core.trainers.mixins.optimizer
+
+Module for Axolotl trainer optimizer mixin
+
+Mixin to handle common optimizer initialization logic for Trainers (mostly TRL) that do not accept optimizer_cls_and_kwargs as kwarg in constructor.
+
+Mixin class for shared handling of building custom optimizers
+
+**Examples:**
+
+Example 1 (python):
+```python
+core.trainers.mixins.optimizer.OptimizerInitMixin(*args, **kwargs)
+```
+
+Example 2 (python):
+```python
+core.trainers.mixins.optimizer.OptimizerMixin()
+```
+
+---
+
+## integrations.kd.trainer
+
+**URL:** https://docs.axolotl.ai/docs/api/integrations.kd.trainer.html
+
+**Contents:**
+- integrations.kd.trainer
+- Classes
+ - AxolotlKDTrainer
+ - Methods
+ - compute_loss
+
+integrations.kd.trainer
+
+Custom trainer subclass for Knowledge Distillation (KD)
+
+How the loss is computed by Trainer. By default, all models return the loss in the first element.
+
+Subclass and override for custom behavior.
+
+**Examples:**
+
+Example 1 (python):
+```python
+integrations.kd.trainer.AxolotlKDTrainer(*args, **kwargs)
+```
+
+Example 2 (python):
+```python
+integrations.kd.trainer.AxolotlKDTrainer.compute_loss(
+ model,
+ inputs,
+ return_outputs=False,
+ num_items_in_batch=None,
+)
+```
+
+---
+
+## integrations.lm_eval.args
+
+**URL:** https://docs.axolotl.ai/docs/api/integrations.lm_eval.args.html
+
+**Contents:**
+- integrations.lm_eval.args
+- Classes
+ - LMEvalArgs
+
+integrations.lm_eval.args
+
+Module for handling lm eval harness input arguments.
+
+Input args for lm eval harness
+
+**Examples:**
+
+Example 1 (python):
+```python
+integrations.lm_eval.args.LMEvalArgs()
+```
+
+---
+
+## integrations.cut_cross_entropy.args
+
+**URL:** https://docs.axolotl.ai/docs/api/integrations.cut_cross_entropy.args.html
+
+**Contents:**
+- integrations.cut_cross_entropy.args
+- Classes
+ - CutCrossEntropyArgs
+
+integrations.cut_cross_entropy.args
+
+Module for handling Cut Cross Entropy input arguments.
+
+Input args for Cut Cross Entropy.
+
+**Examples:**
+
+Example 1 (python):
+```python
+integrations.cut_cross_entropy.args.CutCrossEntropyArgs()
+```
+
+---
+
+## monkeypatch.mistral_attn_hijack_flash
+
+**URL:** https://docs.axolotl.ai/docs/api/monkeypatch.mistral_attn_hijack_flash.html
+
+**Contents:**
+- monkeypatch.mistral_attn_hijack_flash
+
+monkeypatch.mistral_attn_hijack_flash
+
+Flash attention monkey patch for mistral model
+
+---
+
+## loaders.constants
+
+**URL:** https://docs.axolotl.ai/docs/api/loaders.constants.html
+
+**Contents:**
+- loaders.constants
+
+Shared constants for axolotl.loaders module
+
+---
+
+## utils.bench
+
+**URL:** https://docs.axolotl.ai/docs/api/utils.bench.html
+
+**Contents:**
+- utils.bench
+- Functions
+ - check_cuda_device
+
+Benchmarking and measurement utilities
+
+wraps a function and returns the default value instead of running the wrapped function if cuda isn’t available or the device is auto :param default_value: :return:
+
+**Examples:**
+
+Example 1 (python):
+```python
+utils.bench.check_cuda_device(default_value)
+```
+
+---
+
+## utils.trainer
+
+**URL:** https://docs.axolotl.ai/docs/api/utils.trainer.html
+
+**Contents:**
+- utils.trainer
+- Functions
+ - add_pose_position_ids
+ - add_position_ids
+ - drop_long_seq
+ - setup_trainer
+ - Parameters
+ - Returns
+
+Module containing the Trainer class and related functions
+
+use the PoSE technique to extend the context length by randomly skipping positions in the context. We only want to skip right before tokens in the split_on_token_ids list. We should attempt to randomly distribute the skips, but we don’t need the final position_ids to be the full context_len. There may be multiple turns in the context, so we want to make sure we take into account the maximum possible number of skips remaining in each sample.
+
+Handle both single-example and batched data. - single example: sample[‘input_ids’] is a list[int] - batched data: sample[‘input_ids’] is a list[list[int]]
+
+Drop samples whose sequence length is either too long (> sequence_len) or too short (< min_sequence_len).
+
+Works for both single-example (list[int]) or batched (list[list[int]]).
+
+Helper method for instantiating and building a (causal or RLHF) trainer.
+
+**Examples:**
+
+Example 1 (python):
+```python
+utils.trainer.add_pose_position_ids(
+ sample,
+ max_context_len=32768,
+ split_on_token_ids=None,
+ chunks=2,
+)
+```
+
+Example 2 (python):
+```python
+utils.trainer.add_position_ids(sample)
+```
+
+Example 3 (python):
+```python
+utils.trainer.drop_long_seq(sample, sequence_len=2048, min_sequence_len=2)
+```
+
+Example 4 (python):
+```python
+utils.trainer.setup_trainer(
+ cfg,
+ train_dataset,
+ eval_dataset,
+ model,
+ tokenizer,
+ processor,
+ total_num_steps,
+ model_ref=None,
+ peft_config=None,
+)
+```
+
+---
+
+## utils.schemas.config
+
+**URL:** https://docs.axolotl.ai/docs/api/utils.schemas.config.html
+
+**Contents:**
+- utils.schemas.config
+- Classes
+ - AxolotlConfigWCapabilities
+ - AxolotlInputConfig
+
+Module with Pydantic models for configuration.
+
+wrapper to valdiate GPU capabilities with the configured options
+
+Wrapper of all config options.
+
+**Examples:**
+
+Example 1 (python):
+```python
+utils.schemas.config.AxolotlConfigWCapabilities()
+```
+
+Example 2 (python):
+```python
+utils.schemas.config.AxolotlInputConfig()
+```
+
+---
+
+## cli.args
+
+**URL:** https://docs.axolotl.ai/docs/api/cli.args.html
+
+**Contents:**
+- cli.args
+- Classes
+ - EvaluateCliArgs
+ - InferenceCliArgs
+ - PreprocessCliArgs
+ - QuantizeCliArgs
+ - TrainerCliArgs
+ - VllmServeCliArgs
+
+Module for axolotl CLI command arguments.
+
+Dataclass with CLI arguments for axolotl evaluate command.
+
+Dataclass with CLI arguments for axolotl inference command.
+
+Dataclass with CLI arguments for axolotl preprocess command.
+
+Dataclass with CLI arguments for axolotl quantize command.
+
+Dataclass with CLI arguments for axolotl train command.
+
+Dataclass with CLI arguments for axolotl vllm-serve command.
+
+**Examples:**
+
+Example 1 (python):
+```python
+cli.args.EvaluateCliArgs(
+ debug=False,
+ debug_text_only=False,
+ debug_num_examples=0,
+)
+```
+
+Example 2 (python):
+```python
+cli.args.InferenceCliArgs(prompter=None)
+```
+
+Example 3 (python):
+```python
+cli.args.PreprocessCliArgs(
+ debug=False,
+ debug_text_only=False,
+ debug_num_examples=1,
+ prompter=None,
+ download=True,
+ iterable=False,
+)
+```
+
+Example 4 (python):
+```python
+cli.args.QuantizeCliArgs(
+ base_model=None,
+ weight_dtype=None,
+ activation_dtype=None,
+ quantize_embedding=None,
+ group_size=None,
+ output_dir=None,
+ hub_model_id=None,
+)
+```
+
+---
+
+## common.architectures
+
+**URL:** https://docs.axolotl.ai/docs/api/common.architectures.html
+
+**Contents:**
+- common.architectures
+
+Common architecture specific constants
+
+---
+
+## cli.merge_sharded_fsdp_weights
+
+**URL:** https://docs.axolotl.ai/docs/api/cli.merge_sharded_fsdp_weights.html
+
+**Contents:**
+- cli.merge_sharded_fsdp_weights
+- Classes
+ - BFloat16CastPlanner
+- Functions
+ - do_cli
+ - Parameters
+ - merge_fsdp_weights
+ - Parameters
+ - Raises
+
+cli.merge_sharded_fsdp_weights
+
+CLI to merge sharded FSDP model checkpoints into a single combined checkpoint.
+
+A custom planner to cast tensors to bfloat16 on the fly during loading.
+
+Parses axolotl config, CLI args, and calls merge_fsdp_weights.
+
+Merge the weights from sharded FSDP model checkpoints into a single combined checkpoint. Should be used if SHARDED_STATE_DICT was used for the model. Weights will be saved to {output_path}/model.safetensors if safe_serialization else pytorch_model.bin.
+
+Note: this is a CPU-bound process.
+
+**Examples:**
+
+Example 1 (python):
+```python
+cli.merge_sharded_fsdp_weights.BFloat16CastPlanner()
+```
+
+Example 2 (python):
+```python
+cli.merge_sharded_fsdp_weights.do_cli(config=Path('examples/'), **kwargs)
+```
+
+Example 3 (python):
+```python
+cli.merge_sharded_fsdp_weights.merge_fsdp_weights(
+ checkpoint_dir,
+ output_path,
+ safe_serialization=False,
+ remove_checkpoint_dir=False,
+)
+```
+
+---
+
+## utils.data.streaming
+
+**URL:** https://docs.axolotl.ai/docs/api/utils.data.streaming.html
+
+**Contents:**
+- utils.data.streaming
+
+Data handling specific to streaming datasets.
+
+---
+
+## core.chat.format.chatml
+
+**URL:** https://docs.axolotl.ai/docs/api/core.chat.format.chatml.html
+
+**Contents:**
+- core.chat.format.chatml
+
+core.chat.format.chatml
+
+ChatML transformation functions for MessageContents
+
+---
+
+## prompt_strategies.kto.chatml
+
+**URL:** https://docs.axolotl.ai/docs/api/prompt_strategies.kto.chatml.html
+
+**Contents:**
+- prompt_strategies.kto.chatml
+- Functions
+ - argilla_chat
+ - intel
+ - ultra
+
+prompt_strategies.kto.chatml
+
+KTO strategies for chatml
+
+for argilla/kto-mix-15k conversations
+
+For Intel Orca KTO ex: argilla/distilabel-intel-orca-kto
+
+for ultrafeedback binarized conversations ex: argilla/ultrafeedback-binarized-preferences-cleaned-kto
+
+**Examples:**
+
+Example 1 (python):
+```python
+prompt_strategies.kto.chatml.argilla_chat(cfg, **kwargs)
+```
+
+Example 2 (python):
+```python
+prompt_strategies.kto.chatml.intel(cfg, **kwargs)
+```
+
+Example 3 (python):
+```python
+prompt_strategies.kto.chatml.ultra(cfg, **kwargs)
+```
+
+---
+
+## utils.schemas.trl
+
+**URL:** https://docs.axolotl.ai/docs/api/utils.schemas.trl.html
+
+**Contents:**
+- utils.schemas.trl
+- Classes
+ - TRLConfig
+
+Pydantic models for TRL trainer configuration
+
+**Examples:**
+
+Example 1 (python):
+```python
+utils.schemas.trl.TRLConfig()
+```
+
+---
+
+## monkeypatch.llama_attn_hijack_xformers
+
+**URL:** https://docs.axolotl.ai/docs/api/monkeypatch.llama_attn_hijack_xformers.html
+
+**Contents:**
+- monkeypatch.llama_attn_hijack_xformers
+
+monkeypatch.llama_attn_hijack_xformers
+
+Directly copied the code from https://raw.githubusercontent.com/oobabooga/text-generation-webui/main/modules/llama_attn_hijack.py and made some adjustments
+
+---
+
+## kernels.geglu
+
+**URL:** https://docs.axolotl.ai/docs/api/kernels.geglu.html
+
+**Contents:**
+- kernels.geglu
+- Functions
+ - geglu_backward
+ - Parameters
+ - Returns
+ - Note
+ - geglu_forward
+ - Parameters
+ - Returns
+
+Module for definition of GEGLU Triton kernels.
+
+See “GLU Variants Improve Transformer” (https://arxiv.org/abs/2002.05202).
+
+Credit to unsloth (https://unsloth.ai/) for inspiration for this implementation.
+
+GEGLU backward pass using in-place operations.
+
+This function modifies its input tensors in-place to store results.
+
+**Examples:**
+
+Example 1 (python):
+```python
+kernels.geglu.geglu_backward(grad_output, gate, up)
+```
+
+Example 2 (python):
+```python
+kernels.geglu.geglu_forward(gate, up)
+```
+
+---
+
+## utils.callbacks.profiler
+
+**URL:** https://docs.axolotl.ai/docs/api/utils.callbacks.profiler.html
+
+**Contents:**
+- utils.callbacks.profiler
+- Classes
+ - PytorchProfilerCallback
+
+utils.callbacks.profiler
+
+HF Trainer callback for creating pytorch profiling snapshots
+
+PyTorch Profiler callback to create snapshots of GPU memory usage at specified steps.
+
+**Examples:**
+
+Example 1 (python):
+```python
+utils.callbacks.profiler.PytorchProfilerCallback(
+ steps_to_profile=5,
+ profiler_steps_start=0,
+)
+```
+
+---
+
+## kernels.lora
+
+**URL:** https://docs.axolotl.ai/docs/api/kernels.lora.html
+
+**Contents:**
+- kernels.lora
+- Classes
+ - LoRA_MLP
+ - Methods
+ - backward
+ - Parameters
+ - Returns
+ - forward
+ - Parameters
+ - Returns
+
+Module for definition of Low-Rank Adaptation (LoRA) Triton kernels.
+
+See “LoRA: Low-Rank Adaptation of Large Language Models” (https://arxiv.org/abs/2106.09685).
+
+Credit to unsloth (https://unsloth.ai/) for inspiration for this implementation.
+
+Optimized LoRA MLP implementation.
+
+Performs backward pass computation for LoRA MLP.
+
+Forward pass for LoRA MLP.
+
+Optimized LoRA implementation for output projection.
+
+Backward pass computing gradients for LoRA output projection.
+
+Forward pass for output projection with LoRA.
+
+Optimized LoRA QKV implementation with quantization support.
+
+Implements efficient computation of query, key, value projections with LoRA, supporting quantization and memory optimization.
+
+Backward pass computing gradients for LoRA QKV.
+
+Forward pass computing Q, K, V projections with LoRA.
+
+Applies LoRA to MLP layer with GEGLU activation.
+
+Applies LoRA to MLP layer with SwiGLU activation.
+
+Applies LoRA to output projection layer.
+
+Applies LoRA to compute Query, Key, Value projections.
+
+Gets LoRA parameters from a projection module.
+
+Efficient fused matmul + LoRA computation.
+
+**Examples:**
+
+Example 1 (python):
+```python
+kernels.lora.LoRA_MLP()
+```
+
+Example 2 (python):
+```python
+kernels.lora.LoRA_MLP.backward(ctx, grad_output)
+```
+
+Example 3 (python):
+```python
+kernels.lora.LoRA_MLP.forward(
+ ctx,
+ X,
+ gate_weight,
+ gate_bias,
+ gate_quant,
+ gate_A,
+ gate_B,
+ gate_scale,
+ up_weight,
+ up_bias,
+ up_quant,
+ up_A,
+ up_B,
+ up_scale,
+ down_weight,
+ down_bias,
+ down_quant,
+ down_A,
+ down_B,
+ down_scale,
+ activation_fn,
+ activation_fn_backward,
+ inplace=True,
+)
+```
+
+Example 4 (python):
+```python
+kernels.lora.LoRA_O()
+```
+
+---
+
+## monkeypatch.trainer_fsdp_optim
+
+**URL:** https://docs.axolotl.ai/docs/api/monkeypatch.trainer_fsdp_optim.html
+
+**Contents:**
+- monkeypatch.trainer_fsdp_optim
+- Functions
+ - patch_training_loop_for_fsdp
+
+monkeypatch.trainer_fsdp_optim
+
+fix for FSDP optimizer save in trainer w 4.47.0
+
+monkeypatch for fixing the training loop for fsdp with optimizer save
+
+**Examples:**
+
+Example 1 (python):
+```python
+monkeypatch.trainer_fsdp_optim.patch_training_loop_for_fsdp()
+```
+
+---
+
+## utils.schemas.multimodal
+
+**URL:** https://docs.axolotl.ai/docs/api/utils.schemas.multimodal.html
+
+**Contents:**
+- utils.schemas.multimodal
+- Classes
+ - MultiModalConfig
+ - Methods
+ - convert_image_resize_algorithm
+
+utils.schemas.multimodal
+
+Pydantic models for multimodal-related configuration
+
+Multi-modal configuration subset
+
+Convert the image resize algorithm to a PIL.Image.Resampling enum.
+
+**Examples:**
+
+Example 1 (python):
+```python
+utils.schemas.multimodal.MultiModalConfig()
+```
+
+Example 2 (python):
+```python
+utils.schemas.multimodal.MultiModalConfig.convert_image_resize_algorithm(
+ image_resize_algorithm,
+)
+```
+
+---
+
+## prompt_strategies.dpo.llama3
+
+**URL:** https://docs.axolotl.ai/docs/api/prompt_strategies.dpo.llama3.html
+
+**Contents:**
+- prompt_strategies.dpo.llama3
+- Functions
+ - argilla_chat
+ - icr
+ - intel
+ - ultra
+
+prompt_strategies.dpo.llama3
+
+DPO strategies for llama-3 chat template
+
+for argilla/dpo-mix-7k conversations
+
+chatml transforms for datasets with system, input, chosen, rejected ex. https://huggingface.co/datasets/argilla/distilabel-intel-orca-dpo-pairs
+
+For Intel Orca DPO Pairs
+
+for ultrafeedback binarized conversations
+
+**Examples:**
+
+Example 1 (python):
+```python
+prompt_strategies.dpo.llama3.argilla_chat(cfg, **kwargs)
+```
+
+Example 2 (python):
+```python
+prompt_strategies.dpo.llama3.icr(cfg, **kwargs)
+```
+
+Example 3 (python):
+```python
+prompt_strategies.dpo.llama3.intel(cfg, **kwargs)
+```
+
+Example 4 (python):
+```python
+prompt_strategies.dpo.llama3.ultra(cfg, **kwargs)
+```
+
+---
+
+## core.chat.format.shared
+
+**URL:** https://docs.axolotl.ai/docs/api/core.chat.format.shared.html
+
+**Contents:**
+- core.chat.format.shared
+
+core.chat.format.shared
+
+shared functions for format transforms
+
+---
+
+## monkeypatch.llama_expand_mask
+
+**URL:** https://docs.axolotl.ai/docs/api/monkeypatch.llama_expand_mask.html
+
+**Contents:**
+- monkeypatch.llama_expand_mask
+
+monkeypatch.llama_expand_mask
+
+expands the binary attention mask per 3.2.2 of https://arxiv.org/pdf/2107.02027.pdf
+
+---
+
+## core.chat.messages
+
+**URL:** https://docs.axolotl.ai/docs/api/core.chat.messages.html
+
+**Contents:**
+- core.chat.messages
+- Classes
+ - ChatFormattedChats
+ - Chats
+ - MessageContentTypes
+ - MessageContents
+ - MessageRoles
+ - Messages
+ - PreferenceChats
+ - SpecialToken
+
+internal message representations of chat messages
+
+Chat formatted chats with formatter and optional train on inputs
+
+top level data structure for chat conversations
+
+Message content types for text, image, audio, tool calls, and tool responses
+
+Message contents with type, value, metadata, weight, newline, and end of contents
+
+Message roles for the system, user, assistant, and tools
+
+Messages with role, content, metadata, weight, and chat formatting
+
+representation for preference data for chat
+
+Special tokens for beginning of string and end of string
+
+Tool with description, function, and parameters
+
+Tool call contents with name, arguments, and optional id
+
+Tool call function with name and arguments
+
+Tool response contents with name, content, and optional id
+
+**Examples:**
+
+Example 1 (python):
+```python
+core.chat.messages.ChatFormattedChats()
+```
+
+Example 2 (python):
+```python
+core.chat.messages.Chats()
+```
+
+Example 3 (python):
+```python
+core.chat.messages.MessageContentTypes()
+```
+
+Example 4 (python):
+```python
+core.chat.messages.MessageContents()
+```
+
+---
+
+## core.datasets.transforms.chat_builder
+
+**URL:** https://docs.axolotl.ai/docs/api/core.datasets.transforms.chat_builder.html
+
+**Contents:**
+- core.datasets.transforms.chat_builder
+- Functions
+ - chat_message_transform_builder
+ - Parameters
+ - Returns
+
+core.datasets.transforms.chat_builder
+
+This module contains a function that builds a transform that takes a row from the dataset and converts it to a Chat.
+
+Builds a transform that takes a row from the dataset and converts it to a Chat
+
+**Examples:**
+
+Example 1 (python):
+```python
+core.datasets.transforms.chat_builder.chat_message_transform_builder(
+ train_on_inputs=False,
+ conversations_field='messages',
+ message_field_role=None,
+ message_field_content=None,
+ message_field_training=None,
+)
+```
+
+---
+
+## utils.chat_templates
+
+**URL:** https://docs.axolotl.ai/docs/api/utils.chat_templates.html
+
+**Contents:**
+- utils.chat_templates
+
+This module provides functionality for selecting chat templates based on user choices. These templates are used for formatting messages in a conversation.
+
+---
+
+## core.trainers.dpo.trainer
+
+**URL:** https://docs.axolotl.ai/docs/api/core.trainers.dpo.trainer.html
+
+**Contents:**
+- core.trainers.dpo.trainer
+- Classes
+ - AxolotlDPOTrainer
+ - Methods
+ - push_to_hub
+
+core.trainers.dpo.trainer
+
+DPO trainer for axolotl
+
+Extend the base DPOTrainer for axolotl helpers.
+
+Overwrite the push_to_hub method in order to force-add the tags when pushing the model on the Hub. Please refer to ~transformers.Trainer.push_to_hub for more details.
+
+**Examples:**
+
+Example 1 (python):
+```python
+core.trainers.dpo.trainer.AxolotlDPOTrainer(*args, dataset_tags=None, **kwargs)
+```
+
+Example 2 (python):
+```python
+core.trainers.dpo.trainer.AxolotlDPOTrainer.push_to_hub(*args, **kwargs)
+```
+
+---
+
+## monkeypatch.gradient_checkpointing.offload_disk
+
+**URL:** https://docs.axolotl.ai/docs/api/monkeypatch.gradient_checkpointing.offload_disk.html
+
+**Contents:**
+- monkeypatch.gradient_checkpointing.offload_disk
+- Classes
+ - Disco
+ - Methods
+ - backward
+ - forward
+ - get_instance
+ - DiskOffloadManager
+ - Methods
+ - cleanup
+
+monkeypatch.gradient_checkpointing.offload_disk
+
+DISCO - DIsk-based Storage and Checkpointing with Optimized prefetching
+
+Disco: DIsk-based Storage and Checkpointing with Optimized prefetching Advanced disk-based gradient checkpointer with prefetching.
+
+Backward pass that loads activations from disk with prefetching
+
+Forward pass that offloads activations to disk asynchronously
+
+Get or create the offload manager
+
+Manages offloaded tensors and handles prefetching in a separate thread. Includes synchronization to prevent race conditions.
+
+Clean up all temp files and stop prefetch thread with proper synchronization
+
+Clean up a specific tensor file after it’s been used
+
+Load tensor from disk or prefetch cache with proper synchronization
+
+Save tensor to disk asynchronously and return file path with thread-safe operations
+
+Trigger prefetching of the next N tensors with proper synchronization
+
+Wait for a tensor to be saved to disk
+
+**Examples:**
+
+Example 1 (python):
+```python
+monkeypatch.gradient_checkpointing.offload_disk.Disco()
+```
+
+Example 2 (python):
+```python
+monkeypatch.gradient_checkpointing.offload_disk.Disco.backward(
+ ctx,
+ *grad_outputs,
+)
+```
+
+Example 3 (python):
+```python
+monkeypatch.gradient_checkpointing.offload_disk.Disco.forward(
+ ctx,
+ forward_function,
+ hidden_states,
+ *args,
+ prefetch_size=1,
+ prefetch_to_gpu=True,
+ save_workers=4,
+)
+```
+
+Example 4 (python):
+```python
+monkeypatch.gradient_checkpointing.offload_disk.Disco.get_instance(
+ prefetch_size=1,
+ prefetch_to_gpu=True,
+ save_workers=4,
+)
+```
+
+---
+
+## utils.samplers.multipack
+
+**URL:** https://docs.axolotl.ai/docs/api/utils.samplers.multipack.html
+
+**Contents:**
+- utils.samplers.multipack
+- Classes
+ - MultipackBatchSampler
+ - Methods
+ - efficiency
+ - gather_efficiency
+ - Returns
+ - gather_len_batches
+ - generate_batches
+ - Parameters
+
+utils.samplers.multipack
+
+Multipack Batch Sampler - An efficient batch sampler for packing variable-length sequences into fixed-capacity batches to optimize memory usage and training throughput.
+
+Batch sampler class for efficient packing of variable-length sequences
+
+This sampler packs sequences into fixed-capacity bins (batches) to maximize GPU memory utilization and training throughput by reducing padding.
+
+It supports both parallel packing (using FFD algorithm) and sequential packing (preserving original sequence order).
+
+Calculate the packing efficiency (ratio of tokens used to total token slots). Higher is better - 1.0 would mean perfect packing with no wasted space.
+
+Gather and synchronize packing efficiency estimates across all distributed ranks.
+
+Gather and synchronize batch counts across all distributed ranks. Returns the minimum number of batches available on any rank.
+
+Generate packed batches for training.
+
+Set the epoch number, used for reproducible shuffling across epochs
+
+Sequential allocator that preserves example order.
+
+First-fit-decreasing bin packing algorithm check.
+
+Checks if sequences with the given lengths could fit in the specified number of bins.
+
+Pack a group of sequences into bins using First-Fit Decreasing algorithm.
+
+Pack sequences into bins using parallel processing.
+
+Returns: List of bins, where each bin contains indices of sequences assigned to it.
+
+**Examples:**
+
+Example 1 (python):
+```python
+utils.samplers.multipack.MultipackBatchSampler(
+ sampler,
+ batch_size,
+ batch_max_len,
+ lengths,
+ packing_efficiency_estimate=1.0,
+ drop_last=True,
+ num_count_samples=4,
+ sequential=False,
+ group_size=100000,
+ bin_size=200,
+ num_processes=None,
+ safe_mode=True,
+ mp_start_method='fork',
+ **kwargs,
+)
+```
+
+Example 2 (python):
+```python
+utils.samplers.multipack.MultipackBatchSampler.efficiency()
+```
+
+Example 3 (python):
+```python
+utils.samplers.multipack.MultipackBatchSampler.gather_efficiency()
+```
+
+Example 4 (python):
+```python
+utils.samplers.multipack.MultipackBatchSampler.gather_len_batches(num)
+```
+
+---
+
+## core.trainers.mixins.scheduler
+
+**URL:** https://docs.axolotl.ai/docs/api/core.trainers.mixins.scheduler.html
+
+**Contents:**
+- core.trainers.mixins.scheduler
+- Classes
+ - SchedulerMixin
+ - Methods
+ - create_scheduler
+ - Parameters
+
+core.trainers.mixins.scheduler
+
+Module for Axolotl trainer scheduler mixin
+
+Mixin class for scheduler setup in CausalTrainer.
+
+Set up the scheduler. The optimizer of the trainer must have been set up either before this method is called or passed as an argument.
+
+**Examples:**
+
+Example 1 (python):
+```python
+core.trainers.mixins.scheduler.SchedulerMixin()
+```
+
+Example 2 (python):
+```python
+core.trainers.mixins.scheduler.SchedulerMixin.create_scheduler(
+ num_training_steps,
+ optimizer=None,
+)
+```
+
+---
+
+## utils.collators.batching
+
+**URL:** https://docs.axolotl.ai/docs/api/utils.collators.batching.html
+
+**Contents:**
+- utils.collators.batching
+- Classes
+ - BatchSamplerDataCollatorForSeq2Seq
+ - DataCollatorForSeq2Seq
+ - Parameters
+ - PretrainingBatchSamplerDataCollatorForSeq2Seq
+ - V2BatchSamplerDataCollatorForSeq2Seq
+
+utils.collators.batching
+
+Data collators for axolotl to pad labels and position_ids for packed sequences
+
+Collator for multipack specific to the using the BatchSampler
+
+Data collator that will dynamically pad the inputs received, as well as the labels and position_ids
+
+Collator for multipack specific to the using the BatchSampler
+
+Collator for multipack specific to the using the BatchSampler
+
+**Examples:**
+
+Example 1 (python):
+```python
+utils.collators.batching.BatchSamplerDataCollatorForSeq2Seq(
+ tokenizer,
+ model=None,
+ padding=True,
+ max_length=None,
+ pad_to_multiple_of=None,
+ label_pad_token_id=-100,
+ position_pad_token_id=0,
+ return_tensors='pt',
+)
+```
+
+Example 2 (python):
+```python
+utils.collators.batching.DataCollatorForSeq2Seq(
+ tokenizer,
+ model=None,
+ padding=True,
+ max_length=None,
+ pad_to_multiple_of=None,
+ label_pad_token_id=-100,
+ position_pad_token_id=0,
+ return_tensors='pt',
+)
+```
+
+Example 3 (python):
+```python
+utils.collators.batching.PretrainingBatchSamplerDataCollatorForSeq2Seq(
+ *args,
+ multipack_attn=True,
+ **kwargs,
+)
+```
+
+Example 4 (python):
+```python
+utils.collators.batching.V2BatchSamplerDataCollatorForSeq2Seq(
+ tokenizer,
+ model=None,
+ padding=True,
+ max_length=None,
+ pad_to_multiple_of=None,
+ label_pad_token_id=-100,
+ position_pad_token_id=0,
+ return_tensors='pt',
+ squash_position_ids=False,
+)
+```
+
+---
+
+## prompt_strategies.orcamini
+
+**URL:** https://docs.axolotl.ai/docs/api/prompt_strategies.orcamini.html
+
+**Contents:**
+- prompt_strategies.orcamini
+- Classes
+ - OrcaMiniPrompter
+
+prompt_strategies.orcamini
+
+Prompt Strategy for finetuning Orca Mini (v2) models see also https://huggingface.co/psmathur/orca_mini_v2_7b for more information
+
+Use dataset type: orcamini in config.yml to use this prompt style.
+
+Compared to the alpaca_w_system.open_orca dataset type, this one specifies the system prompt with “### System:”.
+
+Not suited/tested for multiple-turn conversations without further adjustments.
+
+Adjusted Prompter for Orca Mini (v2) datasets
+
+**Examples:**
+
+Example 1 (python):
+```python
+prompt_strategies.orcamini.OrcaMiniPrompter(
+ prompt_style=PromptStyle.INSTRUCT.value,
+)
+```
+
+---
+
+## prompt_strategies.dpo.chat_template
+
+**URL:** https://docs.axolotl.ai/docs/api/prompt_strategies.dpo.chat_template.html
+
+**Contents:**
+- prompt_strategies.dpo.chat_template
+- Functions
+ - argilla_chat
+ - Parameters
+ - Returns
+ - Dataset format
+
+prompt_strategies.dpo.chat_template
+
+DPO prompt strategies for using tokenizer chat templates.
+
+DPO chat template strategy for argilla-style datasets.
+
+For argilla-style datasets where chosen/rejected contain full conversations instead of single response messages. Extracts the conversation history from the chosen field and formats both chosen/rejected responses using the configured chat template.
+
+{ “chosen”: [ {“role”: “user”, “content”: “…”}, {“role”: “assistant”, “content”: “…”} ], “rejected”: [ {“role”: “user”, “content”: “…”}, {“role”: “assistant”, “content”: “…”} ] }
+
+**Examples:**
+
+Example 1 (python):
+```python
+prompt_strategies.dpo.chat_template.argilla_chat(cfg, dataset_idx=0, **kwargs)
+```
+
+---
+
+## monkeypatch.relora
+
+**URL:** https://docs.axolotl.ai/docs/api/monkeypatch.relora.html
+
+**Contents:**
+- monkeypatch.relora
+- Classes
+ - ReLoRACallback
+
+Implements the ReLoRA training procedure from https://arxiv.org/abs/2307.05695, minus the initial full fine-tune.
+
+Callback to merge LoRA weights into the base model and save full-weight checkpoints
+
+**Examples:**
+
+Example 1 (python):
+```python
+monkeypatch.relora.ReLoRACallback(cfg)
+```
+
+---
+
+## monkeypatch.transformers_fa_utils
+
+**URL:** https://docs.axolotl.ai/docs/api/monkeypatch.transformers_fa_utils.html
+
+**Contents:**
+- monkeypatch.transformers_fa_utils
+- Functions
+ - fixed_fa_peft_integration_check
+ - Parameters
+
+monkeypatch.transformers_fa_utils
+
+see https://github.com/huggingface/transformers/pull/35834
+
+PEFT usually casts the layer norms in float32 for training stability reasons therefore the input hidden states gets silently casted in float32. Hence, we need cast them back in float16 / bfloat16 just to be sure everything works as expected. This might slowdown training & inference so it is recommended to not cast the LayerNorms!
+
+**Examples:**
+
+Example 1 (python):
+```python
+monkeypatch.transformers_fa_utils.fixed_fa_peft_integration_check(
+ query,
+ key,
+ value,
+ target_dtype=None,
+ preferred_dtype=None,
+)
+```
+
+---
+
+## utils.collators.mm_chat
+
+**URL:** https://docs.axolotl.ai/docs/api/utils.collators.mm_chat.html
+
+**Contents:**
+- utils.collators.mm_chat
+- Classes
+ - MultiModalChatDataCollator
+
+utils.collators.mm_chat
+
+Collators for multi-modal chat messages and packing
+
+Collator for multi-modal chat messages
+
+**Examples:**
+
+Example 1 (python):
+```python
+utils.collators.mm_chat.MultiModalChatDataCollator(
+ tokenizer,
+ processing_strategy,
+ packing=False,
+ return_tensors='pt',
+ padding=True,
+ pad_to_multiple_of=None,
+)
+```
+
+---
+
+## utils.lora
+
+**URL:** https://docs.axolotl.ai/docs/api/utils.lora.html
+
+**Contents:**
+- utils.lora
+- Functions
+ - get_lora_merged_state_dict
+ - Parameters
+ - Returns
+
+module to get the state dict of a merged lora model
+
+Create and return a state_dict that has the LoRA deltas merged into the base model’s weights, without modifying model in place.
+
+**Examples:**
+
+Example 1 (python):
+```python
+utils.lora.get_lora_merged_state_dict(model)
+```
+
+---
+
+## utils.model_shard_quant
+
+**URL:** https://docs.axolotl.ai/docs/api/utils.model_shard_quant.html
+
+**Contents:**
+- utils.model_shard_quant
+- Functions
+ - load_and_quantize
+
+utils.model_shard_quant
+
+module to handle loading model on cpu/meta device for FSDP
+
+Loads value tensor into submodule of module, optionally skipping skip_names and converting to dtype.
+
+Quantizes Params4bit on device then places on “cpu” if to_cpu=True or “meta” if to_meta=True.
+
+**Examples:**
+
+Example 1 (python):
+```python
+utils.model_shard_quant.load_and_quantize(
+ module,
+ name,
+ value,
+ device=None,
+ dtype=None,
+ skip_names=None,
+ to_cpu=False,
+ to_meta=False,
+ verbose=False,
+ quant_method='bnb',
+)
+```
+
+---
+
+## monkeypatch.gradient_checkpointing.offload_cpu
+
+**URL:** https://docs.axolotl.ai/docs/api/monkeypatch.gradient_checkpointing.offload_cpu.html
+
+**Contents:**
+- monkeypatch.gradient_checkpointing.offload_cpu
+- Classes
+ - CPU_Offloaded_Gradient_Checkpointer
+
+monkeypatch.gradient_checkpointing.offload_cpu
+
+CPU offloaded checkpointing
+
+Saves VRAM by smartly offloading to RAM. Tiny hit to performance, since we mask the movement via non blocking calls.
+
+**Examples:**
+
+Example 1 (python):
+```python
+monkeypatch.gradient_checkpointing.offload_cpu.CPU_Offloaded_Gradient_Checkpointer(
+)
+```
+
+---
+
+## core.builders.base
+
+**URL:** https://docs.axolotl.ai/docs/api/core.builders.base.html
+
+**Contents:**
+- core.builders.base
+- Classes
+ - TrainerBuilderBase
+ - Methods
+ - get_post_trainer_create_callbacks
+
+Base class for trainer builder
+
+Base class for trainer builder.
+
+Callbacks added after the trainer is created, usually b/c these need access to the trainer
+
+**Examples:**
+
+Example 1 (python):
+```python
+core.builders.base.TrainerBuilderBase(cfg, model, tokenizer, processor=None)
+```
+
+Example 2 (python):
+```python
+core.builders.base.TrainerBuilderBase.get_post_trainer_create_callbacks(trainer)
+```
+
+---
+
+## core.builders.rl
+
+**URL:** https://docs.axolotl.ai/docs/api/core.builders.rl.html
+
+**Contents:**
+- core.builders.rl
+- Classes
+ - HFRLTrainerBuilder
+
+Builder for RLHF trainers
+
+Trainer factory class for TRL-based RLHF trainers (e.g. DPO)
+
+**Examples:**
+
+Example 1 (python):
+```python
+core.builders.rl.HFRLTrainerBuilder(cfg, model, tokenizer, processor=None)
+```
+
+---
+
+## utils.schemas.integrations
+
+**URL:** https://docs.axolotl.ai/docs/api/utils.schemas.integrations.html
+
+**Contents:**
+- utils.schemas.integrations
+- Classes
+ - CometConfig
+ - GradioConfig
+ - LISAConfig
+ - MLFlowConfig
+ - OpenTelemetryConfig
+ - RayConfig
+ - WandbConfig
+
+utils.schemas.integrations
+
+Pydantic models for Axolotl integrations
+
+Comet configuration subset
+
+Gradio configuration subset
+
+LISA configuration subset
+
+MLFlow configuration subset
+
+OpenTelemetry configuration subset
+
+Ray launcher configuration subset
+
+Wandb configuration subset
+
+**Examples:**
+
+Example 1 (python):
+```python
+utils.schemas.integrations.CometConfig()
+```
+
+Example 2 (python):
+```python
+utils.schemas.integrations.GradioConfig()
+```
+
+Example 3 (python):
+```python
+utils.schemas.integrations.LISAConfig()
+```
+
+Example 4 (python):
+```python
+utils.schemas.integrations.MLFlowConfig()
+```
+
+---
+
+## utils.data.sft
+
+**URL:** https://docs.axolotl.ai/docs/api/utils.data.sft.html
+
+**Contents:**
+- utils.data.sft
+- Functions
+ - prepare_datasets
+ - Parameters
+ - Returns
+
+Data handling specific to SFT.
+
+Prepare training and evaluation datasets based on configuration.
+
+**Examples:**
+
+Example 1 (python):
+```python
+utils.data.sft.prepare_datasets(cfg, tokenizer, processor=None)
+```
+
+---
+
+## integrations.liger.args
+
+**URL:** https://docs.axolotl.ai/docs/api/integrations.liger.args.html
+
+**Contents:**
+- integrations.liger.args
+- Classes
+ - LigerArgs
+
+integrations.liger.args
+
+Module for handling LIGER input arguments.
+
+Input args for LIGER.
+
+**Examples:**
+
+Example 1 (python):
+```python
+integrations.liger.args.LigerArgs()
+```
+
+---
+
+## monkeypatch.mixtral
+
+**URL:** https://docs.axolotl.ai/docs/api/monkeypatch.mixtral.html
+
+**Contents:**
+- monkeypatch.mixtral
+
+Patches to support multipack for mixtral
+
+---
+
+## cli.preprocess
+
+**URL:** https://docs.axolotl.ai/docs/api/cli.preprocess.html
+
+**Contents:**
+- cli.preprocess
+- Functions
+ - do_cli
+ - Parameters
+ - do_preprocess
+ - Parameters
+
+CLI to run preprocessing of a dataset.
+
+Parses axolotl config, CLI args, and calls do_preprocess.
+
+Preprocesses dataset specified in axolotl config.
+
+**Examples:**
+
+Example 1 (python):
+```python
+cli.preprocess.do_cli(config=Path('examples/'), **kwargs)
+```
+
+Example 2 (python):
+```python
+cli.preprocess.do_preprocess(cfg, cli_args)
+```
+
+---
+
+## prompt_strategies.kto.llama3
+
+**URL:** https://docs.axolotl.ai/docs/api/prompt_strategies.kto.llama3.html
+
+**Contents:**
+- prompt_strategies.kto.llama3
+- Functions
+ - argilla_chat
+ - intel
+ - ultra
+
+prompt_strategies.kto.llama3
+
+KTO strategies for llama-3 chat template
+
+for argilla/kto-mix-15k conversations
+
+For Intel Orca KTO ex: argilla/distilabel-intel-orca-kto
+
+for ultrafeedback binarized conversations ex: argilla/ultrafeedback-binarized-preferences-cleaned-kto
+
+**Examples:**
+
+Example 1 (python):
+```python
+prompt_strategies.kto.llama3.argilla_chat(cfg, **kwargs)
+```
+
+Example 2 (python):
+```python
+prompt_strategies.kto.llama3.intel(cfg, **kwargs)
+```
+
+Example 3 (python):
+```python
+prompt_strategies.kto.llama3.ultra(cfg, **kwargs)
+```
+
+---
+
+## prompt_strategies.orpo.chat_template
+
+**URL:** https://docs.axolotl.ai/docs/api/prompt_strategies.orpo.chat_template.html
+
+**Contents:**
+- prompt_strategies.orpo.chat_template
+- Classes
+ - Message
+ - MessageList
+ - ORPODatasetParsingStrategy
+ - Methods
+ - get_chosen_conversation_thread
+ - get_prompt
+ - get_rejected_conversation_thread
+ - ORPOPrompter
+
+prompt_strategies.orpo.chat_template
+
+chatml prompt tokenization strategy for ORPO
+
+Strategy to parse chosen rejected dataset into messagelist
+
+Dataset structure mappings
+
+Map the data to extract everything up to the last turn
+
+Dataset structure mappings
+
+Single Turn prompter for ORPO
+
+rejected_input_ids input_ids rejected_attention_mask attention_mask rejected_labels labels
+
+chatml transforms for datasets with system, input, chosen, rejected
+
+**Examples:**
+
+Example 1 (python):
+```python
+prompt_strategies.orpo.chat_template.Message()
+```
+
+Example 2 (python):
+```python
+prompt_strategies.orpo.chat_template.MessageList()
+```
+
+Example 3 (python):
+```python
+prompt_strategies.orpo.chat_template.ORPODatasetParsingStrategy()
+```
+
+Example 4 (python):
+```python
+prompt_strategies.orpo.chat_template.ORPODatasetParsingStrategy.get_chosen_conversation_thread(
+ prompt,
+)
+```
+
+---
+
+## loaders.processor
+
+**URL:** https://docs.axolotl.ai/docs/api/loaders.processor.html
+
+**Contents:**
+- loaders.processor
+
+Processor loading functionality for multi-modal models
+
+---
+
+## utils.callbacks.comet_
+
+**URL:** https://docs.axolotl.ai/docs/api/utils.callbacks.comet_.html
+
+**Contents:**
+- utils.callbacks.comet_
+- Classes
+ - SaveAxolotlConfigtoCometCallback
+
+utils.callbacks.comet_
+
+Comet module for trainer callbacks
+
+Callback to save axolotl config to comet
+
+**Examples:**
+
+Example 1 (python):
+```python
+utils.callbacks.comet_.SaveAxolotlConfigtoCometCallback(axolotl_config_path)
+```
+
+---
diff --git a/src/skills/bundled/axolotl/references/dataset-formats.md b/src/skills/bundled/axolotl/references/dataset-formats.md
new file mode 100644
index 0000000..aa66b08
--- /dev/null
+++ b/src/skills/bundled/axolotl/references/dataset-formats.md
@@ -0,0 +1,1029 @@
+# Axolotl - Dataset-Formats
+
+**Pages:** 9
+
+---
+
+## Custom Pre-Tokenized Dataset
+
+**URL:** https://docs.axolotl.ai/docs/dataset-formats/tokenized.html
+
+**Contents:**
+- Custom Pre-Tokenized Dataset
+
+**Examples:**
+
+Example 1 (yaml):
+```yaml
+datasets:
+ - path: /path/to/your/file.jsonl
+ ds_type: json
+ type:
+```
+
+Example 2 (json):
+```json
+{"input_ids":[271,299,99],"attention_mask":[1,1,1],"labels":[271,-100,99]}
+{"input_ids":[87,227,8383,12],"attention_mask":[1,1,1,1],"labels":[87,227,8383,12]}
+```
+
+---
+
+## Dataset Formats
+
+**URL:** https://docs.axolotl.ai/docs/dataset-formats/index.html
+
+**Contents:**
+- Dataset Formats
+- Pre-training
+ - Pre-training from Hugging Face hub datasets
+ - Pre-training from local dataset files
+ - Pre-training without streaming
+ - Pre-training dataset configuration tips
+ - Setting max_steps
+ - Group_by_length
+ - Reference
+- Supervised fine-tuning (SFT)
+
+Axolotl is a training framework that aims to make the process convenient yet flexible to users by simply passing a config yaml file.
+
+As there are a lot of available options in Axolotl, this guide aims to provide an simplify the user experience to choosing the proper choice.
+
+Axolotl supports 3 kinds of training methods: pre-training, supervised fine-tuning, and preference-based post-training (e.g. DPO, ORPO, PRMs). Each method has their own dataset format which are described below.
+
+This guide will mainly use JSONL as an introduction. Please refer to the dataset loading docs to understand how to load datasets from other sources.
+
+For pretraining_dataset: specifically, please refer to the Pre-training section.
+
+When aiming to train on large corpora of text datasets, pre-training is your go-to choice. Due to the size of these datasets, downloading the entire-datasets before beginning training would be prohibitively time-consuming. Axolotl supports streaming to only load batches into memory at a time.
+
+A sample format for a pre-training dataset is as follows:
+
+It is typically recommended to save your dataset as .jsonl due to its flexibility and simplicity.
+
+Axolotl supports loading from a Hugging Face hub repo or from local files.
+
+As an example, to train using a Hugging Face dataset hf_org/name, you can pass the following config:
+
+Given a few corpus files: A.jsonl, B.jsonl, and C.jsonl, your config will look like the below:
+
+While we recommend .jsonl, you can also use the other formats (csv, parquet, arrow, SQL, Webdataset) that are supported by Dataset.load_dataset
+
+In the case that the dataset is small and can be loaded entirely into memory, another approach to running pre-training is to use the completion format. This would mean that the entire dataset is pre-tokenized instead of on-demand in streaming.
+
+One benefit of this is that the tokenization can be performed separately on a CPU-only machine, and then transferred to a GPU machine for training to save costs.
+
+For completion only, Axolotl would split texts if it exceeds the context length into multiple smaller prompts. If you are interested in having this for pretraining_dataset too, please let us know or help make a PR!
+
+When using streaming for large datasets, Axolotl does not know in advance how large the dataset is and does not know when to stop.
+
+Therefore, it is necessary to set max_steps: int in your config for pre-training to run, so that Axolotl knows when to stop training.
+
+One step is equal to sequence_len * micro_batch_size * gradient_accumulation_steps * total_num_gpus tokens.
+
+It is recommended to leave this off if downloading from Hugging Face hub as it would download the entire dataset which can be very large.
+
+Please see docs here.
+
+Supervised fine-tuning is the process of training models to respond to an instruction or chat input.
+
+As there are a wide variety of dataset formats, Axolotl tries to support a majority of the formats available in public datasets.
+
+Axolotl provides four approaches for loading datasets, however, it’s easier to work backwards from the dataset you have available to figure out which approach to use.
+
+A flow chart is as follows:
+
+Do you already have the dataset tokenized? If yes, check Pre-Tokenized Dataset.
+
+Do you want to format the dataset yourself and manually choose each section to mask? If yes, check Template Free Dataset
+
+Is your dataset in a “conversation” format, containing a list[messages]? If yes, check Conversation Dataset
+
+Is your dataset in an “instruct” format, containing { instruction, response }? If yes, check Instruction Dataset
+
+If you went through the flow chart and did not find one that matches, it is recommended to preprocess your dataset into one of the above or create a thread on Github Discussion.
+
+You can mix and match within each approach or across approaches to train a model on a variety of datasets.
+
+We suggest this approach when you want to bring your own tokenized dataset.
+
+Axolotl expects the dataset to have three keys:
+
+Make sure to add BOS/EOS tokens to your prompt and mask it appropriately.
+
+A config for this would look like:
+
+Reference: Pre-Tokenized Dataset Documentation.
+
+We recommend this approach when you want granular control over the prompt formatting, special tokens, and masking, whilst letting Axolotl handle the tokenization. This is very useful if your dataset has unique prompts that differ across samples and where one single general template wouldn’t suffice.
+
+In the example below, you could see that there is no proper structure. At the same time, it’s very flexible as there are no constraints on how your prompt can look.
+
+Each prompt must be have a key called segments which is a list of { text, label }.
+
+Reference: Template Free Documentation.
+
+conversation messages are a list of messages which usually contain a role and content key.
+
+Fun fact: Axolotl synonymously refers to “chat” messages as conversation messages due to how FastChat initially used this term to build a widely used fastchat conversation method for formatting chat messages prior to the creation of chat_templates.
+
+The current most popular and convenient method for inference is to use chat_templates for formatting prompts. Axolotl supports using chat_templates for training to ensure that the model performs in the same environment as in inference.
+
+Here’s a quick rundown on chat_template: A chat_template is a Jinja2 template which formats a list of messages into a prompt.
+
+An example of a prompt formatted into a popular template called ChatML can be seen below:
+
+Single prompt (pretty-printed):
+
+The ChatML template is as follows:
+
+The above prompt formatted into this template will result in:
+
+By using delimiters (<|im_start|> and <|im_end|>), a prompt separates different speakers which helps the model identify which portion belongs to whom.
+
+Older conversation datasets with the following format are colloquially called sharegpt datasets.
+
+Newer conversation datasets usually follow the OpenAI format.
+
+Axolotl supports both as well as allowing customization of any kind of key.
+
+To properly use this method, it is important to identify three things:
+
+Which chat_template would you use?
+
+What are the keys in your dataset, and what are the possible roles? For example, in OpenAI format, the keys would be messages, role, and content, respectively, whereas the possible roles are system, user, and assistant.
+
+What do you want to mask? For instance, only assistant messages, only last message, or nothing.
+
+There are a lot of chat_templates out there. Axolotl supports the common ones: supported chat templates. For example, to use ChatML, it would be chat_template: chatml.
+
+However, it is also possible to use the already configured template within the tokenizer by specifying chat_template: tokenizer_default. If you want a fallback (in case some tokenizer does not have it pre-configured), you can do chat_template: tokenizer_default_fallback_chatml to fallback to the ChatML template if a tokenizer template was not found.
+
+One last but powerful approach is to bring your own template. This can be set via:
+
+We currently default to OpenAI format for dataset keys, so if that’s your current dataset format, there’s nothing to do here.
+
+If your dataset format is different, here are the keys you should check (with their defaults):
+
+In some chat_templates (e.g. Gemma), the roles are hardcoded to user and assistant. Consequently, you may find it necessary to map the roles in your dataset to these above. We currently have some defaults that should work for common datasets, but if you get a KeyError, it would be necessary to add mapping for your roles. Here is an example of how it would look like:
+
+In the example above, all gpt and model values are converted to assistant. All human values are converted to user.
+
+The common use case for chat_template is for chat messages, therefore, it is common to mask all non-assistant messages. Assistant messages refer to the bot messages that you want the model to learn on.
+
+To train on all assistant messages, you would set the following configs.
+
+The train_on_eos config means that it would mask all EOS tokens for turns that aren’t assistant-turns. The other options are: all and last to choose which EOS to train on.
+
+Perhaps, you want to train on assistant and narrator roles, you can simply add narrator to the list of roles_to_train. You would also need to add it to the mapping of roles above.
+
+As chat_templates may use hardcoded EOS/EOT tokens that are different from the tokenizer’s EOS, it is highly recommended to set them. For example, ChatML uses <|im_end|> to end turns.
+
+Once all the above steps are completed, you could combine all these configs together to form a bespoke configuration for your custom dataset.
+
+If this config were to be applied to the sample dataset above, the output would look as such (which can be retrieved via axolotl preprocess config.yaml --debug):
+
+The first number refers to the label, the second refers to the token_id. For example, -100 labels appear on non-assistant portions, meaning that they are masked during. For assistant portions, the label is the same as the token_id.
+
+If during preprocess, there are a lot of warnings of Could not find content __ boundary, please check the FAQ section for chat_templates.
+
+Please see docs here.
+
+Instruction datasets are used to train instruction-following models and comprise a prompt, containing an instruction, and a single response. In contrast to chat datasets which may be multi-turn, instruct datasets are typically single-turn.
+
+An example is of a common format called Alpaca:
+
+Using those keys, a prompt can be built based on it.
+
+This can be configured as such:
+
+Axolotl supports many kinds of instruction dataset. All of them can be found in the Instruction Dataset Documentation with their respective type and sample row format.
+
+Due to the myriad possibilities of instruction formats, Axolotl allows customizing your own instruction format without having to dive into the code directly.
+
+In the example below, a sample row is used to output in mistral_v1 format.
+
+The config sets that the field_instruction is actually named input, and the field_input is empty as we don’t have an input in this sample. Generally, instruction can be thought as the question to the model, and input as the additional information with output being the response. It is not necessary to have an input nor system. In the end, the most important part is to understand what format you want it to look like and how you can customize this to your use case.
+
+Reference: Custom Instruct Prompt Format Documentation.
+
+As there are multiple RLHF methods with their own dataset requirements. Please see RLHF documentation for more detail.
+
+**Examples:**
+
+Example 1 (json):
+```json
+{"text": "first row"}
+{"text": "second row"}
+...
+```
+
+Example 2 (yaml):
+```yaml
+pretraining_dataset: hf_org/name
+```
+
+Example 3 (yaml):
+```yaml
+pretraining_dataset:
+ - path: json
+ data_files:
+ - A.jsonl
+ - B.jsonl
+ - C.jsonl
+```
+
+Example 4 (yaml):
+```yaml
+datasets:
+ - path: hf_org/name
+ type: completion
+```
+
+---
+
+## Conversation
+
+**URL:** https://docs.axolotl.ai/docs/dataset-formats/conversation.html
+
+**Contents:**
+- Conversation
+- chat_template
+ - Migrating from sharegpt
+ - Examples
+ - Training on last message
+ - Overriding default chat template
+ - Using default chat template with fallback
+ - Custom Jinja template
+ - Using template with different token for EOT and EOS
+ - Using tool use
+
+Chat Template strategy uses a jinja2 template that converts a list of messages into a prompt. Support using tokenizer’s template, a supported template, or custom jinja2.
+
+See configs for full configs and supported templates.
+
+Most configs can be adapted as follows:
+
+We recommend checking the below examples for other usecases.
+
+(Legacy) Using the default chat template in the tokenizer_config.json on OpenAI messages format, training on only last message.
+
+If you receive an error like “chat_template choice is tokenizer_default but tokenizer’s chat_template is null.”, it means the tokenizer does not have a default chat_template. Follow the examples below instead to set a custom chat_template.
+
+Using the gemma chat template to override the tokenizer_config.json’s chat template on OpenAI messages format, training on all assistant messages.
+
+If you want to use built-in chat_template, use chat_template: tokenizer_default (this is set by default).
+
+Using the tokenizer_config.json’s chat template or chatml as fallback if the former’s chat template does not exist, on OpenAI messages format, training on all assistant messages.
+
+Using a custom jinja template on OpenAI messages format, training on all assistant messages.
+
+Please make sure that your tokenizer.eos_token is same as EOS (End-of-Sequence) token in template. Otherwise, set eos_token under special_tokens:.
+
+See config documentation for detailed explanations of “turn”, “last”, and “all” options for training on tokens.
+
+Using eot_tokens requires each token that exists in chat_template to be a single token in the tokenizer. Otherwise, the tokenizer will split the token and cause unexpected behavior.
+
+You can add those tokens as new tokens under tokens: or (recommended) override unused added_tokens via added_tokens_overrides:. See config for more details.
+
+If EOS token only appears at the end of a prompt, train_on_eos: last is equivalent to train_on_eos: turn. Therefore, generally, you can leave them to their defaults and omit them.
+
+Instead of passing tools via the system prompt, an alternative method would be to have the tools in a separate column and loaded via chat_template to let the template dynamically build it.
+
+Tools need to follow JSON schema.
+
+If you have tool arguments with same name but different dtypes (like "time": string and "time": number), please save arguments: as JSON string to prevent datasets from having casting issues.
+
+Example config for Llama4:
+
+Look into the chat_template you are using to see if it supports tools and what the expected role is for the tool answer. In the example above, the tool answer is expected to be in the tool or ipython role for llama4 template.
+
+(Advanced) Using fine-grained control over tokens and turns to train in a conversation
+
+For a data sample that looks like:
+
+The configuration would look like:
+
+It is not necessary to set both message_field_training and message_field_training_detail at once.
+
+(For Qwen3 template only) Enable reasoning split, where the reasoning is split from the content and passed as a separate field into the template.
+
+For example, a content can look like:
+
+After split, it will look like:
+
+ShareGPT is deprecated!. Please see chat_template section.
+
+**Examples:**
+
+Example 1 (json):
+```json
+{"messages": [{"role": "...", "content": "..."}, {"role": "...", "content": "..."}, ...]}
+```
+
+Example 2 (yaml):
+```yaml
+# old
+chat_template: chatml
+datasets:
+ - path: ...
+ type: sharegpt
+ conversation: chatml
+
+# new (if using tokenizer's chat_template)
+datasets:
+ - path: ...
+ type: chat_template
+
+ field_messages: conversations
+ message_property_mappings:
+ role: from
+ content: value
+
+# new (if setting a new chat_template like chatml, gemma, etc)
+chat_template: chatml
+datasets:
+ - path: ...
+ type: chat_template
+
+ field_messages: conversations
+ message_property_mappings:
+ role: from
+ content: value
+```
+
+Example 3 (yaml):
+```yaml
+datasets:
+ - path: ...
+ type: chat_template
+ roles_to_train:
+ train_on_eos:
+```
+
+Example 4 (yaml):
+```yaml
+chat_template: gemma # this overwrites the tokenizer's chat_template
+datasets:
+ - path: ...
+ type: chat_template
+ roles_to_train: ["assistant"] # default value
+```
+
+---
+
+## Pre-training
+
+**URL:** https://docs.axolotl.ai/docs/dataset-formats/pretraining.html
+
+**Contents:**
+- Pre-training
+
+For pretraining, there is no prompt template or roles. The only required field is text:
+
+Axolotl usually loads the entire dataset into memory. This will be challenging for large datasets. Use the following config to enable streaming:
+
+**Examples:**
+
+Example 1 (json):
+```json
+{"text": "first row"}
+{"text": "second row"}
+...
+```
+
+Example 2 (yaml):
+```yaml
+pretraining_dataset:
+ - name:
+ path:
+ split:
+ text_column: # column in dataset with the data, usually `text`
+ type: pretrain
+ trust_remote_code:
+ skip: # number of rows of data to skip over from the beginning
+```
+
+---
+
+## Template-Free
+
+**URL:** https://docs.axolotl.ai/docs/dataset-formats/template_free.html
+
+**Contents:**
+- Template-Free
+- Background
+ - Masking Inputs
+ - You may not want prompt templates
+ - The input_output format
+- Usage
+ - 1. Prepare Data
+ - 2. Use type: input_output
+ - 3. Check the prompts
+
+One of the most popular features of axolotl is setting the following configuration value:
+
+If you declare a dataset formats such as alpaca or chatml, axolotl knows what is an input (i.e. human) vs. an output (i.e. the assistant) and masks the input labels so that your model can focus on predicting the outputs only.
+
+However, there are many situations where you don’t want to use one of these formats or templates. This is because they can:
+
+You can construct your prompts without a template by using the input_output format, by setting type: input_output in your configuration file like this:
+
+Unlike type: completion, which is also template-free, type: input_output allows you to mask segments of your text. More details on how this works are described below.
+
+This is how you can use the input_output format:
+
+To use the input_output format, collect your data in the following format into a jsonl file (below is the first row from the file output.jsonl` pretty printed):
+
+Set label:false when you want to mask a segment of text so that the model isn’t trained on it. Some things to keep in mind:
+
+[!IMPORTANT] 1. EOS, BOS, spaces, newlines etc. are entirely up to you. Axolotl concatenates all the segments as-is. The tokenizer doesn’t add anything additional. Notice how I added spaces, newlines, (BOS), and (EOS) myself. 2. Make sure you check the materialized output to validate that the prompt is getting assembled how you like.
+
+Let’s materialize data with our output.jsonl file by setting type: input_output in our axolotl config:
+
+You can use the following command to materialize your data. The --debug flag will print the tokens, along with the labels so you can verify that the correct items are being ignored:
+
+The format is decoded_token(label, token_id), for example, (1, 1) means that the token is , the label is 1 and the token_id is 1. When the label is -100 then that token is ignored for training.
+
+Here is another way to check the materialized output:
+
+We can check that the right tokens are ignored by comparing the labels to each token:
+
+If we look at the input data, the above table seems correct! (The jsonl version is repeated below for reference):
+
+**Examples:**
+
+Example 1 (yaml):
+```yaml
+train_on_inputs: false
+```
+
+Example 2 (yaml):
+```yaml
+train_on_inputs: false # Mask segments of your data
+datasets:
+ - path: output.jsonl
+ type: input_output # use template free prompt construction
+```
+
+Example 3 (bash):
+```bash
+$ head -n1 output.jsonl | python -m json.tool
+```
+
+Example 4 (unknown):
+```unknown
+{
+ "segments": [
+ {
+ "label": true,
+ "text": "Hello\n"
+ },
+ {
+ "label": true,
+ "text": "hi there!. "
+ },
+ {
+ "label": false,
+ "text": "goodbye "
+ },
+ {
+ "label": true,
+ "text": "farewell"
+ }
+ ]
+}
+```
+
+---
+
+## Dataset Formats
+
+**URL:** https://docs.axolotl.ai/docs/dataset-formats/
+
+**Contents:**
+- Dataset Formats
+- Pre-training
+ - Pre-training from Hugging Face hub datasets
+ - Pre-training from local dataset files
+ - Pre-training without streaming
+ - Pre-training dataset configuration tips
+ - Setting max_steps
+ - Group_by_length
+ - Reference
+- Supervised fine-tuning (SFT)
+
+Axolotl is a training framework that aims to make the process convenient yet flexible to users by simply passing a config yaml file.
+
+As there are a lot of available options in Axolotl, this guide aims to provide an simplify the user experience to choosing the proper choice.
+
+Axolotl supports 3 kinds of training methods: pre-training, supervised fine-tuning, and preference-based post-training (e.g. DPO, ORPO, PRMs). Each method has their own dataset format which are described below.
+
+This guide will mainly use JSONL as an introduction. Please refer to the dataset loading docs to understand how to load datasets from other sources.
+
+For pretraining_dataset: specifically, please refer to the Pre-training section.
+
+When aiming to train on large corpora of text datasets, pre-training is your go-to choice. Due to the size of these datasets, downloading the entire-datasets before beginning training would be prohibitively time-consuming. Axolotl supports streaming to only load batches into memory at a time.
+
+A sample format for a pre-training dataset is as follows:
+
+It is typically recommended to save your dataset as .jsonl due to its flexibility and simplicity.
+
+Axolotl supports loading from a Hugging Face hub repo or from local files.
+
+As an example, to train using a Hugging Face dataset hf_org/name, you can pass the following config:
+
+Given a few corpus files: A.jsonl, B.jsonl, and C.jsonl, your config will look like the below:
+
+While we recommend .jsonl, you can also use the other formats (csv, parquet, arrow, SQL, Webdataset) that are supported by Dataset.load_dataset
+
+In the case that the dataset is small and can be loaded entirely into memory, another approach to running pre-training is to use the completion format. This would mean that the entire dataset is pre-tokenized instead of on-demand in streaming.
+
+One benefit of this is that the tokenization can be performed separately on a CPU-only machine, and then transferred to a GPU machine for training to save costs.
+
+For completion only, Axolotl would split texts if it exceeds the context length into multiple smaller prompts. If you are interested in having this for pretraining_dataset too, please let us know or help make a PR!
+
+When using streaming for large datasets, Axolotl does not know in advance how large the dataset is and does not know when to stop.
+
+Therefore, it is necessary to set max_steps: int in your config for pre-training to run, so that Axolotl knows when to stop training.
+
+One step is equal to sequence_len * micro_batch_size * gradient_accumulation_steps * total_num_gpus tokens.
+
+It is recommended to leave this off if downloading from Hugging Face hub as it would download the entire dataset which can be very large.
+
+Please see docs here.
+
+Supervised fine-tuning is the process of training models to respond to an instruction or chat input.
+
+As there are a wide variety of dataset formats, Axolotl tries to support a majority of the formats available in public datasets.
+
+Axolotl provides four approaches for loading datasets, however, it’s easier to work backwards from the dataset you have available to figure out which approach to use.
+
+A flow chart is as follows:
+
+Do you already have the dataset tokenized? If yes, check Pre-Tokenized Dataset.
+
+Do you want to format the dataset yourself and manually choose each section to mask? If yes, check Template Free Dataset
+
+Is your dataset in a “conversation” format, containing a list[messages]? If yes, check Conversation Dataset
+
+Is your dataset in an “instruct” format, containing { instruction, response }? If yes, check Instruction Dataset
+
+If you went through the flow chart and did not find one that matches, it is recommended to preprocess your dataset into one of the above or create a thread on Github Discussion.
+
+You can mix and match within each approach or across approaches to train a model on a variety of datasets.
+
+We suggest this approach when you want to bring your own tokenized dataset.
+
+Axolotl expects the dataset to have three keys:
+
+Make sure to add BOS/EOS tokens to your prompt and mask it appropriately.
+
+A config for this would look like:
+
+Reference: Pre-Tokenized Dataset Documentation.
+
+We recommend this approach when you want granular control over the prompt formatting, special tokens, and masking, whilst letting Axolotl handle the tokenization. This is very useful if your dataset has unique prompts that differ across samples and where one single general template wouldn’t suffice.
+
+In the example below, you could see that there is no proper structure. At the same time, it’s very flexible as there are no constraints on how your prompt can look.
+
+Each prompt must be have a key called segments which is a list of { text, label }.
+
+Reference: Template Free Documentation.
+
+conversation messages are a list of messages which usually contain a role and content key.
+
+Fun fact: Axolotl synonymously refers to “chat” messages as conversation messages due to how FastChat initially used this term to build a widely used fastchat conversation method for formatting chat messages prior to the creation of chat_templates.
+
+The current most popular and convenient method for inference is to use chat_templates for formatting prompts. Axolotl supports using chat_templates for training to ensure that the model performs in the same environment as in inference.
+
+Here’s a quick rundown on chat_template: A chat_template is a Jinja2 template which formats a list of messages into a prompt.
+
+An example of a prompt formatted into a popular template called ChatML can be seen below:
+
+Single prompt (pretty-printed):
+
+The ChatML template is as follows:
+
+The above prompt formatted into this template will result in:
+
+By using delimiters (<|im_start|> and <|im_end|>), a prompt separates different speakers which helps the model identify which portion belongs to whom.
+
+Older conversation datasets with the following format are colloquially called sharegpt datasets.
+
+Newer conversation datasets usually follow the OpenAI format.
+
+Axolotl supports both as well as allowing customization of any kind of key.
+
+To properly use this method, it is important to identify three things:
+
+Which chat_template would you use?
+
+What are the keys in your dataset, and what are the possible roles? For example, in OpenAI format, the keys would be messages, role, and content, respectively, whereas the possible roles are system, user, and assistant.
+
+What do you want to mask? For instance, only assistant messages, only last message, or nothing.
+
+There are a lot of chat_templates out there. Axolotl supports the common ones: supported chat templates. For example, to use ChatML, it would be chat_template: chatml.
+
+However, it is also possible to use the already configured template within the tokenizer by specifying chat_template: tokenizer_default. If you want a fallback (in case some tokenizer does not have it pre-configured), you can do chat_template: tokenizer_default_fallback_chatml to fallback to the ChatML template if a tokenizer template was not found.
+
+One last but powerful approach is to bring your own template. This can be set via:
+
+We currently default to OpenAI format for dataset keys, so if that’s your current dataset format, there’s nothing to do here.
+
+If your dataset format is different, here are the keys you should check (with their defaults):
+
+In some chat_templates (e.g. Gemma), the roles are hardcoded to user and assistant. Consequently, you may find it necessary to map the roles in your dataset to these above. We currently have some defaults that should work for common datasets, but if you get a KeyError, it would be necessary to add mapping for your roles. Here is an example of how it would look like:
+
+In the example above, all gpt and model values are converted to assistant. All human values are converted to user.
+
+The common use case for chat_template is for chat messages, therefore, it is common to mask all non-assistant messages. Assistant messages refer to the bot messages that you want the model to learn on.
+
+To train on all assistant messages, you would set the following configs.
+
+The train_on_eos config means that it would mask all EOS tokens for turns that aren’t assistant-turns. The other options are: all and last to choose which EOS to train on.
+
+Perhaps, you want to train on assistant and narrator roles, you can simply add narrator to the list of roles_to_train. You would also need to add it to the mapping of roles above.
+
+As chat_templates may use hardcoded EOS/EOT tokens that are different from the tokenizer’s EOS, it is highly recommended to set them. For example, ChatML uses <|im_end|> to end turns.
+
+Once all the above steps are completed, you could combine all these configs together to form a bespoke configuration for your custom dataset.
+
+If this config were to be applied to the sample dataset above, the output would look as such (which can be retrieved via axolotl preprocess config.yaml --debug):
+
+The first number refers to the label, the second refers to the token_id. For example, -100 labels appear on non-assistant portions, meaning that they are masked during. For assistant portions, the label is the same as the token_id.
+
+If during preprocess, there are a lot of warnings of Could not find content __ boundary, please check the FAQ section for chat_templates.
+
+Please see docs here.
+
+Instruction datasets are used to train instruction-following models and comprise a prompt, containing an instruction, and a single response. In contrast to chat datasets which may be multi-turn, instruct datasets are typically single-turn.
+
+An example is of a common format called Alpaca:
+
+Using those keys, a prompt can be built based on it.
+
+This can be configured as such:
+
+Axolotl supports many kinds of instruction dataset. All of them can be found in the Instruction Dataset Documentation with their respective type and sample row format.
+
+Due to the myriad possibilities of instruction formats, Axolotl allows customizing your own instruction format without having to dive into the code directly.
+
+In the example below, a sample row is used to output in mistral_v1 format.
+
+The config sets that the field_instruction is actually named input, and the field_input is empty as we don’t have an input in this sample. Generally, instruction can be thought as the question to the model, and input as the additional information with output being the response. It is not necessary to have an input nor system. In the end, the most important part is to understand what format you want it to look like and how you can customize this to your use case.
+
+Reference: Custom Instruct Prompt Format Documentation.
+
+As there are multiple RLHF methods with their own dataset requirements. Please see RLHF documentation for more detail.
+
+**Examples:**
+
+Example 1 (json):
+```json
+{"text": "first row"}
+{"text": "second row"}
+...
+```
+
+Example 2 (yaml):
+```yaml
+pretraining_dataset: hf_org/name
+```
+
+Example 3 (yaml):
+```yaml
+pretraining_dataset:
+ - path: json
+ data_files:
+ - A.jsonl
+ - B.jsonl
+ - C.jsonl
+```
+
+Example 4 (yaml):
+```yaml
+datasets:
+ - path: hf_org/name
+ type: completion
+```
+
+---
+
+## Dataset Formats
+
+**URL:** https://docs.axolotl.ai/docs/dataset-formats
+
+**Contents:**
+- Dataset Formats
+- Pre-training
+ - Pre-training from Hugging Face hub datasets
+ - Pre-training from local dataset files
+ - Pre-training without streaming
+ - Pre-training dataset configuration tips
+ - Setting max_steps
+ - Group_by_length
+ - Reference
+- Supervised fine-tuning (SFT)
+
+Axolotl is a training framework that aims to make the process convenient yet flexible to users by simply passing a config yaml file.
+
+As there are a lot of available options in Axolotl, this guide aims to provide an simplify the user experience to choosing the proper choice.
+
+Axolotl supports 3 kinds of training methods: pre-training, supervised fine-tuning, and preference-based post-training (e.g. DPO, ORPO, PRMs). Each method has their own dataset format which are described below.
+
+This guide will mainly use JSONL as an introduction. Please refer to the dataset loading docs to understand how to load datasets from other sources.
+
+For pretraining_dataset: specifically, please refer to the Pre-training section.
+
+When aiming to train on large corpora of text datasets, pre-training is your go-to choice. Due to the size of these datasets, downloading the entire-datasets before beginning training would be prohibitively time-consuming. Axolotl supports streaming to only load batches into memory at a time.
+
+A sample format for a pre-training dataset is as follows:
+
+It is typically recommended to save your dataset as .jsonl due to its flexibility and simplicity.
+
+Axolotl supports loading from a Hugging Face hub repo or from local files.
+
+As an example, to train using a Hugging Face dataset hf_org/name, you can pass the following config:
+
+Given a few corpus files: A.jsonl, B.jsonl, and C.jsonl, your config will look like the below:
+
+While we recommend .jsonl, you can also use the other formats (csv, parquet, arrow, SQL, Webdataset) that are supported by Dataset.load_dataset
+
+In the case that the dataset is small and can be loaded entirely into memory, another approach to running pre-training is to use the completion format. This would mean that the entire dataset is pre-tokenized instead of on-demand in streaming.
+
+One benefit of this is that the tokenization can be performed separately on a CPU-only machine, and then transferred to a GPU machine for training to save costs.
+
+For completion only, Axolotl would split texts if it exceeds the context length into multiple smaller prompts. If you are interested in having this for pretraining_dataset too, please let us know or help make a PR!
+
+When using streaming for large datasets, Axolotl does not know in advance how large the dataset is and does not know when to stop.
+
+Therefore, it is necessary to set max_steps: int in your config for pre-training to run, so that Axolotl knows when to stop training.
+
+One step is equal to sequence_len * micro_batch_size * gradient_accumulation_steps * total_num_gpus tokens.
+
+It is recommended to leave this off if downloading from Hugging Face hub as it would download the entire dataset which can be very large.
+
+Please see docs here.
+
+Supervised fine-tuning is the process of training models to respond to an instruction or chat input.
+
+As there are a wide variety of dataset formats, Axolotl tries to support a majority of the formats available in public datasets.
+
+Axolotl provides four approaches for loading datasets, however, it’s easier to work backwards from the dataset you have available to figure out which approach to use.
+
+A flow chart is as follows:
+
+Do you already have the dataset tokenized? If yes, check Pre-Tokenized Dataset.
+
+Do you want to format the dataset yourself and manually choose each section to mask? If yes, check Template Free Dataset
+
+Is your dataset in a “conversation” format, containing a list[messages]? If yes, check Conversation Dataset
+
+Is your dataset in an “instruct” format, containing { instruction, response }? If yes, check Instruction Dataset
+
+If you went through the flow chart and did not find one that matches, it is recommended to preprocess your dataset into one of the above or create a thread on Github Discussion.
+
+You can mix and match within each approach or across approaches to train a model on a variety of datasets.
+
+We suggest this approach when you want to bring your own tokenized dataset.
+
+Axolotl expects the dataset to have three keys:
+
+Make sure to add BOS/EOS tokens to your prompt and mask it appropriately.
+
+A config for this would look like:
+
+Reference: Pre-Tokenized Dataset Documentation.
+
+We recommend this approach when you want granular control over the prompt formatting, special tokens, and masking, whilst letting Axolotl handle the tokenization. This is very useful if your dataset has unique prompts that differ across samples and where one single general template wouldn’t suffice.
+
+In the example below, you could see that there is no proper structure. At the same time, it’s very flexible as there are no constraints on how your prompt can look.
+
+Each prompt must be have a key called segments which is a list of { text, label }.
+
+Reference: Template Free Documentation.
+
+conversation messages are a list of messages which usually contain a role and content key.
+
+Fun fact: Axolotl synonymously refers to “chat” messages as conversation messages due to how FastChat initially used this term to build a widely used fastchat conversation method for formatting chat messages prior to the creation of chat_templates.
+
+The current most popular and convenient method for inference is to use chat_templates for formatting prompts. Axolotl supports using chat_templates for training to ensure that the model performs in the same environment as in inference.
+
+Here’s a quick rundown on chat_template: A chat_template is a Jinja2 template which formats a list of messages into a prompt.
+
+An example of a prompt formatted into a popular template called ChatML can be seen below:
+
+Single prompt (pretty-printed):
+
+The ChatML template is as follows:
+
+The above prompt formatted into this template will result in:
+
+By using delimiters (<|im_start|> and <|im_end|>), a prompt separates different speakers which helps the model identify which portion belongs to whom.
+
+Older conversation datasets with the following format are colloquially called sharegpt datasets.
+
+Newer conversation datasets usually follow the OpenAI format.
+
+Axolotl supports both as well as allowing customization of any kind of key.
+
+To properly use this method, it is important to identify three things:
+
+Which chat_template would you use?
+
+What are the keys in your dataset, and what are the possible roles? For example, in OpenAI format, the keys would be messages, role, and content, respectively, whereas the possible roles are system, user, and assistant.
+
+What do you want to mask? For instance, only assistant messages, only last message, or nothing.
+
+There are a lot of chat_templates out there. Axolotl supports the common ones: supported chat templates. For example, to use ChatML, it would be chat_template: chatml.
+
+However, it is also possible to use the already configured template within the tokenizer by specifying chat_template: tokenizer_default. If you want a fallback (in case some tokenizer does not have it pre-configured), you can do chat_template: tokenizer_default_fallback_chatml to fallback to the ChatML template if a tokenizer template was not found.
+
+One last but powerful approach is to bring your own template. This can be set via:
+
+We currently default to OpenAI format for dataset keys, so if that’s your current dataset format, there’s nothing to do here.
+
+If your dataset format is different, here are the keys you should check (with their defaults):
+
+In some chat_templates (e.g. Gemma), the roles are hardcoded to user and assistant. Consequently, you may find it necessary to map the roles in your dataset to these above. We currently have some defaults that should work for common datasets, but if you get a KeyError, it would be necessary to add mapping for your roles. Here is an example of how it would look like:
+
+In the example above, all gpt and model values are converted to assistant. All human values are converted to user.
+
+The common use case for chat_template is for chat messages, therefore, it is common to mask all non-assistant messages. Assistant messages refer to the bot messages that you want the model to learn on.
+
+To train on all assistant messages, you would set the following configs.
+
+The train_on_eos config means that it would mask all EOS tokens for turns that aren’t assistant-turns. The other options are: all and last to choose which EOS to train on.
+
+Perhaps, you want to train on assistant and narrator roles, you can simply add narrator to the list of roles_to_train. You would also need to add it to the mapping of roles above.
+
+As chat_templates may use hardcoded EOS/EOT tokens that are different from the tokenizer’s EOS, it is highly recommended to set them. For example, ChatML uses <|im_end|> to end turns.
+
+Once all the above steps are completed, you could combine all these configs together to form a bespoke configuration for your custom dataset.
+
+If this config were to be applied to the sample dataset above, the output would look as such (which can be retrieved via axolotl preprocess config.yaml --debug):
+
+The first number refers to the label, the second refers to the token_id. For example, -100 labels appear on non-assistant portions, meaning that they are masked during. For assistant portions, the label is the same as the token_id.
+
+If during preprocess, there are a lot of warnings of Could not find content __ boundary, please check the FAQ section for chat_templates.
+
+Please see docs here.
+
+Instruction datasets are used to train instruction-following models and comprise a prompt, containing an instruction, and a single response. In contrast to chat datasets which may be multi-turn, instruct datasets are typically single-turn.
+
+An example is of a common format called Alpaca:
+
+Using those keys, a prompt can be built based on it.
+
+This can be configured as such:
+
+Axolotl supports many kinds of instruction dataset. All of them can be found in the Instruction Dataset Documentation with their respective type and sample row format.
+
+Due to the myriad possibilities of instruction formats, Axolotl allows customizing your own instruction format without having to dive into the code directly.
+
+In the example below, a sample row is used to output in mistral_v1 format.
+
+The config sets that the field_instruction is actually named input, and the field_input is empty as we don’t have an input in this sample. Generally, instruction can be thought as the question to the model, and input as the additional information with output being the response. It is not necessary to have an input nor system. In the end, the most important part is to understand what format you want it to look like and how you can customize this to your use case.
+
+Reference: Custom Instruct Prompt Format Documentation.
+
+As there are multiple RLHF methods with their own dataset requirements. Please see RLHF documentation for more detail.
+
+**Examples:**
+
+Example 1 (json):
+```json
+{"text": "first row"}
+{"text": "second row"}
+...
+```
+
+Example 2 (yaml):
+```yaml
+pretraining_dataset: hf_org/name
+```
+
+Example 3 (yaml):
+```yaml
+pretraining_dataset:
+ - path: json
+ data_files:
+ - A.jsonl
+ - B.jsonl
+ - C.jsonl
+```
+
+Example 4 (yaml):
+```yaml
+datasets:
+ - path: hf_org/name
+ type: completion
+```
+
+---
+
+## Instruction Tuning
+
+**URL:** https://docs.axolotl.ai/docs/dataset-formats/inst_tune.html
+
+**Contents:**
+- Instruction Tuning
+- alpaca
+- jeopardy
+- oasst
+- gpteacher
+- reflection
+- explainchoice
+- concisechoice
+- summarizetldr
+- alpaca_chat
+
+instruction; input(optional)
+
+instruction; input(optional)
+
+instruction with reflect; input(optional)
+
+question, choices, (solution OR explanation)
+
+question, choices, (solution OR explanation)
+
+basic instruct for alpaca chat
+
+question and answer for alpaca chat
+
+question and answer for alpaca chat, for concise answers
+
+question and answer for alpaca chat, for load_camel_ai
+
+support for open orca datasets with included system prompts, instruct
+
+in context question answering from an article
+
+in context question answering (alternate)
+
+in context question answering from an article, with default response for no answer from context
+
+instruction and revision
+
+instruction, adds additional eos tokens
+
+For a dataset that is preprocessed for instruction purposes:
+
+You can use this example in your YAML config:
+
+See full config options under here.
+
+**Examples:**
+
+Example 1 (json):
+```json
+{"instruction": "...", "input": "...", "output": "..."}
+```
+
+Example 2 (json):
+```json
+{"question": "...", "category": "...", "answer": "..."}
+```
+
+Example 3 (json):
+```json
+{"INSTRUCTION": "...", "RESPONSE": "..."}
+```
+
+Example 4 (json):
+```json
+{"instruction": "...", "input": "...", "response": "..."}
+```
+
+---
+
+## Stepwise Supervised Format
+
+**URL:** https://docs.axolotl.ai/docs/dataset-formats/stepwise_supervised.html
+
+**Contents:**
+- Stepwise Supervised Format
+- Stepwise Supervised
+ - Example
+
+The stepwise supervised format is designed for chain-of-thought (COT) reasoning datasets where each example contains multiple completion steps and a preference label for each step.
+
+Here’s a simple example of a stepwise supervised dataset entry:
+
+**Examples:**
+
+Example 1 (json):
+```json
+{
+ "prompt": "Which number is larger, 9.8 or 9.11?",
+ "completions": [
+ "The fractional part of 9.8 is 0.8, while the fractional part of 9.11 is 0.11.",
+ "Since 0.11 is greater than 0.8, the number 9.11 is larger than 9.8."
+ ],
+ "labels": [true, false]
+}
+```
+
+---
diff --git a/src/skills/bundled/axolotl/references/index.md b/src/skills/bundled/axolotl/references/index.md
new file mode 100644
index 0000000..2f2acb1
--- /dev/null
+++ b/src/skills/bundled/axolotl/references/index.md
@@ -0,0 +1,15 @@
+# Axolotl Documentation Index
+
+## Categories
+
+### Api
+**File:** `api.md`
+**Pages:** 150
+
+### Dataset-Formats
+**File:** `dataset-formats.md`
+**Pages:** 9
+
+### Other
+**File:** `other.md`
+**Pages:** 26
diff --git a/src/skills/bundled/axolotl/references/other.md b/src/skills/bundled/axolotl/references/other.md
new file mode 100644
index 0000000..2b4d2f7
--- /dev/null
+++ b/src/skills/bundled/axolotl/references/other.md
@@ -0,0 +1,3563 @@
+# Axolotl - Other
+
+**Pages:** 26
+
+---
+
+## Mixed Precision Training
+
+**URL:** https://docs.axolotl.ai/docs/mixed_precision.html
+
+**Contents:**
+- Mixed Precision Training
+- 1 FP16 Mixed Precision
+ - 1.1 Overview
+ - 1.2 Configuration
+ - 1.3 FP16 Considerations
+- 2 BF16 Mixed Precision
+ - 2.1 Overview
+ - 2.2 Configuration
+- 3 FP8 Mixed Precision
+ - 3.1 What is FP8?
+
+Mixed precision training uses lower precision data types to reduce memory usage and increase training speed while maintaining model quality. Axolotl supports several mixed precision formats:
+
+FP16 is the traditional half-precision format, supported on older GPUs but can be less numerically stable than BF16.
+
+BF16 (Brain Float 16) offers better numerical stability than FP16 and is the recommended mixed precision format for modern GPUs. It provides the same dynamic range as FP32 while using half the memory.
+
+FP8 support is experimental and requires compatible hardware (H100, H200) and recent PyTorch versions with TorchAO.
+
+FP8 (8-bit floating point) can provide significant time savings compared to FP16/BF16 while maintaining training stability. Axolotl’s implementation uses PyTorch’s TorchAO library with “tensorwise” scaling strategy.
+
+Add to your YAML config:
+
+torch.compile is critical for FP8 performance
+
+FP8 training requires torch_compile: true to see meaningful speedups. Without compilation, FP8 may actually be slower and use more memory than FP16/BF16.
+
+For FSDP (Fully Sharded Data Parallel) training:
+
+Always validate your mixed precision setup:
+
+See examples/llama-3/3b-fp8-fsdp2.yaml for an optimized example config. Enabling FP8 mixed precision + FP8 all-gather training results in ~10% faster iterations per second vs. BF16 for a relatively small (3B param) model
+
+For more information on multi-GPU training, see our Multi-GPU guide.
+
+**Examples:**
+
+Example 1 (yaml):
+```yaml
+# Automatic BF16 detection (recommended)
+bf16: auto
+
+# Or explicitly enable
+bf16: true
+
+# For evaluation with BF16
+bf16: full # Equivalent to bf16_full_eval in the HF trainer
+```
+
+Example 2 (yaml):
+```yaml
+# Enable FP8 mixed precision
+fp8: true
+
+# Optional: Enable FP8 for FSDP all-gather operations
+fp8_enable_fsdp_float8_all_gather: true
+
+# Enable torch.compile (almost always necessary for FP8 speedups)
+torch_compile: true
+```
+
+Example 3 (yaml):
+```yaml
+fp8: true
+fp8_enable_fsdp_float8_all_gather: true
+
+torch_compile: true
+
+# FSDP configuration
+fsdp_version: 2
+fsdp_config:
+ offload_params: false
+ cpu_ram_efficient_loading: true
+ auto_wrap_policy: TRANSFORMER_BASED_WRAP
+ transformer_layer_cls_to_wrap: LlamaDecoderLayer
+ state_dict_type: FULL_STATE_DICT
+ reshard_after_forward: true
+```
+
+---
+
+## FAQ
+
+**URL:** https://docs.axolotl.ai/docs/faq.html
+
+**Contents:**
+- FAQ
+ - General
+ - Chat templates
+
+Q: The trainer stopped and hasn’t progressed in several minutes.
+
+A: Usually an issue with the GPUs communicating with each other. See the NCCL doc
+
+A: This usually happens when you run out of system RAM.
+
+Q: exitcode: -7 while using deepspeed
+
+A: Try upgrading deepspeed w: pip install -U deepspeed
+
+Q: AttributeError: ‘DummyOptim’ object has no attribute ‘step’
+
+Q: ModuleNotFoundError: No module named ‘mpi4py’ using single GPU with deepspeed
+
+A: You may be using deepspeed with single gpu. Please remove the deepspeed: section in the yaml file or --deepspeed CLI flag.
+
+Q: The codes is stuck on saving preprocessed datasets.
+
+A: This is usually an issue with the GPU. This can be resolved through setting the os environment variable CUDA_VISIBLE_DEVICES=0. If you are on runpod, this is usually a pod issue. Starting a new pod should take care of it.
+
+Q: Received mismatch error on merge adapters / loading adapters between torch.Size of checkpoint and model.
+
+A: This is likely due to vocab size mismatch. By default, Axolotl expands the model’s embeddings if the tokenizer has more tokens than the model. Please use the axolotl merge-lora command to merge the adapters instead of using your own scripts.
+
+On the other hand, if the model has more tokens than the tokenizer, Axolotl does not shrink the model’s embeddings unless shrink_embeddings: true is set in the config.
+
+Q: How to call Axolotl via custom python scripts?
+
+A: Since Axolotl is just Python, please see src/axolotl/cli/main.py on how each command is called.
+
+Q: How to know the value to use for fsdp_transformer_layer_cls_to_wrap?
+
+A: This is the class name of the transformer layer to wrap with FSDP. For example, for LlamaForCausalLM, the value is LlamaDecoderLayer. To find this for a specific model, check the model’s PreTrainedModel definition and look for _no_split_modules variable in the modeling_.py file within transformers library.
+
+Q: ValueError: Asking to pad but the tokenizer does not have a padding token. Please select a token to use as pad_token
+
+A: This is because the tokenizer does not have a padding token. Please add a padding token to the tokenizer via:
+
+Q: IterableDataset error or KeyError: 'input_ids' when using preprocess CLI
+
+A: This is because you may be using preprocess CLI with pretraining_dataset: or skip_prepare_dataset: true respectively. Please use axolotl train CLI directly instead as these datasets are prepared on demand.
+
+Q: vLLM is not working with Axolotl
+
+A: We currently recommend torch 2.6.0 for use with vllm. Please ensure you use the right version. For Docker, please use the main-py3.11-cu124-2.6.0 tag.
+
+Q: FA2 2.8.0 undefined symbol runtime error on CUDA 12.4
+
+A: There seems to be a wheel issue with FA2 2.8.0 on CUDA 12.4. Try CUDA 12.6 instead or downgrade to FA2 2.7.4. Please refer to the upstream issue: https://github.com/Dao-AILab/flash-attention/issues/1717.
+
+Q: Can we mix text and text+image datasets for VLM training?
+
+A: Yes, you can for newer VLM arch. The ones that would not work are LLaVA / Pixtral arch. If you notice one not working, please let us know!
+
+Q: Why is memory/max_* different from nvidia-smi?
+
+A: We use torch APIs to retrieve this information. You can see https://docs.pytorch.org/docs/stable/notes/cuda.html#cuda-memory-management for more information.
+
+Q: jinja2.exceptions.UndefinedError: 'dict object' has no attribute 'content' / 'role' / ____
+
+A: This means that the property mapping for the stated attribute does not exist when building chat_template prompt. For example, if no attribute 'content', please check you have added the correct mapping for content under message_property_mappings.
+
+Q: Empty template generated for turn ___
+
+A: The content is empty for that turn.
+
+Q: Could not find content start/end boundary for turn __
+
+A: The specific turn’s start/end could not be detected. Please ensure you have set the eos_token following your chat_template. Otherwise, this could be a chat_template which doesn’t use proper boundaries for each turn (like system). On the rare occurrence, make sure your content is not [[dummy_message]]. Please let us know about this.
+
+Q: Content end boundary is before start boundary for turn ___
+
+A: This is an edge case which should not occur. Please create an Issue if this happens.
+
+Q: Content end boundary is the same as start boundary for turn ___. This is likely an empty turn.
+
+A: This is likely an empty turn.
+
+Q: The EOS token is incorrectly being masked or not being masked / EOS token __ not found in chat template.
+
+A: There can be two reasons:
+
+Q: “chat_template choice is tokenizer_default but tokenizer’s chat_template is null. Please add a chat_template in tokenizer config”
+
+A: This is because the tokenizer does not have a chat template. Please add a chat template in the tokenizer config. See chat_template for more details.
+
+Q: The EOT token(s) are incorrectly being masked or not being masked / EOT token __ not found in chat template.
+
+A: There can be two reasons:
+
+Q: EOT token encoding failed. Please check if the token is valid and can be encoded.
+
+A: There could be some issue with the tokenizer or unicode encoding. Please raise an issue with examples with the EOT token & tokenizer causing the issue.
+
+Q: EOT token __ is encoded as multiple tokens.
+
+A: This is because the EOT token is encoded as multiple tokens which can cause unexpected behavior. Please add it under tokens: or (recommended) override unused added_tokens via added_tokens_overrides:.
+
+Q: Conflict between train_on_eos and train_on_eot. eos_token is in eot_tokens and train_on_eos != train_on_eot
+
+A: This is because the EOS token is in the eot_tokens: while mismatch between train_on_eos: and train_on_eot:. This will cause one to override the other. Please ensure that train_on_eos: and train_on_eot: are the same or remove the EOS token from eot_tokens:.
+
+Q: If eot_tokens: is not provided, what happens?
+
+A: If eot_tokens: is not provided, the default behavior is the same as before. EOS tokens used to delimit turns are masked/unmasked depending on whether the turn is trainable.
+
+Internally, eot_tokens: tokenizer.eos_token and train_on_eot: train_on_eos (which defaults to turn). This transition helps clarify the naming and behavior of EOT/EOS tokens.
+
+Q: Data processing error: CAS service error
+
+A: Try disabling XET with export HF_HUB_DISABLE_XET=1
+
+Q: torch._inductor.exc.LoweringException: NoValidChoicesError: No choices to select, please consider adding ATEN into max_autotune_gemm_backends config (defined in torch/_inductor/config.py) to allow at least one choice.
+
+A: Depending on the version of torch, you may need to include this in your YAML:
+
+**Q: ValueError("Backward pass should have cleared tracker of all tensors")
+
+A: This may happen due to edge cases in using the modern OffloadActivations context manager for CUDA streams. If you encounter this error, you may have success using the naive implementation with offload_activations: legacy in your YAML.
+
+**Q: Error parsing tool_calls arguments as JSON.
+
+A: There is an error parsing string arguments to a dict. Please check your dataset and the error message for more details.
+
+**Examples:**
+
+Example 1 (yaml):
+```yaml
+special_tokens:
+ # str. If you're not sure, set to same as `eos_token`.
+ pad_token: "..."
+```
+
+Example 2 (yaml):
+```yaml
+flex_attn_compile_kwargs:
+ dynamic: false
+ mode: max-autotune-no-cudagraphs
+```
+
+---
+
+## Installation
+
+**URL:** https://docs.axolotl.ai/docs/installation.html
+
+**Contents:**
+- Installation
+- 1 Requirements
+- 2 Installation Methods
+ - 2.1 PyPI Installation (Recommended)
+ - 2.2 uv Installation
+ - 2.3 Edge/Development Build
+ - 2.4 Docker
+- 3 Cloud Environments
+ - 3.1 Cloud GPU Providers
+ - 3.2 Google Colab
+
+This guide covers all the ways you can install and set up Axolotl for your environment.
+
+Please make sure to have Pytorch installed before installing Axolotl in your local environment.
+
+Follow the instructions at: https://pytorch.org/get-started/locally/
+
+For Blackwell GPUs, please use Pytorch 2.7.0 and CUDA 12.8.
+
+We use --no-build-isolation in order to detect the installed PyTorch version (if installed) in order not to clobber it, and so that we set the correct version of dependencies that are specific to the PyTorch version or other installed co-dependencies.
+
+uv is a fast, reliable Python package installer and resolver built in Rust. It offers significant performance improvements over pip and provides better dependency resolution, making it an excellent choice for complex environments.
+
+Install uv if not already installed
+
+Choose your CUDA version to use with PyTorch; e.g. cu124, cu126, cu128, then create the venv and activate
+
+Install PyTorch - PyTorch 2.6.0 recommended
+
+Install axolotl from PyPi
+
+For the latest features between releases:
+
+For development with Docker:
+
+For Blackwell GPUs, please use axolotlai/axolotl:main-py3.11-cu128-2.7.0 or the cloud variant axolotlai/axolotl-cloud:main-py3.11-cu128-2.7.0.
+
+Please refer to the Docker documentation for more information on the different Docker images that are available.
+
+For providers supporting Docker:
+
+See Section 6 for Mac-specific issues.
+
+We recommend using WSL2 (Windows Subsystem for Linux) or Docker.
+
+Install PyTorch: https://pytorch.org/get-started/locally/
+
+(Optional) Login to Hugging Face:
+
+If you encounter installation issues, see our FAQ and Debugging Guide.
+
+**Examples:**
+
+Example 1 (bash):
+```bash
+pip3 install -U packaging setuptools wheel ninja
+pip3 install --no-build-isolation axolotl[flash-attn,deepspeed]
+```
+
+Example 2 (bash):
+```bash
+curl -LsSf https://astral.sh/uv/install.sh | sh
+source $HOME/.local/bin/env
+```
+
+Example 3 (bash):
+```bash
+export UV_TORCH_BACKEND=cu126
+uv venv --no-project --relocatable
+source .venv/bin/activate
+```
+
+Example 4 (bash):
+```bash
+uv pip install packaging setuptools wheel
+uv pip install torch==2.6.0
+uv pip install awscli pydantic
+```
+
+---
+
+## Dataset Preprocessing
+
+**URL:** https://docs.axolotl.ai/docs/dataset_preprocessing.html
+
+**Contents:**
+- Dataset Preprocessing
+- Overview
+ - What are the benefits of pre-processing?
+ - What are the edge cases?
+
+Dataset pre-processing is the step where Axolotl takes each dataset you’ve configured alongside the dataset format and prompt strategies to:
+
+The processing of the datasets can happen one of two ways:
+
+When training interactively or for sweeps (e.g. you are restarting the trainer often), processing the datasets can oftentimes be frustratingly slow. Pre-processing will cache the tokenized/formatted datasets according to a hash of dependent training parameters so that it will intelligently pull from its cache when possible.
+
+The path of the cache is controlled by dataset_prepared_path: and is often left blank in example YAMLs as this leads to a more robust solution that prevents unexpectedly reusing cached data.
+
+If dataset_prepared_path: is left empty, when training, the processed dataset will be cached in a default path of ./last_run_prepared/, but will ignore anything already cached there. By explicitly setting dataset_prepared_path: ./last_run_prepared, the trainer will use whatever pre-processed data is in the cache.
+
+Let’s say you are writing a custom prompt strategy or using a user-defined prompt template. Because the trainer cannot readily detect these changes, we cannot change the calculated hash value for the pre-processed dataset.
+
+If you have dataset_prepared_path: ... set and change your prompt templating logic, it may not pick up the changes you made and you will be training over the old prompt.
+
+---
+
+## Inference and Merging
+
+**URL:** https://docs.axolotl.ai/docs/inference.html
+
+**Contents:**
+- Inference and Merging
+- 1 Quick Start
+ - 1.1 Basic Inference
+- 2 Advanced Usage
+ - 2.1 Gradio Interface
+ - 2.2 File-based Prompts
+ - 2.3 Memory Optimization
+- 3 Merging LoRA Weights
+ - 3.1 Memory Management for Merging
+- 4 Tokenization
+
+This guide covers how to use your trained models for inference, including model loading, interactive testing, merging adapters, and common troubleshooting steps.
+
+Use the same config used for training on inference/merging.
+
+Launch an interactive web interface:
+
+Process prompts from a text file:
+
+For large models or limited memory:
+
+Merge LoRA adapters with the base model:
+
+Tokenization mismatches between training and inference are a common source of problems.
+
+Verify inference tokenization by decoding tokens before model input
+
+Compare token IDs between training and inference
+
+Configure special tokens in your YAML:
+
+For more details, see our debugging guide.
+
+**Examples:**
+
+Example 1 (bash):
+```bash
+axolotl inference your_config.yml --lora-model-dir="./lora-output-dir"
+```
+
+Example 2 (bash):
+```bash
+axolotl inference your_config.yml --base-model="./completed-model"
+```
+
+Example 3 (bash):
+```bash
+axolotl inference your_config.yml --gradio
+```
+
+Example 4 (bash):
+```bash
+cat /tmp/prompt.txt | axolotl inference your_config.yml \
+ --base-model="./completed-model" --prompter=None
+```
+
+---
+
+## MultiModal / Vision Language Models (BETA)
+
+**URL:** https://docs.axolotl.ai/docs/multimodal.html
+
+**Contents:**
+- MultiModal / Vision Language Models (BETA)
+- Supported Models
+- Usage
+ - Mllama
+ - Llama4
+ - Pixtral
+ - Llava-1.5
+ - Mistral-Small-3.1
+ - Magistral-Small-2509
+ - Voxtral
+
+Multimodal support is limited and doesn’t have full feature parity.
+
+Here are the hyperparams you’ll need to use to finetune a multimodal model.
+
+Please see examples folder for full configs.
+
+Some of our chat_templates have been extended to support broader dataset types. This should not break any existing configs.
+
+As of now, we do not truncate nor drop samples based on sequence_len as each arch has different ways to process non-text tokens. We are looking for help on this.
+
+Please make sure to install vision lib via pip install 'mistral-common[opencv]==1.8.5'
+
+Please make sure to install vision lib via pip install 'mistral-common[opencv]==1.8.5'
+
+Please make sure to install audio lib via pip3 install librosa==0.11.0 'mistral_common[audio]==1.8.3'
+
+The Gemma3-1B model is a text-only model, so please train as regular text model.
+
+For multi-modal 4B/12B/27B models, use the following config:
+
+The model’s initial loss and grad norm will be very high. We suspect this to be due to the Conv in the vision layers.
+
+Please make sure to install timm via pip3 install timm==1.0.17
+
+Please make sure to install num2words via pip3 install num2words==0.5.14
+
+Please uninstall causal-conv1d via pip3 uninstall -y causal-conv1d
+
+For multi-modal datasets, we adopt an extended chat_template format similar to OpenAI’s Message format.
+
+For backwards compatibility:
+
+For image loading, you can use the following keys within content alongside "type": "image":
+
+For audio loading, you can use the following keys within content alongside "type": "audio":
+
+You may need to install librosa via pip3 install librosa==0.11.0.
+
+This is not well tested at the moment. We welcome contributors!
+
+For video loading, you can use the following keys within content alongside "type": "video":
+
+Here is an example of a multi-modal dataset:
+
+PIL could not retrieve the file at url using requests. Please check for typo. One alternative reason is that the request is blocked by the server.
+
+**Examples:**
+
+Example 1 (yaml):
+```yaml
+processor_type: AutoProcessor
+
+skip_prepare_dataset: true
+remove_unused_columns: false # leave columns in place as they are needed to handle image embeddings during training
+sample_packing: false # not yet supported with multimodal
+
+chat_template: # see in next section if specified
+
+# example dataset
+datasets:
+ - path: HuggingFaceH4/llava-instruct-mix-vsft
+ type: chat_template
+ split: train[:1%]
+
+# (optional) if doing lora, only finetune the Language model,
+# leave the vision model and vision tower frozen
+# load_in_8bit: true
+adapter: lora
+lora_target_modules: 'model.language_model.layers.[\d]+.(mlp|cross_attn|self_attn).(up|down|gate|q|k|v|o)_proj'
+
+# (optional) if you want to resize images to a set size
+image_size: 512
+image_resize_algorithm: bilinear
+```
+
+Example 2 (yaml):
+```yaml
+base_model: meta-llama/Llama-3.2-11B-Vision-Instruct
+
+chat_template: llama3_2_vision
+```
+
+Example 3 (yaml):
+```yaml
+base_model: meta-llama/Llama-4-Scout-17B-16E-Instruct
+
+chat_template: llama4
+```
+
+Example 4 (yaml):
+```yaml
+base_model: mistralai/Pixtral-12B-2409
+
+chat_template: pixtral
+```
+
+---
+
+## Reward Modelling
+
+**URL:** https://docs.axolotl.ai/docs/reward_modelling.html
+
+**Contents:**
+- Reward Modelling
+ - Overview
+ - (Outcome) Reward Models
+ - Process Reward Models (PRM)
+
+Reward modelling is a technique used to train models to predict the reward or value of a given input. This is particularly useful in reinforcement learning scenarios where the model needs to evaluate the quality of its actions or predictions. We support the reward modelling techniques supported by trl.
+
+Outcome reward models are trained using data which contains preference annotations for an entire interaction between the user and model (e.g. rather than per-turn or per-step). For improved training stability, you can use the center_rewards_coefficient parameter to encourage mean-zero reward outputs (see TRL docs).
+
+Bradley-Terry chat templates expect single-turn conversations in the following format:
+
+Check out our PRM blog.
+
+Process reward models are trained using data which contains preference annotations for each step in a series of interactions. Typically, PRMs are trained to provide reward signals over each step of a reasoning trace and are used for downstream reinforcement learning.
+
+Please see stepwise_supervised for more details on the dataset format.
+
+**Examples:**
+
+Example 1 (yaml):
+```yaml
+base_model: google/gemma-2-2b
+model_type: AutoModelForSequenceClassification
+num_labels: 1
+tokenizer_type: AutoTokenizer
+
+reward_model: true
+chat_template: gemma
+datasets:
+ - path: argilla/distilabel-intel-orca-dpo-pairs
+ type: bradley_terry.chat_template
+
+val_set_size: 0.1
+eval_steps: 100
+```
+
+Example 2 (json):
+```json
+{
+ "system": "...", // optional
+ "input": "...",
+ "chosen": "...",
+ "rejected": "..."
+}
+```
+
+Example 3 (yaml):
+```yaml
+base_model: Qwen/Qwen2.5-3B
+model_type: AutoModelForTokenClassification
+num_labels: 2
+
+process_reward_model: true
+datasets:
+ - path: trl-lib/math_shepherd
+ type: stepwise_supervised
+ split: train
+
+val_set_size: 0.1
+eval_steps: 100
+```
+
+---
+
+## RLHF (Beta)
+
+**URL:** https://docs.axolotl.ai/docs/rlhf.html
+
+**Contents:**
+- RLHF (Beta)
+- Overview
+- RLHF using Axolotl
+ - DPO
+ - chatml.argilla
+ - chatml.argilla_chat
+ - chatml.icr
+ - chatml.intel
+ - chatml.prompt_pairs
+ - chatml.ultra
+
+Reinforcement Learning from Human Feedback is a method whereby a language model is optimized from data using human feedback. Various methods include, but not limited to:
+
+This is a BETA feature and many features are not fully implemented. You are encouraged to open new PRs to improve the integration and functionality.
+
+We rely on the TRL library for implementations of various RL training methods, which we wrap around to expose in axolotl. Each method has their own supported ways of loading datasets and prompt formats.
+
+You can find what each method supports by going into src/axolotl/prompt_strategies/{method} where {method} is one of our supported methods. The type: can be retrieved from {method}.{function_name}.
+
+DPO supports the following types with the following dataset format:
+
+For custom behaviors,
+
+The input format is a simple JSON input with customizable fields based on the above config.
+
+As IPO is just DPO with a different loss function, all supported dataset formats for DPO are also supported for IPO.
+
+Paper: https://arxiv.org/abs/2403.07691
+
+ORPO supports the following types with the following dataset format:
+
+KTO supports the following types with the following dataset format:
+
+For custom behaviors,
+
+The input format is a simple JSON input with customizable fields based on the above config.
+
+Check out our GRPO cookbook.
+
+In the latest GRPO implementation, vLLM is used to significantly speedup trajectory generation during training. In this example, we’re using 4 GPUs - 2 for training, and 2 for vLLM:
+
+Make sure you’ve installed the correct version of vLLM by including it as an extra when installing axolotl, e.g. pip install axolotl[vllm].
+
+Your vLLM instance will now attempt to spin up, and it’s time to kick off training utilizing our remaining two GPUs. In another terminal, execute:
+
+Due to TRL’s implementation with vLLM, the vLLM instance must use the last N GPUs instead of the first N GPUs. This is why in the example above, we use CUDA_VISIBLE_DEVICES=2,3 for the vLLM instance.
+
+GRPO uses custom reward functions and transformations. Please have them ready locally.
+
+For example, to load OpenAI’s GSM8K and use a random reward for completions:
+
+To see other examples of custom reward functions, please see TRL GRPO Docs.
+
+To see all configs, please see TRLConfig.
+
+The DAPO paper and subsequently Dr. GRPO paper proposed an alternative loss function for GRPO to remediate the penalty in longer responses.
+
+For more information, see GRPO docs.
+
+SimPO uses CPOTrainer but with alternative loss function.
+
+This method uses the same dataset format as DPO.
+
+TRL supports auto-unwrapping PEFT models for RL training paradigms which rely on a reference model. This significantly reduces memory pressure as an additional refreference model does not need to be loaded, and reference model log-probabilities can be obtained by disabling PEFT adapters. This is enabled by default. To turn it off, pass the following config:
+
+**Examples:**
+
+Example 1 (yaml):
+```yaml
+rl: dpo
+datasets:
+ - path: Intel/orca_dpo_pairs
+ split: train
+ type: chatml.intel
+ - path: argilla/ultrafeedback-binarized-preferences
+ split: train
+ type: chatml
+```
+
+Example 2 (json):
+```json
+{
+ "system": "...", // optional
+ "instruction": "...",
+ "chosen_response": "...",
+ "rejected_response": "..."
+}
+```
+
+Example 3 (json):
+```json
+{
+ "chosen": [
+ {"role": "user", "content": "..."},
+ {"role": "assistant", "content": "..."}
+ ],
+ "rejected": [
+ {"role": "user", "content": "..."},
+ {"role": "assistant", "content": "..."}
+ ]
+}
+```
+
+Example 4 (json):
+```json
+{
+ "system": "...", // optional
+ "input": "...",
+ "chosen": "...",
+ "rejected": "..."
+}
+```
+
+---
+
+## LoRA Optimizations
+
+**URL:** https://docs.axolotl.ai/docs/lora_optims.html
+
+**Contents:**
+- LoRA Optimizations
+- Usage
+- Requirements
+- Implementation details
+ - Custom autograd functions
+ - Triton kernels
+ - Integration
+- Future Work
+
+Inspired by Unsloth, we’ve implemented two optimizations for LoRA and QLoRA fine-tuning, supporting both single GPU and multi-GPU (including the DDP, DeepSpeed, and FSDP2 settings) training. These include (1) SwiGLU and GEGLU activation function Triton kernels, and (2) LoRA MLP and attention custom autograd functions. Our goal was to leverage operator fusion and tensor re-use in order to improve speed and reduce memory usage during the forward and backward passes of these calculations.
+
+We currently support several common model architectures, including (but not limited to):
+
+The set of models we support is currently limited by our attention patching strategy, which assumes (and replaces) specific code blocks for query / key / value and output projections:
+
+Where apply_qkv and apply_o are defined in the axolotl.kernels.lora module.
+
+We welcome testing of other model architectures and / or PRs to expand our patching logic to be compatible with more of them.
+
+Check out our LoRA optimizations blog.
+
+These optimizations can be enabled in your Axolotl config YAML file. The lora_mlp_kernel option enables the optimized MLP path, while lora_qkv_kernel and lora_o_kernel enable the fused query-key-value projection and optimized output projection, respectively.
+
+Currently, LoRA kernels are not supported for RLHF training, only SFT.
+
+Models with pre-existing LoRA adapters that use Dropout or have bias terms may need to be re-finetuned without these features in order to be useful.
+
+The LoRA MLP autograd function optimizes the entire MLP computation path. It fuses the LoRA and base weight computations together and provides a single, efficient backward pass for the entire MLP block.
+
+For attention components, similar optimizations are provided through a function that handles the query, key, and value projections, and a function that handles the output projection. They are designed to work with the existing transformers attention implementation via some monkey-patching logic.
+
+Two activation functions (SwiGLU and GeGLU) are implemented with Triton kernels for improved speed and memory performance. These kernels handle both the forward and backward passes.
+
+The custom autograd functions and Triton kernels are designed to work together. The autograd function manages the high-level computation flow and gradient tracking, while calling the Triton kernels for the activation function computation. During the backward pass, the kernel computes both the activation output and the required gradients, which the autograd function then uses to compute the final gradients for the entire computation path.
+
+**Examples:**
+
+Example 1 (python):
+```python
+ORIGINAL_QKV_CODE = """
+ query_states = self.q_proj(hidden_states).view(hidden_shape).transpose(1, 2)
+ key_states = self.k_proj(hidden_states).view(hidden_shape).transpose(1, 2)
+ value_states = self.v_proj(hidden_states).view(hidden_shape).transpose(1, 2)
+""".lstrip(
+ "\n"
+)
+
+ORIGINAL_O_CODE = """
+ attn_output = self.o_proj(attn_output)
+""".lstrip(
+ "\n"
+)
+```
+
+Example 2 (python):
+```python
+PATCHED_QKV_CODE = """
+ query_states, key_states, value_states = self.apply_qkv(hidden_states)
+ query_states = query_states.view(hidden_shape).transpose(1, 2)
+ key_states = key_states.view(hidden_shape).transpose(1, 2)
+ value_states = value_states.view(hidden_shape).transpose(1, 2)
+""".lstrip(
+ "\n"
+)
+
+PATCHED_O_CODE = """
+ attn_output = self.apply_o(attn_output)
+""".lstrip(
+ "\n"
+)
+```
+
+Example 3 (yaml):
+```yaml
+lora_mlp_kernel: true
+lora_qkv_kernel: true
+lora_o_kernel: true
+```
+
+---
+
+## Quantization with torchao
+
+**URL:** https://docs.axolotl.ai/docs/quantize.html
+
+**Contents:**
+- Quantization with torchao
+- Configuring Quantization in Axolotl
+
+Quantization is a technique to lower the memory footprint of your model, potentially at the cost of accuracy or model performance. We support quantizing your model using the torchao library. Quantization is supported for both post-training quantization (PTQ) and quantization-aware training (QAT).
+
+We do not currently support quantization techniques such as GGUF/GPTQ,EXL2 at the moment.
+
+Quantization is configured using the quantization key in your configuration file.
+
+Once quantization is complete, your quantized model will be saved in the {output_dir}/quantized directory.
+
+You may also use the quantize command to quantize a model which has been trained with QAT - you can do this by using the existing QAT configuration file which you used to train the model:
+
+This ensures that an identical quantization configuration is used to quantize the model as was used to train it.
+
+If you have configured pushing to hub with hub_model_id, your model hub name will have the quantization schema appended to it, e.g. axolotl-ai-cloud/qat-nvfp4-llama3B will become axolotl-ai-cloud/qat-nvfp4-llama3B-nvfp4w
+
+**Examples:**
+
+Example 1 (yaml):
+```yaml
+base_model: # The path to the model to quantize.
+quantization:
+ activation_dtype: # Optional[str] = "int8". Fake quantization layout to use for activation quantization. Valid options are "int4", "int8", "float8"
+ weight_dtype: # Optional[str] = "int8". Fake quantization layout to use for weight quantization. Valid options are "int4", "fp8", and "nvfp4".
+ group_size: # Optional[int] = 32. The number of elements in each group for per-group fake quantization
+ quantize_embedding: # Optional[bool] = False. Whether to quantize the embedding layer.
+
+output_dir: # The path to the output directory.
+```
+
+Example 2 (yaml):
+```yaml
+# qat.yml
+qat:
+ activation_dtype: int8
+ weight_dtype: int4
+ group_size: 256
+
+output_dir: # The path to the output directory used during training where the final checkpoint has been saved.
+```
+
+Example 3 (bash):
+```bash
+axolotl quantize qat.yml
+```
+
+---
+
+## NCCL
+
+**URL:** https://docs.axolotl.ai/docs/nccl.html
+
+**Contents:**
+- NCCL
+
+NVIDIA NCCL is a library to facilitate and optimize multi-GPU communication operations, such as broadcast, all-gather, reduce, all-reduce, etc. Broadly, NCCL configuration is highly environment-specific and is configured via several environment variables. A common NCCL-related problem occurs when a long-running operation times out causing the training process to abort:
+
+Often, this timeout will happen after 30 minutes (the default setting) and is accompanied by below-average power consumption with near 100% GPU utilization before the error is raised. Nvidia recommends disabling PCI access control services (ACS) as a possible solution if this is available to you.
+
+Forcing cross-GPU communication via NVLink may help without increasing timeouts. To verify that your configuration is leveraging NVLink run the following command:
+
+To force NCCL to use NVLink, simply set this in the environment:
+
+If NVLink is not available in your environment there are other options for NCCL_P2P_LEVEL in the table below:
+
+To validate that acceptable data transfer speeds exist for your training job, running NCCL Tests can help pinpoint bottlenecks, for example:
+
+It can be useful when debugging NCCL communication timeouts to activate additional logging in both PyTorch and NCCL:
+
+Finally, if you believe your training job needs more time you can increase the timeout past 30 minutes by setting the ddp_timeout value in the Axolotl configuration. See PyTorch init_process_group for documentation on this value.
+
+**Examples:**
+
+Example 1 (unknown):
+```unknown
+Watchdog caught collective operation timeout: WorkNCCL(SeqNum=42, OpType=ALLGATHER, Timeout(ms)=1800000) ran for 1806948 milliseconds before timing out.
+```
+
+Example 2 (bash):
+```bash
+nvidia-smi nvlink --status
+```
+
+Example 3 (bash):
+```bash
+export NCCL_P2P_LEVEL=NVL
+```
+
+Example 4 (bash):
+```bash
+./build/all_reduce_perf -b 8 -e 128M -f 2 -g 3
+```
+
+---
+
+## Multi Node
+
+**URL:** https://docs.axolotl.ai/docs/multi-node.html
+
+**Contents:**
+- Multi Node
+- Accelerate
+- Raytrain
+- Torchrun
+ - Option 1: New Axolotl CLI with launcher args (Recommended)
+ - Option 2: Direct torchrun (Legacy)
+
+The below are three ways to train multi-node in Axolotl.
+
+Each machine needs a copy of Axolotl, we suggest using the same commit to ensure compatibility.
+
+You will also need to have the same configuration file for your model on each machine.
+
+Make sure the main machine is reachable by other machines.
+
+You will need to create a configuration for accelerate, either by using accelerate config and follow the instructions or you can use one of the preset below:
+
+~/.cache/huggingface/accelerate/default_config.yaml
+
+Configure your model to use FSDP in the Axolotl yaml. For example:
+
+All you have to do now is launch using accelerate as you would usually do on each machine and voila, the processes will start once you have launched accelerate on every machine.
+
+Please see ray train doc here.
+
+If you are using Infiniband, we recommend torchrun to utilize the full bandwidth.
+
+Set the following env (change buffersize/socketname depending on your system):
+
+Run the following on each node:
+
+Please make sure to substitute the placeholder variables:
+
+The new CLI approach (Option 1) is recommended as it provides consistent argument handling and works seamlessly with other Axolotl CLI features.
+
+More info on the available configs can be found on the Pytorch docs here
+
+**Examples:**
+
+Example 1 (yaml):
+```yaml
+compute_environment: LOCAL_MACHINE
+debug: false
+distributed_type: FSDP
+downcast_bf16: 'no'
+machine_rank: 0 # Set to 0 for the main machine, increment by one for other machines
+main_process_ip: 10.0.0.4 # Set to main machine's IP
+main_process_port: 5000
+main_training_function: main
+mixed_precision: bf16
+num_machines: 2 # Change to the number of machines
+num_processes: 4 # That's the total number of GPUs, (for example: if you have 2 machines with 4 GPU, put 8)
+rdzv_backend: static
+same_network: true
+tpu_env: []
+tpu_use_cluster: false
+tpu_use_sudo: false
+use_cpu: false
+```
+
+Example 2 (yaml):
+```yaml
+fsdp_version: 2
+fsdp_config:
+ offload_params: true
+ state_dict_type: FULL_STATE_DICT
+ auto_wrap_policy: TRANSFORMER_BASED_WRAP
+ transformer_layer_cls_to_wrap: LlamaDecoderLayer
+ reshard_after_forward: true
+```
+
+Example 3 (bash):
+```bash
+export NCCL_IB_DISABLE=0
+export NCCL_SOCKET_IFNAME="eth0,en,eth,em,bond"
+export NCCL_BUFFSIZE=2097152
+```
+
+Example 4 (bash):
+```bash
+axolotl train config.yaml --launcher torchrun -- --nnodes $num_nodes --nproc_per_node $gpu_per_node --rdzv_id $rdzv_id --rdzv_backend c10d --rdzv_endpoint "$head_node_ip:$head_node_port"
+```
+
+---
+
+## Dataset Loading
+
+**URL:** https://docs.axolotl.ai/docs/dataset_loading.html
+
+**Contents:**
+- Dataset Loading
+- Overview
+- Loading Datasets
+ - Local dataset
+ - Files
+ - Directory
+ - Loading entire directory
+ - Loading specific files in directory
+ - HuggingFace Hub
+ - Folder uploaded
+
+Datasets can be loaded in a number of different ways depending on the how it is saved (the extension of the file) and where it is stored.
+
+We use the datasets library to load datasets and a mix of load_dataset and load_from_disk to load them.
+
+You may recognize the similar named configs between load_dataset and the datasets section of the config file.
+
+Do not feel overwhelmed by the number of options here. A lot of them are optional. In fact, the most common config to use would be path and sometimes data_files.
+
+This matches the API of datasets.load_dataset, so if you’re familiar with that, you will feel right at home.
+
+For HuggingFace’s guide to load different dataset types, see here.
+
+For full details on the config, see config-reference.qmd.
+
+You can set multiple datasets in the config file by more than one entry under datasets.
+
+To load a JSON file, you would do something like this:
+
+Which translates to the following config:
+
+In the example above, it can be seen that we can just point the path to the file or directory along with the ds_type to load the dataset.
+
+This works for CSV, JSON, Parquet, and Arrow files.
+
+If path points to a file and ds_type is not specified, we will automatically infer the dataset type from the file extension, so you could omit ds_type if you’d like.
+
+If you’re loading a directory, you can point the path to the directory.
+
+Then, you have two options:
+
+You do not need any additional configs.
+
+We will attempt to load in the following order: - datasets saved with datasets.save_to_disk - loading entire directory of files (such as with parquet/arrow files)
+
+Provide data_files with a list of files to load.
+
+The method you use to load the dataset depends on how the dataset was created, whether a folder was uploaded directly or a HuggingFace Dataset was pushed.
+
+If you’re using a private dataset, you will need to enable the hf_use_auth_token flag in the root-level of the config file.
+
+This would mean that the dataset is a single file or file(s) uploaded to the Hub.
+
+This means that the dataset is created as a HuggingFace Dataset and pushed to the Hub via datasets.push_to_hub.
+
+There are some other configs which may be required like name, split, revision, trust_remote_code, etc depending on the dataset.
+
+Via the storage_options config under load_dataset, you can load datasets from remote filesystems like S3, GCS, Azure, and OCI.
+
+This is currently experimental. Please let us know if you run into any issues!
+
+The only difference between the providers is that you need to prepend the path with the respective protocols.
+
+For directory, we load via load_from_disk.
+
+Prepend the path with s3://.
+
+The credentials are pulled in the following order:
+
+We assume you have credentials setup and not using anonymous access. If you want to use anonymous access, let us know! We may have to open a config option for this.
+
+Other environment variables that can be set can be found in boto3 docs
+
+Prepend the path with gs:// or gcs://.
+
+The credentials are loaded in the following order:
+
+Prepend the path with adl://.
+
+Ensure you have the following environment variables set:
+
+Prepend the path with abfs:// or az://.
+
+Ensure you have the following environment variables set:
+
+Other environment variables that can be set can be found in adlfs docs
+
+Prepend the path with oci://.
+
+It would attempt to read in the following order:
+
+Other environment variables:
+
+Please see the ocifs docs.
+
+The path should start with https://.
+
+This must be publicly accessible.
+
+Now that you know how to load datasets, you can learn more on how to load your specific dataset format into your target output format dataset formats docs.
+
+**Examples:**
+
+Example 1 (yaml):
+```yaml
+datasets:
+ - path:
+ name:
+ data_files:
+ split:
+ revision:
+ trust_remote_code:
+```
+
+Example 2 (yaml):
+```yaml
+datasets:
+ - path: /path/to/your/dataset
+ - path: /path/to/your/other/dataset
+```
+
+Example 3 (python):
+```python
+from datasets import load_dataset
+
+dataset = load_dataset("json", data_files="data.json")
+```
+
+Example 4 (yaml):
+```yaml
+datasets:
+ - path: data.json
+ ds_type: json
+```
+
+---
+
+## Multi-GPU
+
+**URL:** https://docs.axolotl.ai/docs/multi-gpu.html
+
+**Contents:**
+- Multi-GPU
+- 1 Overview
+- 2 DeepSpeed
+ - 2.1 Configuration
+ - 2.2 Usage
+ - 2.3 ZeRO Stages
+- 3 Fully Sharded Data Parallel (FSDP)
+ - 3.1 Migrating from FSDP1 to FSDP2
+ - 3.1.1 Config mapping
+ - 3.2 FSDP1 (deprecated)
+
+This guide covers advanced training configurations for multi-GPU setups using Axolotl.
+
+Axolotl supports several methods for multi-GPU training:
+
+Add to your YAML config:
+
+We provide default configurations for:
+
+Choose the configuration that offloads the least amount to memory while still being able to fit on VRAM for best performance.
+
+Start from Stage 1 -> Stage 2 -> Stage 3.
+
+FSDP2 is recommended for new users. FSDP1 is deprecated and will be removed in an upcoming release of Axolotl.
+
+To migrate your config from FSDP1 to FSDP2, you must use the fsdp_version top-level config field to specify the FSDP version, and also follow the config field mapping below to update field names.
+
+For more details, please see the migration guide in the torchtitan repo. In Axolotl, if you were using the following FSDP1 config:
+
+You can migrate to the following FSDP2 config:
+
+Using fsdp to configure FSDP is deprecated and will be removed in an upcoming release of Axolotl. Please use fsdp_config as above instead.
+
+We support sequence parallelism (SP) via the ring-flash-attention project. This allows one to split up sequences across GPUs, which is useful in the event that a single sequence causes OOM errors during model training.
+
+See our dedicated guide for more information.
+
+For combining FSDP with QLoRA, see our dedicated guide.
+
+Please see docs for more info.
+
+For NCCL-related problems, see our NCCL troubleshooting guide.
+
+For more detailed troubleshooting, see our debugging guide.
+
+**Examples:**
+
+Example 1 (yaml):
+```yaml
+deepspeed: deepspeed_configs/zero1.json
+```
+
+Example 2 (bash):
+```bash
+# Fetch deepspeed configs (if not already present)
+axolotl fetch deepspeed_configs
+
+# Passing arg via config
+axolotl train config.yml
+
+# Passing arg via cli
+axolotl train config.yml --deepspeed deepspeed_configs/zero1.json
+```
+
+Example 3 (yaml):
+```yaml
+fsdp_version: 1
+fsdp_config:
+ fsdp_offload_params: false
+ fsdp_cpu_ram_efficient_loading: true
+ fsdp_auto_wrap_policy: TRANSFORMER_BASED_WRAP
+ fsdp_transformer_layer_cls_to_wrap: Qwen3DecoderLayer
+ fsdp_state_dict_type: FULL_STATE_DICT
+ fsdp_sharding_strategy: FULL_SHARD
+```
+
+Example 4 (yaml):
+```yaml
+fsdp_version: 2
+fsdp_config:
+ offload_params: false
+ cpu_ram_efficient_loading: true
+ auto_wrap_policy: TRANSFORMER_BASED_WRAP
+ transformer_layer_cls_to_wrap: Qwen3DecoderLayer
+ state_dict_type: FULL_STATE_DICT
+ reshard_after_forward: true
+```
+
+---
+
+## Ray Train
+
+**URL:** https://docs.axolotl.ai/docs/ray-integration.html
+
+**Contents:**
+- Ray Train
+- Ray cluster setup
+- Sanity check
+- Configuring training with Ray Train
+- Launching training
+
+Axolotl supports using Ray as an alternative to accelerate for orchestrating training. This is especially useful for multi-node training since you only have to setup code and dependencies in a single node and launch training as if you were using a single node.
+
+With the --use-ray CLI flag, Axolotl will use Ray Train’s TorchTrainer to run training.
+
+A prerequisite using the Ray Train integration is to setup a Ray cluster on your desired node(s). For a detailed guide on how you can get started with ray clusters, check the official Ray docs here.
+
+Every Ray cluster has one head node and a set of worker nodes. The head node is just like any other worker node, but it also runs certain special processes related to scheduling and orchestration. Ray-enabled scripts are run on the head node and depending on the resources (number of CPUs, GPUs, etc) they request, will be scheduled to run certain tasks on the worker nodes. For more on key concepts behind a Ray cluster, you can refer this doc.
+
+To run a sanity check on whether your ray cluster is setup properly, execute the following on the head node:
+
+The output should have a summary of your Ray cluster - list of all the nodes in your cluster, the number of CPUs and GPUs in your cluster, etc. For example, if you have a cluster with 1 CPU-only head node and 2 4xL40S worker nodes, the output can look like this:
+
+You should also be able to see the same on the Ray dashboard.
+
+You can find an example configuration at configs/llama-3/lora-1b-ray.yaml.
+
+The key parameters to note here are:
+
+You can simply run the following command on the head node:
+
+This will launch training on the head node and workers will be scheduled automatically by Ray Train to run on the appropriate head or worker nodes.
+
+You can also monitor training progress on the Ray dashboard.
+
+Coming back to the example on a Ray cluster with 1 head node and 2 4xL40S worker nodes, let’s say you want to make use of all 8 GPUs. You would be able to just set ray_num_workers: 8 and run the previous command. The Cluster tab will show the following:
+
+**Examples:**
+
+Example 1 (unknown):
+```unknown
+Node status
+---------------------------------------------------------------
+Active:
+ 1 head
+Idle:
+ 2 4xL40S:48CPU-384GB
+Pending:
+ (no pending nodes)
+Recent failures:
+ (no failures)
+
+Resources
+---------------------------------------------------------------
+Usage:
+ 0.0/96.0 CPU
+ 0.0/8.0 GPU
+ 0B/800.00GiB memory
+ 0B/229.57GiB object_store_memory
+
+Demands:
+ (no resource demands)
+```
+
+Example 2 (yaml):
+```yaml
+use_ray: true
+ray_num_workers: 4
+# optional
+resources_per_worker:
+ GPU: 1
+```
+
+Example 3 (yaml):
+```yaml
+resources_per_worker:
+ accelerator_type:L40S: 0.001
+```
+
+Example 4 (bash):
+```bash
+axolotl train examples/llama-3/lora-1b-ray.yml --use-ray
+```
+
+---
+
+## Sequence Parallelism
+
+**URL:** https://docs.axolotl.ai/docs/sequence_parallelism.html
+
+**Contents:**
+- Sequence Parallelism
+- When to Use Sequence Parallelism
+- Configuration
+- Implementation Details
+- Requirements
+- Limitations
+- Example
+- Sample Packing with Sequence Parallelism
+- Effect on Batch Size
+
+Sequence parallelism is a technique that splits sequences across multiple GPUs, allowing you to train with very long sequences that wouldn’t fit on a single GPU. Each GPU processes a different portion of the sequence, and the results are aggregated through a ring communication pattern.
+
+Use sequence parallelism when:
+
+To enable sequence parallelism, add the following to your configuration file:
+
+The context_parallel_size should be a divisor of the total number of GPUs. For example:
+
+When sequence parallelism is enabled:
+
+To use sequence parallelism, you need:
+
+This will train the Llama 3 8B model with 8K context length, with each sequence split into 2 subsequences of length 4096 across 2 GPUs.
+
+Sequence parallelism is compatible with Axolotl’s sample packing functionality. When using both features together:
+
+When using sequence parallelism, your effective global batch size is divided by the context_parallel_size. This happens because:
+
+For example: - With 8 GPUs and no sequence parallelism: 8 different batches processed per step - With 8 GPUs and context_parallel_size=4: Only 2 different batches processed per step (each split across 4 GPUs) - If your per-GPU micro_batch_size is 2, the global batch size decreases from 16 to 4
+
+**Examples:**
+
+Example 1 (yaml):
+```yaml
+# Set to a divisor (> 1) of the number of GPUs available
+context_parallel_size: 4 # Split sequences across 4 GPUs
+# Optional; strides across the key dimension. Larger values use more memory but should make training faster.
+heads_k_stride: 1
+# Optional; one of "varlen_llama3" or "batch_ring". Defaults to
+# "varlen_llama3" when `sample_packing: true`, and "batch_ring" otherwise.
+ring_attn_func:
+```
+
+Example 2 (yaml):
+```yaml
+base_model: meta-llama/Llama-3-8B-Instruct
+sequence_len: 8192
+
+...
+
+context_parallel_size: 4 # Split each sequence into 4 parts, one per GPU
+# Optional; strides across the key dimension. Larger values use more memory but should make training faster.
+heads_k_stride: 1
+# Optional; one of "varlen_llama3" or "batch_ring". Defaults to
+# "varlen_llama3" when `sample_packing: true`, and "batch_ring" otherwise.
+ring_attn_func:
+
+...
+```
+
+---
+
+## Quantization Aware Training (QAT)
+
+**URL:** https://docs.axolotl.ai/docs/qat.html
+
+**Contents:**
+- Quantization Aware Training (QAT)
+- Overview
+- Configuring QAT in Axolotl
+
+Quantization Aware Training (QAT) is a technique for improving the accuracy of models which are quantized by applying “fake” quantizations to the model’s weights (and optionally, activations) during training. This fake quantization allows for the model to adjust for noise introduced by the quantization, so when the model is eventually quantized, the accuracy loss is minimized. We use the quantization techniques implemented in torchao to provide support for QAT and post-training quantization (PTQ) in axolotl.
+
+We recommend reviewing the excellent QAT tutorial in the torchtune library, and the QAT documentation in the torchao library, for more details.
+
+To enable QAT in axolotl, add the following to your configuration file:
+
+We support the following quantization schemas:
+
+Once you have finished training, you must quantize your model by using the same quantization configuration which you used to train the model with. You can use the quantize command to do this.
+
+**Examples:**
+
+Example 1 (yaml):
+```yaml
+qat:
+ activation_dtype: # Optional[str] = "int8". Fake quantization layout to use for activation quantization. Valid options are "int4", "int8", "float8"
+ weight_dtype: # Optional[str] = "int8". Fake quantization layout to use for weight quantization. Valid options are "int4", "fp8", and "nvfp4".
+ group_size: # Optional[int] = 32. The number of elements in each group for per-group fake quantization
+ fake_quant_after_n_steps: # Optional[int] = None. The number of steps to apply fake quantization after
+```
+
+---
+
+## FSDP + QLoRA
+
+**URL:** https://docs.axolotl.ai/docs/fsdp_qlora.html
+
+**Contents:**
+- FSDP + QLoRA
+- Background
+- Usage
+- Enabling Swap for FSDP2
+- Example Config
+- References
+- Footnotes
+
+Using FSDP with QLoRA is essential for fine-tuning larger (70b+ parameter) LLMs on consumer GPUs. For example, you can use FSDP + QLoRA to train a 70b model on two 24GB GPUs1.
+
+Below, we describe how to use this feature in Axolotl.
+
+To enable QLoRA with FSDP, you need to perform the following steps:
+
+![Tip] See the example config file in addition to reading these instructions.
+
+If available memory is insufficient even after FSDP’s CPU offloading, you can enable swap memory usage by setting cpu_offload_pin_memory: false alongside offload_params: true in FSDP config.
+
+This disables memory pinning, allowing FSDP to use disk swap space as fallback. Disabling memory pinning itself incurs performance overhead, and actually having to use swap adds more, but it may enable training larger models that would otherwise cause OOM errors on resource constrained systems.
+
+examples/llama-2/qlora-fsdp.yml contains an example of how to enable QLoRA + FSDP in axolotl.
+
+This was enabled by this work from the Answer.AI team.↩︎
+
+---
+
+## Custom Integrations
+
+**URL:** https://docs.axolotl.ai/docs/custom_integrations.html
+
+**Contents:**
+- Custom Integrations
+- Cut Cross Entropy
+ - Requirements
+ - Installation
+ - Usage
+ - Supported Models
+ - Citation
+- DenseMixer
+- Diffusion LM Training Plugin for Axolotl
+ - Overview
+
+Axolotl adds custom features through integrations. They are located within the src/axolotl/integrations directory.
+
+To enable them, please check the respective documentations.
+
+Cut Cross Entropy (CCE) reduces VRAM usage through optimization on the cross-entropy operation during loss calculation.
+
+See https://github.com/apple/ml-cross-entropy
+
+Run the following command to install cut_cross_entropy[transformers] if you don’t have it already.
+
+Please see reference here
+
+Simply add the following to your axolotl YAML config:
+
+Please see reference here
+
+This plugin enables diffusion language model training using an approach inspired by LLaDA (Large Language Diffusion Models) within Axolotl.
+
+LLaDA is a diffusion-based approach to language model training that uses: - Random token masking during training instead of next-token prediction - Bidirectional attention to allow the model to attend to the full context - Importance weighting based on masking probabilities for stable training
+
+This approach can lead to more robust language models with better understanding of bidirectional context.
+
+The plugin is included with Axolotl. See our installation docs.
+
+Train with an example config (Llama‑3.2 1B): - Pretrain: axolotl train examples/llama-3/diffusion-3.2-1b-pretrain.yaml - SFT: axolotl train examples/llama-3/diffusion-3.2-1b-sft.yaml
+
+You can also modify your existing configs to enable / customize diffusion training.
+
+Add the following to your Axolotl config:
+
+And, configure the nested diffusion block (defaults shown):
+
+Any models that support 4D attention masks should work out of the box. If not, please create an issue or open a PR!
+
+During training, tokens are randomly masked: - Sample timestep t uniformly from [0, 1] - Calculate masking probability: p = (1 - eps) * t + eps - Randomly mask tokens with probability p
+
+Loss is computed only on masked tokens with (optional) importance weighting:
+
+When diffusion.generate_samples: true, the plugin generates samples during training:
+
+Samples are logged to console and wandb (if enabled).
+
+Diffusion inference is integrated into the standard Axolotl CLI. Use the same config you trained with and run:
+
+Optionally, pass --gradio to use a simple web interface.
+
+Interactive controls (prefix the prompt with commands): - :complete N → completion mode with N new masked tokens appended (default 64) - :mask R → random masking mode with target mask ratio R in [0.0, 1.0]
+
+The plugin adds (or modifies) several metrics to track diffusion training:
+
+Please see reference here
+
+See https://github.com/ironjr/grokfast
+
+Please see reference here
+
+An example dataset can be found at axolotl-ai-co/evolkit-logprobs-pipeline-75k-v2-sample
+
+Please see reference here
+
+Fine-tune sparsified models in Axolotl using Neural Magic’s LLMCompressor.
+
+This integration enables fine-tuning of models sparsified using LLMCompressor within the Axolotl training framework. By combining LLMCompressor’s model compression capabilities with Axolotl’s distributed training pipelines, users can efficiently fine-tune sparse models at scale.
+
+It uses Axolotl’s plugin system to hook into the fine-tuning flows while maintaining sparsity throughout training.
+
+Axolotl with llmcompressor extras:
+
+Requires llmcompressor >= 0.5.1
+
+This will install all necessary dependencies to fine-tune sparsified models using the integration.
+
+To enable sparse fine-tuning with this integration, include the plugin in your Axolotl config:
+
+This plugin does not apply pruning or sparsification itself — it is intended for fine-tuning models that have already been sparsified.
+
+Pre-sparsified checkpoints can be: - Generated using LLMCompressor - Downloaded from Neural Magic’s Hugging Face page - Any custom LLM with compatible sparsity patterns that you’ve created yourself
+
+To learn more about writing and customizing LLMCompressor recipes, refer to the official documentation: https://github.com/vllm-project/llm-compressor/blob/main/README.md
+
+Setting save_compressed: true in your configuration enables saving models in a compressed format, which: - Reduces disk space usage by approximately 40% - Maintains compatibility with vLLM for accelerated inference - Maintains compatibility with llmcompressor for further optimization (example: quantization)
+
+This option is highly recommended when working with sparse models to maximize the benefits of model compression.
+
+See examples/llama-3/sparse-finetuning.yaml for a complete example.
+
+After fine-tuning your sparse model, you can leverage vLLM for efficient inference. You can also use LLMCompressor to apply additional quantization to your fine-tuned sparse model before inference for even greater performance benefits.:
+
+For more details on vLLM’s capabilities and advanced configuration options, see the official vLLM documentation.
+
+For details on available sparsity and quantization schemes, fine-tuning recipes, and usage examples, visit the official LLMCompressor repository:
+
+https://github.com/vllm-project/llm-compressor
+
+Please see reference here
+
+Run evaluation on model using the popular lm-evaluation-harness library.
+
+See https://github.com/EleutherAI/lm-evaluation-harness
+
+Please see reference here
+
+Liger Kernel provides efficient Triton kernels for LLM training, offering:
+
+See https://github.com/linkedin/Liger-Kernel
+
+Please see reference here
+
+by Eric Hartford, Lucas Atkins, Fernando Fernandes, David Golchinfar
+
+This plugin contains code to freeze the bottom fraction of modules in a model, based on the Signal-to-Noise Ratio (SNR).
+
+See https://github.com/cognitivecomputations/spectrum
+
+Spectrum is a tool for scanning and evaluating the Signal-to-Noise Ratio (SNR) of layers in large language models. By identifying the top n% of layers with the highest SNR, you can optimize training efficiency.
+
+Please see reference here
+
+Plugins can be used to customize the behavior of the training pipeline through hooks. See axolotl.integrations.BasePlugin for the possible hooks.
+
+To add a new integration, please follow these steps:
+
+See src/axolotl/integrations/cut_cross_entropy for a minimal integration example.
+
+If you could not load your integration, please ensure you are pip installing in editable mode.
+
+and correctly spelled the integration name in the config file.
+
+It is not necessary to place your integration in the integrations folder. It can be in any location, so long as it’s installed in a package in your python env.
+
+See this repo for an example: https://github.com/axolotl-ai-cloud/diff-transformer
+
+**Examples:**
+
+Example 1 (bash):
+```bash
+python scripts/cutcrossentropy_install.py | sh
+```
+
+Example 2 (bash):
+```bash
+pip3 uninstall -y cut-cross-entropy && pip3 install "cut-cross-entropy[transformers] @ git+https://github.com/axolotl-ai-cloud/ml-cross-entropy.git@8a1a0ec"
+```
+
+Example 3 (yaml):
+```yaml
+plugins:
+ - axolotl.integrations.cut_cross_entropy.CutCrossEntropyPlugin
+```
+
+Example 4 (unknown):
+```unknown
+@article{wijmans2024cut,
+ author = {Erik Wijmans and
+ Brody Huval and
+ Alexander Hertzberg and
+ Vladlen Koltun and
+ Philipp Kr\"ahenb\"uhl},
+ title = {Cut Your Losses in Large-Vocabulary Language Models},
+ journal = {arXiv},
+ year = {2024},
+ url = {https://arxiv.org/abs/2411.09009},
+}
+```
+
+---
+
+## Config Reference
+
+**URL:** https://docs.axolotl.ai/docs/config-reference.html
+
+**Contents:**
+- Config Reference
+
+**Examples:**
+
+Example 1 (yaml):
+```yaml
+# Allow overwrite yml config using from cli
+strict: bool | None = False
+# Resume from a specific checkpoint dir
+resume_from_checkpoint: str | None
+# If resume_from_checkpoint isn't set and you simply want it to start where it left off.
+# Be careful with this being turned on between different models.
+auto_resume_from_checkpoints: bool | None
+# Resize the model embeddings when new tokens are added to multiples of 32. This is
+# reported to improve training speed on some models
+resize_token_embeddings_to_32x: bool | None
+mean_resizing_embeddings: bool | None = False
+
+# Whether to shrink the embeddings to len(tokenizer). By default, we won't shrink.
+shrink_embeddings: bool | None
+# Don't upcast the embeddings to float32 when using PEFT. Useful for low-VRAM GPUs
+embeddings_skip_upcast: bool | None
+# Reinitialize model weights randomly instead of loading pretrained weights
+reinit_weights: bool | None
+
+# module to custom trainer class to use for training
+trainer_cls: str | None
+
+# Use RL training: 'dpo', 'ipo', 'kto', 'simpo', 'orpo', 'grpo'
+rl: RLType | None
+
+trl: TRLConfig | None
+ # For TRLConfig:
+ # Beta parameter for the RL training. Same as `rl_beta`. Use
+ beta: float | None
+ # Maximum length of the completion for RL training.
+ max_completion_length: int | None
+
+ # Whether to use VLLM for RL training.
+ use_vllm: bool = False
+ # VLLM mode to use, one of 'server' or 'colocate'
+ vllm_mode: Literal['server', 'colocate'] | None
+ # Host of the vLLM server to connect to.
+ vllm_server_host: str | None = 0.0.0.0
+ # Port of the vLLM server to connect to.
+ vllm_server_port: int | None = 8000
+ # Total timeout (in seconds) to wait for the vLLM server to respond.
+ vllm_server_timeout: int | None
+ # Regex for vLLM guided decoding.
+ vllm_guided_decoding_regex: str | None
+
+ # List of reward functions to load. Paths must be importable from current dir.
+ reward_funcs: list[str] | None
+ # List of reward weights for the reward functions.
+ reward_weights: list[float] | None
+ # Number of generations to sample.
+ num_generations: int | None
+ # Whether to log completions.
+ log_completions: bool | None = False
+ # Number of completions to print when log_completions is True.
+ num_completions_to_print: int | None
+ # Controls whether importance sampling ratios are computed at the `'token'` or
+ # `'sequence'` level. For GSPO, use `sequence`, default is None which corresponds to
+ # the original GRPO paper.
+ importance_sampling_level: Literal['sequence', 'token'] | None
+
+ # Whether to sync the reference model.
+ sync_ref_model: bool | None = False
+ # Mixup alpha for the reference model.
+ ref_model_mixup_alpha: float | None = 0.9
+ # Sync steps for the reference model.
+ ref_model_sync_steps: int | None = 64
+ # Whether to scale rewards by their standard deviation.
+ scale_rewards: bool = True
+
+ # Sampling temperature for the GRPO policy.
+ temperature: float | None
+ # Top-p sampling probability for the generation policy.
+ top_p: float | None
+ # Top-k sampling for the generation policy.
+ top_k: int | None
+ # Minimum probability for the generation policy.
+ min_p: float | None
+ # Penalty for tokens that appear in prompt and generated text.
+ repetition_penalty: float | None
+ # Number of iterations per batch (μ) for GRPO.
+ num_iterations: int | None
+ # Epsilon value for clipping in the GRPO algorithm.
+ epsilon: float | None
+ # Upper-bound epsilon value for clipping in the GRPO algorithm.
+ epsilon_high: float | None
+ # Whether to use Liger loss for GRPO.
+ use_liger_loss: bool | None
+ # Loss formulation to use. Supported values: grpo, bnpo, dr_grpo.
+ loss_type: str | None
+ # Whether to exclude truncated completions from loss calculation.
+ mask_truncated_completions: bool = False
+ # Enable sleep mode for vLLM to offload VRAM when idle
+ vllm_enable_sleep_mode: bool | None
+
+vllm: VllmConfig | None
+ # For VllmConfig:
+ # Device to use for VLLM
+ device: str | None = auto
+ # Tensor parallel size for VLLM
+ tensor_parallel_size: int | None
+ # Data parallel size for VLLM
+ data_parallel_size: int | None
+ # GPU memory utilization for VLLM
+ gpu_memory_utilization: float | None = 0.9
+ # Data type for VLLM
+ dtype: str | None = auto
+ # Maximum length of the model context for VLLM
+ max_model_len: int | None
+ # Enable prefix caching for VLLM
+ enable_prefix_caching: bool | None
+ # Host for the vLLM server to start on
+ host: str | None = 0.0.0.0
+ # Port of the vLLM server to start on
+ port: int | None = 8000
+
+ # Enable reasoning for VLLM
+ enable_reasoning: bool | None
+ # Reasoning parser for VLLM
+ reasoning_parser: str | None
+
+qat: QATConfig | None
+ # For QATConfig:
+ # Fake quantization layout to use for activation quantization.
+ activation_dtype: TorchAOQuantDType | None
+ # Fake quantization layout to use for weight quantization.
+ weight_dtype: TorchAOQuantDType = TorchAOQuantDType.int8
+ # Quantize embedding
+ quantize_embedding: bool | None = False
+ # The number of elements in each group for per-group fake quantization
+ group_size: int | None = 32
+ # The number of steps to apply fake quantization after
+ fake_quant_after_n_steps: int | None
+
+quantization: PTQConfig | None
+ # For PTQConfig:
+ # Fake quantization layout to use for weight quantization.
+ weight_dtype: TorchAOQuantDType = TorchAOQuantDType.int8
+ # Fake quantization layout to use for activation quantization.
+ activation_dtype: TorchAOQuantDType | None
+ # Whether to quantize the embedding layer.
+ quantize_embedding: bool | None
+ # The number of elements in each group for per-group fake quantization
+ group_size: int | None = 32
+
+# Reward modelling: `True` or `False`
+reward_model: bool | None
+# Process reward modelling: `True` or `False`
+process_reward_model: bool | None
+# Coefficient to incentivize the reward model to output mean-zero rewards (proposed by
+# https://huggingface.co/papers/2312.09244, Eq. 2). Recommended value: `0.01`.
+center_rewards_coefficient: float | None
+num_labels: int | None
+
+# Whether to perform weighting in DPO trainer
+dpo_use_weighting: bool | None
+dpo_use_logits_to_keep: bool | None
+dpo_label_smoothing: float | None
+dpo_norm_loss: bool | None
+dpo_padding_free: bool | None
+dpo_generate_during_eval: bool | None
+
+# A list of one or more datasets to finetune the model with
+datasets: Annotated[list[SFTDataset | DPODataset | KTODataset | StepwiseSupervisedDataset], MinLen(1)] | None
+ # For SFTDataset:
+ # HuggingFace dataset repo | s3:// | gs:// | path to local file or directory
+ path: str | None
+ # name of dataset split to load from
+ split: str | None
+ # The type of prompt to use for training. [alpaca, gpteacher, oasst, reflection]
+ type: str | UserDefinedPrompterType | None
+ # For UserDefinedPrompterType:
+ # Custom user instruction prompt
+ system_prompt: str | None
+ # Use {system} as key to be replaced
+ system_format: str | None
+ field_system: str | None
+ field_instruction: str | None
+ field_input: str | None
+ field_output: str | None
+
+ # Customizable to be single line or multi-line. Use {instruction}/{input} as key to
+ # be replaced. 'format' can include {input}
+ format: str | None
+ # 'no_input_format' cannot include {input}
+ no_input_format: str | None
+ input_transform: str | None
+ # split dataset into N pieces (use with shards_idx)
+ shards: int | None
+ # the index of sharded dataset to use
+ shards_idx: int | None
+ # process dataset in N sequential chunks for memory efficiency (exclusive with
+ # `shards`)
+ preprocess_shards: int | None
+ conversation: str | None
+
+ # The name of the chat template to use for training, following values are supported:
+ # tokenizer_default: Uses the chat template that is available in the
+ # tokenizer_config.json. If the chat template is not available in the tokenizer, it
+ # will raise an error. This is the default.
+ # alpaca/inst/chatml/gemma/cohere/llama3/phi_3/deepseek_v2/jamba: These chat templates
+ # are available in the axolotl codebase at src/axolotl/utils/chat_templates.py.
+ # tokenizer_default_fallback_*: where * is the name of the chat template to fallback
+ # to if the tokenizer does not have a chat template else default to tokenizer. E.g.
+ # tokenizer_default_fallback_chatml. jinja: Uses a custom jinja template for the chat
+ # template. The custom jinja template should be provided in the chat_template_jinja
+ # field.
+ chat_template: ChatTemplate | str | None
+ # Custom jinja chat template or path to jinja file. Used only if `chat_template:
+ # jinja` or empty.
+ chat_template_jinja: str | None
+ # path to source data files
+ data_files: str | list[str] | None
+ input_format: str | None
+ # name of dataset configuration to load
+ name: str | None
+ # defines the datatype when path is a file
+ ds_type: str | None
+ # For `completion` datasets only, uses the provided field instead of `text` column
+ field: str | None
+ field_human: str | None
+ field_model: str | None
+ # Key containing the messages (default: "messages")
+ field_messages: str | None
+ # Key containing the tools (default: "tools"). Must be a list[dict] and follow [JSON
+ # schema](https://json-schema.org/learn/getting-started-step-by-step).
+ field_tools: str | None
+ # Key containing the reasoning trace (default: "reasoning_content").
+ field_thinking: str | None
+ # The key the chat template expects that indicates the reasoning trace.
+ template_thinking_key: str | None
+
+ message_field_role: str | None
+
+ message_field_content: str | None
+ # Mapping of properties from the input dataset to the chat template. (default:
+ # message_property_mappings={'role':'role', 'content':'content'}) If a property exists
+ # in the template but not in this mapping, the system will attempt to load it directly
+ # from the message using the property name as the key. Example: In the mapping below,
+ # 'from' is loaded from input dataset and used as 'role', while 'value' is loaded and
+ # used as 'content' in the chat template.
+ message_property_mappings: dict[str, str] | None
+ # The key in the message turn that indicates via boolean whether tokens of a turn
+ # should be considered for training. Useful to selectively train on certain turns
+ # besides the `roles_to_train`.
+ message_field_training: str | None
+ # The key in the message turn that contains the training details. Useful to
+ # selectively train on certain tokens in a turn. The value of the key is a List[Dict]
+ # containing `begin_offset` (start character index in content), `end_offset` (end
+ # character index in content), and `train` (boolean whether to train).
+ message_field_training_detail: str | None
+ # (for Qwen3 template only) Whether to split the assistant content based on a
+ # reasoning trace inside delimited tags
+ split_thinking: bool | None
+ logprobs_field: str | None
+ temperature: float | None
+ # Roles to train on. The tokens from these roles will be considered for the loss.
+ roles_to_train: list[str] | None
+ # Which EOS tokens to train on in the conversation. Possible values are: all: train on
+ # all EOS tokens, turn (default): train on the EOS token at the end of each trainable
+ # turn, last: train on the last EOS token in the conversation
+ train_on_eos: Literal['all', 'turn', 'last'] | None
+ # Roles mapping in the messages. The format is {target_role: [source_roles]}. All
+ # source roles will be mapped to the target role. The default is: user: ["human",
+ # "user"], assistant: ["gpt", "assistant"], system: ["system"], tool: ["tool"]
+ roles: dict[str, list[str]] | None
+ # Whether to drop the system turn from the dataset. Only works with chat_template.
+ # This does not drop the default system message from chat_template if it exists. If
+ # you wish to, we recommend using a custom jinja template with the default system
+ # message removed or adding a system turn with empty content.
+ drop_system_message: bool | None
+ # Trust remote code for untrusted source
+ trust_remote_code: bool | None = False
+ # The specific revision of the dataset to use when loading from the Hugging Face Hub.
+ # This can be a commit hash, tag, or branch name. If not specified, the latest version
+ # will be used. This parameter is ignored for local datasets.
+ revision: str | None
+
+ # For DPODataset:
+ path: str | None
+ split: str | None
+ type: UserDefinedDPOType | str | None
+ # For UserDefinedDPOType:
+ field_system: str | None
+ field_prompt: str | None
+ field_chosen: str | None
+ field_rejected: str | None
+ prompt_format: str | None
+ chosen_format: str | None
+ rejected_format: str | None
+ data_files: list[str] | None
+ revision: str | None
+ field_messages: str | None
+
+ # For KTODataset:
+ path: str | None
+ split: str | None
+ type: UserDefinedKTOType | str | None
+ # For UserDefinedKTOType:
+ field_system: str | None
+ field_prompt: str | None
+ field_completion: str | None
+ field_label: bool | None
+ prompt_format: str | None
+ completion_format: str | None
+ data_files: list[str] | None
+ trust_remote_code: bool | None = False
+ revision: str | None
+
+ # For StepwiseSupervisedDataset:
+ path: str | None
+ split: str | None
+ data_files: list[str] | None
+ revision: str | None
+ step_separator: str | None
+ max_completion_length: int | None
+ train_on_last_step_only: bool | None
+
+# A list of one or more datasets to eval the model with. You can use either
+# test_datasets, or val_set_size, but not both.
+test_datasets: Annotated[list[SFTDataset | DPODataset | KTODataset | StepwiseSupervisedDataset], MinLen(1)] | None
+ # For SFTDataset:
+ # HuggingFace dataset repo | s3:// | gs:// | path to local file or directory
+ path: str | None
+ # name of dataset split to load from
+ split: str | None
+ # The type of prompt to use for training. [alpaca, gpteacher, oasst, reflection]
+ type: str | UserDefinedPrompterType | None
+ # For UserDefinedPrompterType:
+ # Custom user instruction prompt
+ system_prompt: str | None
+ # Use {system} as key to be replaced
+ system_format: str | None
+ field_system: str | None
+ field_instruction: str | None
+ field_input: str | None
+ field_output: str | None
+
+ # Customizable to be single line or multi-line. Use {instruction}/{input} as key to
+ # be replaced. 'format' can include {input}
+ format: str | None
+ # 'no_input_format' cannot include {input}
+ no_input_format: str | None
+ input_transform: str | None
+ # split dataset into N pieces (use with shards_idx)
+ shards: int | None
+ # the index of sharded dataset to use
+ shards_idx: int | None
+ # process dataset in N sequential chunks for memory efficiency (exclusive with
+ # `shards`)
+ preprocess_shards: int | None
+ conversation: str | None
+
+ # The name of the chat template to use for training, following values are supported:
+ # tokenizer_default: Uses the chat template that is available in the
+ # tokenizer_config.json. If the chat template is not available in the tokenizer, it
+ # will raise an error. This is the default.
+ # alpaca/inst/chatml/gemma/cohere/llama3/phi_3/deepseek_v2/jamba: These chat templates
+ # are available in the axolotl codebase at src/axolotl/utils/chat_templates.py.
+ # tokenizer_default_fallback_*: where * is the name of the chat template to fallback
+ # to if the tokenizer does not have a chat template else default to tokenizer. E.g.
+ # tokenizer_default_fallback_chatml. jinja: Uses a custom jinja template for the chat
+ # template. The custom jinja template should be provided in the chat_template_jinja
+ # field.
+ chat_template: ChatTemplate | str | None
+ # Custom jinja chat template or path to jinja file. Used only if `chat_template:
+ # jinja` or empty.
+ chat_template_jinja: str | None
+ # path to source data files
+ data_files: str | list[str] | None
+ input_format: str | None
+ # name of dataset configuration to load
+ name: str | None
+ # defines the datatype when path is a file
+ ds_type: str | None
+ # For `completion` datasets only, uses the provided field instead of `text` column
+ field: str | None
+ field_human: str | None
+ field_model: str | None
+ # Key containing the messages (default: "messages")
+ field_messages: str | None
+ # Key containing the tools (default: "tools"). Must be a list[dict] and follow [JSON
+ # schema](https://json-schema.org/learn/getting-started-step-by-step).
+ field_tools: str | None
+ # Key containing the reasoning trace (default: "reasoning_content").
+ field_thinking: str | None
+ # The key the chat template expects that indicates the reasoning trace.
+ template_thinking_key: str | None
+
+ message_field_role: str | None
+
+ message_field_content: str | None
+ # Mapping of properties from the input dataset to the chat template. (default:
+ # message_property_mappings={'role':'role', 'content':'content'}) If a property exists
+ # in the template but not in this mapping, the system will attempt to load it directly
+ # from the message using the property name as the key. Example: In the mapping below,
+ # 'from' is loaded from input dataset and used as 'role', while 'value' is loaded and
+ # used as 'content' in the chat template.
+ message_property_mappings: dict[str, str] | None
+ # The key in the message turn that indicates via boolean whether tokens of a turn
+ # should be considered for training. Useful to selectively train on certain turns
+ # besides the `roles_to_train`.
+ message_field_training: str | None
+ # The key in the message turn that contains the training details. Useful to
+ # selectively train on certain tokens in a turn. The value of the key is a List[Dict]
+ # containing `begin_offset` (start character index in content), `end_offset` (end
+ # character index in content), and `train` (boolean whether to train).
+ message_field_training_detail: str | None
+ # (for Qwen3 template only) Whether to split the assistant content based on a
+ # reasoning trace inside delimited tags
+ split_thinking: bool | None
+ logprobs_field: str | None
+ temperature: float | None
+ # Roles to train on. The tokens from these roles will be considered for the loss.
+ roles_to_train: list[str] | None
+ # Which EOS tokens to train on in the conversation. Possible values are: all: train on
+ # all EOS tokens, turn (default): train on the EOS token at the end of each trainable
+ # turn, last: train on the last EOS token in the conversation
+ train_on_eos: Literal['all', 'turn', 'last'] | None
+ # Roles mapping in the messages. The format is {target_role: [source_roles]}. All
+ # source roles will be mapped to the target role. The default is: user: ["human",
+ # "user"], assistant: ["gpt", "assistant"], system: ["system"], tool: ["tool"]
+ roles: dict[str, list[str]] | None
+ # Whether to drop the system turn from the dataset. Only works with chat_template.
+ # This does not drop the default system message from chat_template if it exists. If
+ # you wish to, we recommend using a custom jinja template with the default system
+ # message removed or adding a system turn with empty content.
+ drop_system_message: bool | None
+ # Trust remote code for untrusted source
+ trust_remote_code: bool | None = False
+ # The specific revision of the dataset to use when loading from the Hugging Face Hub.
+ # This can be a commit hash, tag, or branch name. If not specified, the latest version
+ # will be used. This parameter is ignored for local datasets.
+ revision: str | None
+
+ # For DPODataset:
+ path: str | None
+ split: str | None
+ type: UserDefinedDPOType | str | None
+ # For UserDefinedDPOType:
+ field_system: str | None
+ field_prompt: str | None
+ field_chosen: str | None
+ field_rejected: str | None
+ prompt_format: str | None
+ chosen_format: str | None
+ rejected_format: str | None
+ data_files: list[str] | None
+ revision: str | None
+ field_messages: str | None
+
+ # For KTODataset:
+ path: str | None
+ split: str | None
+ type: UserDefinedKTOType | str | None
+ # For UserDefinedKTOType:
+ field_system: str | None
+ field_prompt: str | None
+ field_completion: str | None
+ field_label: bool | None
+ prompt_format: str | None
+ completion_format: str | None
+ data_files: list[str] | None
+ trust_remote_code: bool | None = False
+ revision: str | None
+
+ # For StepwiseSupervisedDataset:
+ path: str | None
+ split: str | None
+ data_files: list[str] | None
+ revision: str | None
+ step_separator: str | None
+ max_completion_length: int | None
+ train_on_last_step_only: bool | None
+
+# If false, the datasets will not be shuffled and will keep their original order in
+# `datasets`. The same applies to the `test_datasets` option and the
+# `pretraining_dataset` option. Default is true.
+shuffle_merged_datasets: bool | None = True
+# If true, each dataset in `datasets` will be shuffled before merging. This allows
+# curriculum learning strategies to be applied at the dataset level. Default is false.
+shuffle_before_merging_datasets: bool | None = False
+# Axolotl attempts to save the dataset as an arrow after packing the data together so
+# subsequent training attempts load faster, relative path
+dataset_prepared_path: str | None
+# Num shards for whole dataset
+dataset_shard_num: int | None
+# Index of shard to use for whole dataset
+dataset_shard_idx: int | None
+skip_prepare_dataset: bool | None = False
+# Number of shards to save the prepared dataset
+num_dataset_shards_to_save: int | None
+
+# Set to HF dataset for type: 'completion' for streaming instead of pre-tokenize
+pretraining_dataset: Annotated[list[PretrainingDataset | SFTDataset], MinLen(1)] | None
+ # For PretrainingDataset:
+ name: str | None
+ path: str | None
+ split: str | None = train
+ text_column: str | None = text
+ type: str | None = pretrain
+ trust_remote_code: bool | None = False
+ data_files: str | None
+ skip: int | None
+
+ # For SFTDataset:
+ # HuggingFace dataset repo | s3:// | gs:// | path to local file or directory
+ path: str | None
+ # name of dataset split to load from
+ split: str | None
+ # The type of prompt to use for training. [alpaca, gpteacher, oasst, reflection]
+ type: str | UserDefinedPrompterType | None
+ # For UserDefinedPrompterType:
+ # Custom user instruction prompt
+ system_prompt: str | None
+ # Use {system} as key to be replaced
+ system_format: str | None
+ field_system: str | None
+ field_instruction: str | None
+ field_input: str | None
+ field_output: str | None
+
+ # Customizable to be single line or multi-line. Use {instruction}/{input} as key to
+ # be replaced. 'format' can include {input}
+ format: str | None
+ # 'no_input_format' cannot include {input}
+ no_input_format: str | None
+ input_transform: str | None
+ # split dataset into N pieces (use with shards_idx)
+ shards: int | None
+ # the index of sharded dataset to use
+ shards_idx: int | None
+ # process dataset in N sequential chunks for memory efficiency (exclusive with
+ # `shards`)
+ preprocess_shards: int | None
+ conversation: str | None
+
+ # The name of the chat template to use for training, following values are supported:
+ # tokenizer_default: Uses the chat template that is available in the
+ # tokenizer_config.json. If the chat template is not available in the tokenizer, it
+ # will raise an error. This is the default.
+ # alpaca/inst/chatml/gemma/cohere/llama3/phi_3/deepseek_v2/jamba: These chat templates
+ # are available in the axolotl codebase at src/axolotl/utils/chat_templates.py.
+ # tokenizer_default_fallback_*: where * is the name of the chat template to fallback
+ # to if the tokenizer does not have a chat template else default to tokenizer. E.g.
+ # tokenizer_default_fallback_chatml. jinja: Uses a custom jinja template for the chat
+ # template. The custom jinja template should be provided in the chat_template_jinja
+ # field.
+ chat_template: ChatTemplate | str | None
+ # Custom jinja chat template or path to jinja file. Used only if `chat_template:
+ # jinja` or empty.
+ chat_template_jinja: str | None
+ # path to source data files
+ data_files: str | list[str] | None
+ input_format: str | None
+ # name of dataset configuration to load
+ name: str | None
+ # defines the datatype when path is a file
+ ds_type: str | None
+ # For `completion` datasets only, uses the provided field instead of `text` column
+ field: str | None
+ field_human: str | None
+ field_model: str | None
+ # Key containing the messages (default: "messages")
+ field_messages: str | None
+ # Key containing the tools (default: "tools"). Must be a list[dict] and follow [JSON
+ # schema](https://json-schema.org/learn/getting-started-step-by-step).
+ field_tools: str | None
+ # Key containing the reasoning trace (default: "reasoning_content").
+ field_thinking: str | None
+ # The key the chat template expects that indicates the reasoning trace.
+ template_thinking_key: str | None
+
+ message_field_role: str | None
+
+ message_field_content: str | None
+ # Mapping of properties from the input dataset to the chat template. (default:
+ # message_property_mappings={'role':'role', 'content':'content'}) If a property exists
+ # in the template but not in this mapping, the system will attempt to load it directly
+ # from the message using the property name as the key. Example: In the mapping below,
+ # 'from' is loaded from input dataset and used as 'role', while 'value' is loaded and
+ # used as 'content' in the chat template.
+ message_property_mappings: dict[str, str] | None
+ # The key in the message turn that indicates via boolean whether tokens of a turn
+ # should be considered for training. Useful to selectively train on certain turns
+ # besides the `roles_to_train`.
+ message_field_training: str | None
+ # The key in the message turn that contains the training details. Useful to
+ # selectively train on certain tokens in a turn. The value of the key is a List[Dict]
+ # containing `begin_offset` (start character index in content), `end_offset` (end
+ # character index in content), and `train` (boolean whether to train).
+ message_field_training_detail: str | None
+ # (for Qwen3 template only) Whether to split the assistant content based on a
+ # reasoning trace inside delimited tags
+ split_thinking: bool | None
+ logprobs_field: str | None
+ temperature: float | None
+ # Roles to train on. The tokens from these roles will be considered for the loss.
+ roles_to_train: list[str] | None
+ # Which EOS tokens to train on in the conversation. Possible values are: all: train on
+ # all EOS tokens, turn (default): train on the EOS token at the end of each trainable
+ # turn, last: train on the last EOS token in the conversation
+ train_on_eos: Literal['all', 'turn', 'last'] | None
+ # Roles mapping in the messages. The format is {target_role: [source_roles]}. All
+ # source roles will be mapped to the target role. The default is: user: ["human",
+ # "user"], assistant: ["gpt", "assistant"], system: ["system"], tool: ["tool"]
+ roles: dict[str, list[str]] | None
+ # Whether to drop the system turn from the dataset. Only works with chat_template.
+ # This does not drop the default system message from chat_template if it exists. If
+ # you wish to, we recommend using a custom jinja template with the default system
+ # message removed or adding a system turn with empty content.
+ drop_system_message: bool | None
+ # Trust remote code for untrusted source
+ trust_remote_code: bool | None = False
+ # The specific revision of the dataset to use when loading from the Hugging Face Hub.
+ # This can be a commit hash, tag, or branch name. If not specified, the latest version
+ # will be used. This parameter is ignored for local datasets.
+ revision: str | None
+
+# The maximum number of processes to use while preprocessing your input dataset. This
+# defaults to `os.cpu_count()` if not set. For Runpod VMs, it will default to number of
+# vCPUs via RUNPOD_CPU_COUNT.
+dataset_processes: int | None
+# The maximum number of processes to use while preprocessing your input dataset. This
+# defaults to `os.cpu_count()` if not set. For Runpod VMs, it will default to number of
+# vCPUs via RUNPOD_CPU_COUNT.
+dataset_num_proc: int | None
+
+# Deduplicates datasets and test_datasets with identical entries
+dataset_exact_deduplication: bool | None
+# Keep dataset in memory while preprocessing. Only needed if cached dataset is taking
+# too much storage
+dataset_keep_in_memory: bool | None
+dataloader_pin_memory: bool | None
+dataloader_num_workers: int | None
+dataloader_prefetch_factor: int | None
+dataloader_drop_last: bool | None
+
+accelerator_config: dict[str, Any] | None
+
+remove_unused_columns: bool | None
+
+# Push prepared dataset to hub - repo_org/repo_name
+push_dataset_to_hub: str | None
+# Whether to use hf `use_auth_token` for loading datasets. Useful for fetching private
+# datasets. Required to be true when used in combination with `push_dataset_to_hub`
+hf_use_auth_token: bool | None
+
+device: Any | None
+# Passed through to transformers when loading the model when launched without
+# accelerate. Use `sequential` when training w/ model parallelism to limit memory
+device_map: Any | None
+world_size: int | None
+# Don't mess with this, it's here for accelerate and torchrun
+local_rank: int | None
+ddp: bool | None
+
+# Seed for reproducibility
+seed: int | None
+# Advanced DDP Arguments - timeout
+ddp_timeout: int | None
+# Advanced DDP Arguments - bucket cap in MB
+ddp_bucket_cap_mb: int | None
+# Advanced DDP Arguments - broadcast buffers
+ddp_broadcast_buffers: bool | None
+ddp_find_unused_parameters: bool | None
+
+# Approximate number of predictions sent to wandb depending on batch size. Enabled above
+# 0. Default is 0
+eval_table_size: int | None
+# Total number of tokens generated for predictions sent to wandb. Default is 128
+eval_max_new_tokens: int | None
+# Whether to run causal language model evaluation for metrics in
+# `eval_causal_lm_metrics`
+do_causal_lm_eval: bool | None
+# HF evaluate metrics used during evaluation. Default is ['sacrebleu', 'comet', 'ter',
+# 'chrf', 'perplexity']
+eval_causal_lm_metrics: list[str] | None
+do_bench_eval: bool | None
+bench_dataset: str | None
+bench_split: str | None
+metric_for_best_model: str | None
+greater_is_better: bool | None
+
+# High loss value, indicating the learning has broken down (a good estimate is ~2 times
+# the loss at the start of training)
+loss_watchdog_threshold: float | None
+# Number of high-loss steps in a row before the trainer aborts (default: 3)
+loss_watchdog_patience: int | None
+
+# Run garbage collection every `gc_steps` steps. -1 will run on epoch end and before
+# evaluations. Default is 0 (disabled).
+gc_steps: int | None
+
+# Use CUDA bf16. bool or 'full' for `bf16_full_eval`, or 'auto' for automatic detection.
+# require >=ampere
+bf16: Literal['auto'] | bool | None = auto
+# Use CUDA fp16
+fp16: bool | None
+# Enable FP8 mixed precision training using TorchAO. Best used in combination with
+# torch.compile.
+fp8: bool | None
+# Enable FSDP float8 all-gather optimization for FP8 training. Can improve training
+# speed by 10-15% when FSDP is enabled.
+fp8_enable_fsdp_float8_all_gather: bool | None
+# No AMP (automatic mixed precision) - require >=ampere
+bfloat16: bool | None
+# No AMP (automatic mixed precision)
+float16: bool | None
+# Use CUDA tf32 - require >=ampere
+tf32: bool | None
+float32: bool | None
+
+# Whether to use gradient checkpointing. Available options are: true, false, 'offload',
+# 'offload_disk'.
+# https://huggingface.co/docs/transformers/v4.18.0/en/performance#gradient-checkpointing
+gradient_checkpointing: Literal['offload', 'offload_disk'] | bool | None = False
+# Additional kwargs to pass to the trainer for gradient checkpointing
+gradient_checkpointing_kwargs: dict[str, Any] | None
+# Whether to offload activations. Available options are: true, false, 'legacy', 'disk'.
+activation_offloading: Literal['legacy', 'disk'] | bool | None = False
+
+unfrozen_parameters: list[str] | None
+
+# The maximum length of an input to train with, this should typically be less than 2048
+# as most models have a token/context limit of 2048
+sequence_len: int = 512
+# What to do when a tokenized row exceeds sequence_len. 'drop' removes the row;
+# 'truncate' slices tensors to sequence_len. Defaults to 'drop' for backward
+# compatibility.
+excess_length_strategy: Literal['drop', 'truncate'] | None
+# The maximum length of an input for evaluation. If not specified, defaults to
+# sequence_len
+eval_sequence_len: int | None
+min_sample_len: int | None
+# maximum prompt length for RL training
+max_prompt_len: int | None
+# Use efficient multi-packing with block diagonal attention and per sequence
+# position_ids. Recommend set to 'true'
+sample_packing: bool | None
+# The number of samples packed at a time. Increasing the following values helps with
+# packing, but usually only slightly (<%1.)
+sample_packing_group_size: int | None = 100000
+# The number of samples which can be packed into one sequence. Increase if using a large
+# sequence_len with many short samples.
+sample_packing_bin_size: int | None = 200
+# Whether to pack samples sequentially
+sample_packing_sequentially: bool | None
+# The multiprocessing start method to use for packing. Should be 'fork', 'spawn' or
+# 'forkserver'
+sample_packing_mp_start_method: str | None
+# Set to 'false' if getting errors during eval with sample_packing on
+eval_sample_packing: bool | None
+# Pad inputs so each step uses constant sized buffers. This will reduce memory
+# fragmentation and may prevent OOMs, by re-using memory more efficiently. Defaults to
+# True if `sample_packing` enabled
+pad_to_sequence_len: bool | None
+# Whether to use sequential sampling for curriculum learning
+curriculum_sampling: bool | None
+multipack_real_batches: bool | None
+
+# Use batch flattening for speedups when not using sample_packing
+batch_flattening: Literal['auto'] | bool | None
+
+use_pose: bool | None
+pose_split_on_token_ids: list[int] | None
+pose_max_context_len: int | None
+pose_num_chunks: int | None
+
+pretrain_multipack_buffer_size: int | None
+# whether to prevent cross attention for packed sequences during pretraining
+pretrain_multipack_attn: bool | None = True
+# whether to concatenate samples during pretraining
+pretraining_sample_concatenation: bool | None
+
+# Use streaming mode for loading datasets
+streaming: bool | None
+# Buffer size for multipack streaming datasets
+streaming_multipack_buffer_size: int | None = 10000
+
+# Whether to use xformers attention patch https://github.com/facebookresearch/xformers
+xformers_attention: bool | None
+# Whether to use scaled-dot-product attention https://pytorch.org/docs/stable/generated/
+# torch.nn.functional.scaled_dot_product_attention.html
+sdp_attention: bool | None
+# Shifted-sparse attention (only llama) - https://arxiv.org/pdf/2309.12307.pdf
+s2_attention: bool | None
+flex_attention: bool | None
+flex_attn_compile_kwargs: dict[str, Any] | None
+# Whether to use flash attention patch https://github.com/Dao-AILab/flash-attention
+flash_attention: bool | None
+# Whether to use flash-attention cross entropy implementation - advanced use only
+flash_attn_cross_entropy: bool | None
+# Whether to use flash-attention rms norm implementation - advanced use only
+flash_attn_rms_norm: bool | None
+# Whether to fuse part of the MLP into a single operation
+flash_attn_fuse_mlp: bool | None
+# Whether to use bettertransformers
+flash_optimum: bool | None
+
+eager_attention: bool | None
+
+# Specify a custom attention implementation, used mostly for kernels.
+attn_implementation: str | None
+
+unsloth_cross_entropy_loss: bool | None
+unsloth_lora_mlp: bool | None
+unsloth_lora_qkv: bool | None
+unsloth_lora_o: bool | None
+unsloth_rms_norm: bool | None
+unsloth_rope: bool | None
+
+# Apply custom LoRA autograd functions and activation function Triton kernels for speed
+# and memory savings. See: https://docs.axolotl.ai/docs/lora_optims.html
+lora_mlp_kernel: bool | None
+# Apply custom LoRA autograd functions and activation function Triton kernels for speed
+# and memory savings. See: https://docs.axolotl.ai/docs/lora_optims.html
+lora_qkv_kernel: bool | None
+# Apply custom LoRA autograd functions and activation function Triton kernels for speed
+# and memory savings. See: https://docs.axolotl.ai/docs/lora_optims.html
+lora_o_kernel: bool | None
+
+# Whether to use chunked cross entropy loss for memory efficiency
+chunked_cross_entropy: bool | None
+# Number of chunks to use for chunked cross entropy loss
+chunked_cross_entropy_num_chunks: int | None
+
+# Whether to use ALST tiled mlp for memory efficient long context
+tiled_mlp: bool | None
+
+# Number of shards to use for ALST tiled mlp. If unset, it will be set based on
+# seqlen/hidden_size
+tiled_mlp_num_shards: int | None
+
+# Whether to use original mlp for ALST tiled mlp. Otherwise uses a generic MLP based on
+# llama.
+tiled_mlp_use_original_mlp: bool | None = True
+
+llama4_linearized_experts: bool | None
+
+# Deepspeed config path. e.g., deepspeed_configs/zero3.json
+deepspeed: str | dict[str, Any] | None
+# Whether to use deepcompile for faster training with deepspeed
+deepcompile: bool | None
+# FSDP configuration
+fsdp: list[str] | None
+
+# FSDP configuration options
+fsdp_config: FSDPConfig | None
+ # For FSDPConfig:
+ # Enable activation checkpointing to reduce memory usage during forward passes
+ activation_checkpointing: bool | None
+ # Offload parameters to CPU to reduce GPU memory usage
+ offload_params: bool | None
+ # Synchronize module states across all processes
+ sync_module_states: bool | None
+ # Enable CPU RAM efficient loading to reduce memory usage during model loading
+ cpu_ram_efficient_loading: bool | None
+ # Disabling this enables swap memory usage for resource-constrained setups when
+ # offload_params is enabled.
+ cpu_offload_pin_memory: bool | None
+ # Use original parameters instead of flattened parameters
+ use_orig_params: bool | None
+
+ # Type of state dict to use for saving/loading checkpoints
+ state_dict_type: Literal['FULL_STATE_DICT', 'LOCAL_STATE_DICT', 'SHARDED_STATE_DICT'] | None
+ # Final state dict type to use after training completion
+ final_state_dict_type: Literal['FULL_STATE_DICT', 'LOCAL_STATE_DICT', 'SHARDED_STATE_DICT'] | None
+
+ # Policy for automatically wrapping modules with FSDP
+ auto_wrap_policy: Literal['TRANSFORMER_BASED_WRAP', 'SIZE_BASED_WRAP'] | None
+ # Class name of transformer layers to wrap (e.g., 'LlamaDecoderLayer')
+ transformer_layer_cls_to_wrap: str | None
+
+ # Reshard parameters after forward pass to save memory
+ reshard_after_forward: bool | None
+ # Mixed precision policy for FSDP (e.g., 'fp16', 'bf16')
+ mixed_precision_policy: str | None
+
+# FSDP version
+fsdp_version: int | None
+fsdp_final_state_dict_type: Literal['FULL_STATE_DICT', 'LOCAL_STATE_DICT', 'SHARDED_STATE_DICT'] | None
+
+# How much of the dataset to set aside as evaluation. 1 = 100%, 0.50 = 50%, etc. 0 for
+# no eval.
+val_set_size: float | None = 0.0
+
+# Number of devices to shard across. If not set, will use all available devices.
+dp_shard_size: int | None
+# Number of devices to replicate across.
+dp_replicate_size: int | None
+# Deprecated: use `context_parallel_size` instead
+sequence_parallel_degree: int | None
+# Set to a divisor of the number of GPUs available to split sequences into chunks of
+# equal size. Use in long context training to prevent OOM when sequences cannot fit into
+# a single GPU's VRAM. E.g., if 4 GPUs are available, set this value to 2 to split each
+# sequence into two equal-sized subsequences, or set to 4 to split into four equal-sized
+# subsequences. See https://docs.axolotl.ai/docs/sequence_parallelism.html for more
+# details.
+context_parallel_size: int | None
+# Optional; strides across the key dimension. Larger values use more memory but should
+# make training faster. Must evenly divide the number of KV heads in your model.
+heads_k_stride: int | None
+# One of 'varlen_llama3', 'batch_ring', 'batch_zigzag', 'batch_stripe'. Defaults to
+# 'varlen_llama3' in the sample packing case, and 'batch_ring' in the non-sample packing
+# case.
+ring_attn_func: RingAttnFunc | None
+# Number of tensor parallel processes in TP group. Only supported with DeepSpeed AutoTP.
+tensor_parallel_size: int | None
+
+# Add or change special tokens. If you add tokens here, you don't need to add them to
+# the `tokens` list.
+special_tokens: SpecialTokensConfig | None
+ # For SpecialTokensConfig:
+ bos_token: str | None
+ eos_token: str | None
+ pad_token: str | None
+ unk_token: str | None
+ additional_special_tokens: list[str] | None
+
+# Add extra tokens to the tokenizer
+tokens: list[str] | None
+# Mapping token_id to new_token_string to override reserved added_tokens in the
+# tokenizer. Only works for tokens that are not part of the base vocab (aka are
+# added_tokens). Can be checked if they exist in tokenizer.json added_tokens.
+added_tokens_overrides: dict[int, str] | None
+
+# Whether to use torch.compile and which backend to use. setting to `auto` will enable
+# torch compile when torch>=2.6.0
+torch_compile: Literal['auto'] | bool | None
+# Backend to use for torch.compile
+torch_compile_backend: str | None
+torch_compile_mode: Literal['default', 'reduce-overhead', 'max-autotune'] | None
+
+# Maximum number of iterations to train for. It precedes num_epochs which means that if
+# both are set, num_epochs will not be guaranteed. e.g., when 1 epoch is 1000 steps =>
+# `num_epochs: 2` and `max_steps: 100` will train for 100 steps
+max_steps: int | None
+# Number of warmup steps. Cannot use with warmup_ratio
+warmup_steps: int | None
+# Warmup ratio. Cannot use with warmup_steps
+warmup_ratio: float | None
+# Leave empty to eval at each epoch, integer for every N steps. float for fraction of
+# total steps
+eval_steps: int | float | None
+# Number of times per epoch to run evals, mutually exclusive with eval_steps
+evals_per_epoch: int | None
+# Set to `no` to skip evaluation, `epoch` at end of each epoch, leave empty to infer
+# from `eval_steps`
+eval_strategy: str | None
+
+# Leave empty to save at each epoch, integer for every N steps. float for fraction of
+# total steps
+save_steps: int | float | None
+# Number of times per epoch to save a checkpoint, mutually exclusive with save_steps
+saves_per_epoch: int | None
+# Set to `no` to skip checkpoint saves, `epoch` at end of each epoch, `best` when better
+# result is achieved, leave empty to infer from `save_steps`
+save_strategy: str | None
+# Checkpoints saved at a time
+save_total_limit: int | None
+# Whether to checkpoint a model after the first step of training. Defaults to False.
+save_first_step: bool | None
+
+# Logging frequency
+logging_steps: int | None
+# Stop training after this many evaluation losses have increased in a row. https://huggi
+# ngface.co/transformers/v4.2.2/_modules/transformers/trainer_callback.html#EarlyStoppin
+# gCallback
+early_stopping_patience: int | None
+load_best_model_at_end: bool | None = False
+# Save only the model weights, skipping the optimizer. Using this means you can't resume
+# from checkpoints.
+save_only_model: bool | None = False
+# Use tensorboard for logging
+use_tensorboard: bool | None
+# Enable the pytorch profiler to capture the first N steps of training to the
+# output_dir. see https://pytorch.org/blog/understanding-gpu-memory-1/ for more
+# information. Snapshots can be visualized @ https://pytorch.org/memory_viz
+profiler_steps: int | None
+# Which step to start the profiler at. Useful for only capturing a few steps mid-run.
+profiler_steps_start: int | None = 0
+# bool of whether to report tokens per second at the end of training. This is not
+# supported with pre-training datasets.
+include_tokens_per_second: bool | None
+# bool of whether to report tokens per second per-gpu during training by measuring
+# throughput of non-padding tokens.
+include_tkps: bool | None = True
+# NEFT https://arxiv.org/abs/2310.05914, set this to a number (paper default is 5) to
+# add noise to embeddings. Currently only supported on Llama and Mistral
+neftune_noise_alpha: float | None
+
+# Parameter controlling the relative ratio loss weight in the ORPO loss. Passed to
+# `beta` in `ORPOConfig` due to trl mapping.
+orpo_alpha: float | None
+# Weighting of NLL term in loss from RPO paper
+rpo_alpha: float | None
+# Target reward margin for the SimPO loss
+simpo_gamma: float | None
+# Weight of the BC regularizer
+cpo_alpha: float | None
+
+# Factor for desirable loss term in KTO loss
+kto_desirable_weight: float | None
+# Factor for undesirable loss term in KTO loss
+kto_undesirable_weight: float | None
+# The beta parameter for the RL training
+rl_beta: float | None
+
+# Defines the max memory usage per gpu on the system. Passed through to transformers
+# when loading the model.
+max_memory: dict[int | Literal['cpu', 'disk'], int | str] | None
+# Limit the memory for all available GPUs to this amount (if an integer, expressed in
+# gigabytes); default: unset
+gpu_memory_limit: int | str | None
+# Whether to use low_cpu_mem_usage
+low_cpu_mem_usage: bool | None
+
+# The name of the chat template to use for training, following values are supported:
+# tokenizer_default: Uses the chat template that is available in the
+# tokenizer_config.json. If the chat template is not available in the tokenizer, it will
+# raise an error. This is the default value.
+# alpaca/inst/chatml/gemma/cohere/llama3/phi_3/deepseek_v2/jamba: These chat templates
+# are available in the axolotl codebase at src/axolotl/utils/chat_templates.py.
+# tokenizer_default_fallback_*: where * is the name of the chat template to fallback to.
+# E.g. tokenizer_default_fallback_chatml. This is useful when the chat template is not
+# available in the tokenizer. jinja: Uses a custom jinja template for the chat template.
+# The custom jinja template should be provided in the chat_template_jinja field. The
+# selected chat template will be saved to the tokenizer_config.json for easier
+# inferencing
+chat_template: ChatTemplate | Annotated[str, StringConstraints(pattern='^tokenizer_default_fallback_')] | None
+# Custom jinja template or path to jinja file for chat template. This will be only used
+# if chat_template is set to `jinja` or `null` (in which case chat_template is
+# automatically set to `jinja`). Default is null.
+chat_template_jinja: str | None
+# Additional kwargs to pass to the chat template. This is useful for customizing the
+# chat template. For example, you can pass `thinking=False` to add a generation prompt
+# to the chat template.
+chat_template_kwargs: dict[str, Any] | None
+# Custom EOT (End-of-Turn) tokens to mask/unmask during training. These tokens mark the
+# boundaries between conversation turns. For example: ['/INST', '',
+# '[/SYSTEM_PROMPT]']. If not specified, defaults to just the model's eos_token. This is
+# useful for templates that use multiple delimiter tokens.
+eot_tokens: list[str] | None
+# Changes the default system message. Currently only supports chatml.
+default_system_message: str | None
+
+# Token index or indices to adjust embedding weights to the mean of the other tokens.
+# This is useful when the model has untrained embeddings.
+fix_untrained_tokens: int | list[int] | None
+
+is_preprocess: bool | None
+preprocess_iterable: bool | None
+
+# Total number of tokens - internal use
+total_num_tokens: int | None
+total_supervised_tokens: int | None
+# You can set these packing optimizations AFTER starting a training at least once. The
+# trainer will provide recommended values for these values.
+sample_packing_eff_est: float | None
+axolotl_config_path: str | None
+
+# Internal use only - Used to identify which the model is based on
+is_falcon_derived_model: bool | None
+# Internal use only - Used to identify which the model is based on
+is_llama_derived_model: bool | None
+# Internal use only - Used to identify which the model is based on. Please note that if
+# you set this to true, `padding_side` will be set to 'left' by default
+is_mistral_derived_model: bool | None
+# Internal use only - Used to identify which the model is based on
+is_qwen_derived_model: bool | None
+
+# Add plugins to extend the pipeline. See `src/axolotl/integrations` for the available
+# plugins or doc below for more details.
+# https://docs.axolotl.ai/docs/custom_integrations.html
+plugins: list[str] | None
+
+# This is the huggingface model that contains *.pt, *.safetensors, or *.bin files. This
+# can also be a relative path to a model on disk
+base_model: str (required)
+# If the base_model repo on hf hub doesn't include configuration .json files, You can
+# set that here, or leave this empty to default to base_model
+base_model_config: str | None
+cls_model_config: str | None
+# Optional tokenizer configuration path in case you want to use a different tokenizer
+# than the one defined in the base model
+tokenizer_config: str | None
+# use_fast option for tokenizer loading from_pretrained, default to True
+tokenizer_use_fast: bool | None
+# Whether to use the legacy tokenizer setting, defaults to True
+tokenizer_legacy: bool | None
+# Whether to use mistral-common tokenizer. If set to True, it will use the mistral-
+# common tokenizer.
+tokenizer_use_mistral_common: bool | None
+# Corresponding tokenizer for the model AutoTokenizer is a good choice
+tokenizer_type: str | None
+# transformers processor class
+processor_type: str | None
+# Whether to save jinja files for tokenizer, transformers default is True
+tokenizer_save_jinja_files: bool | None = True
+# Trust remote code for untrusted source
+trust_remote_code: bool | None
+
+# Don't move the model to the device before sharding. Set to `false` to revert to legacy
+# behavior.
+experimental_skip_move_to_device: bool | None = True
+
+# Use custom kernels, e.g. MegaBlocks.
+use_kernels: bool | None
+
+# Model loading quantization config
+model_quantization_config: Literal['Mxfp4Config'] | None
+# kwargs for model quantization config
+model_quantization_config_kwargs: dict[str, Any] | None
+
+# Where to save the full-finetuned model to
+output_dir: str = ./model-out
+# push checkpoints to hub
+hub_model_id: str | None
+# how to push checkpoints to hub
+hub_strategy: str | None
+# Save model as safetensors (require safetensors package). Default True
+save_safetensors: bool | None = True
+
+# This will attempt to quantize the model down to 8 bits and use adam 8 bit optimizer
+load_in_8bit: bool | None = False
+# Use bitsandbytes 4 bit
+load_in_4bit: bool | None = False
+
+# If you want to use 'lora' or 'qlora' or leave blank to train all parameters in
+# original model
+adapter: str | None
+# If you already have a lora model trained that you want to load, put that here. This
+# means after training, if you want to test the model, you should set this to the value
+# of `output_dir`. Note that if you merge an adapter to the base model, a new
+# subdirectory `merged` will be created under the `output_dir`.
+lora_model_dir: str | None
+lora_r: int | None
+lora_alpha: int | None
+lora_fan_in_fan_out: bool | None
+lora_target_modules: str | list[str] | None
+lora_target_parameters: str | list[str] | None
+# If true, will target all linear modules
+lora_target_linear: bool | None
+# If you added new tokens to the tokenizer, you may need to save some LoRA modules
+# because they need to know the new tokens. For LLaMA and Mistral, you need to save
+# `embed_tokens` and `lm_head`. It may vary for other models. `embed_tokens` converts
+# tokens to embeddings, and `lm_head` converts embeddings to token probabilities.
+lora_modules_to_save: list[str] | None
+lora_dropout: float | None = 0.0
+# The layer indices to transform, otherwise, apply to all layers
+peft_layers_to_transform: list[int] | None
+peft_layers_pattern: list[str] | None
+
+peft: PeftConfig | None
+ # For PeftConfig:
+ # Configuration options for loftq initialization for LoRA
+ loftq_config: LoftQConfig | None
+ # For LoftQConfig:
+ # typically 4 bits
+ loftq_bits: int = 4
+
+# Whether to use DoRA.
+peft_use_dora: bool | None
+# Whether to use RSLoRA.
+peft_use_rslora: bool | None
+# List of layer indices to replicate.
+peft_layer_replication: list[tuple[int, int]] | None
+# How to initialize LoRA weights. Default to True which is MS original implementation.
+peft_init_lora_weights: bool | str | None
+# A list of token indices to fine-tune on the `embed_tokens` layer. Otherwise, a dict
+# mapping an embedding layer name to its trainable token indices. See
+# https://huggingface.co/docs/peft/v0.17.0/en/developer_guides/lora#efficiently-train-
+# tokens-alongside-lora
+peft_trainable_token_indices: list[int] | dict[str, list[int]] | None
+
+# load qlora model in sharded format for FSDP using answer.ai technique.
+qlora_sharded_model_loading: bool | None = False
+# Do the LoRA/PEFT loading on CPU -- this is required if the base model is so large it
+# takes up most or all of the available GPU VRAM, e.g. during a model and LoRA merge
+lora_on_cpu: bool | None
+# Whether you are training a 4-bit GPTQ quantized model
+gptq: bool | None
+# optional overrides to the bnb 4bit quantization configuration
+bnb_config_kwargs: dict[str, Any] | None
+
+# loraplus learning rate ratio lr_B / lr_A. Recommended value is 2^4.
+loraplus_lr_ratio: float | None
+# loraplus learning rate for lora embedding layers. Default value is 1e-6.
+loraplus_lr_embedding: float | None = 1e-06
+
+merge_lora: bool | None
+
+# Whether to use ReLoRA. Use with jagged_restart_*steps options.
+relora: bool | None
+# threshold for optimizer magnitude when pruning
+relora_prune_ratio: float | None
+# True to perform lora weight merges on cpu during restarts, for modest gpu memory
+# savings
+relora_cpu_offload: bool | None
+
+# how often to reset for jagged restarts
+jagged_restart_steps: int | None
+# how many warmup steps to take after reset for jagged restarts
+jagged_restart_warmup_steps: int | None
+# how many anneal steps to take before reset for jagged restarts
+jagged_restart_anneal_steps: int | None
+
+# If greater than 1, backpropagation will be skipped and the gradients will be
+# accumulated for the given number of steps.
+gradient_accumulation_steps: int | None = 1
+# The number of samples to include in each batch. This is the number of samples sent to
+# each GPU. Batch size per gpu = micro_batch_size * gradient_accumulation_steps
+micro_batch_size: int | None = 1
+# Total batch size, we do not recommended setting this manually
+batch_size: int | None
+# per gpu micro batch size for evals, defaults to value of micro_batch_size
+eval_batch_size: int | None
+
+# whether to find batch size that fits in memory. Passed to underlying transformers
+# Trainer
+auto_find_batch_size: bool | None
+
+# Whether to mask out or include the human's prompt from the training labels
+train_on_inputs: bool | None = False
+# Group similarly sized data to minimize padding. May be slower to start, as it must
+# download and sort the entire dataset. Note that training loss may have an oscillating
+# pattern with this enabled.
+group_by_length: bool | None
+
+learning_rate: str | float (required)
+embedding_lr: float | None
+embedding_lr_scale: float | None
+# Specify weight decay
+weight_decay: float | None = 0.0
+# Specify optimizer
+optimizer: OptimizerNames | CustomSupportedOptimizers | None = OptimizerNames.ADAMW_TORCH_FUSED
+# Dictionary of arguments to pass to the optimizer
+optim_args: str | dict[str, Any] | None
+# The target modules to optimize, i.e. the module names that you would like to train,
+# right now this is used only for GaLore algorithm
+optim_target_modules: list[str] | Literal['all_linear'] | None
+# Path to torch distx for optim 'adamw_anyprecision'
+torchdistx_path: str | None
+lr_scheduler: SchedulerType | Literal['one_cycle'] | Literal['rex'] | None = SchedulerType.COSINE
+# Specify a scheduler and kwargs to use with the optimizer
+lr_scheduler_kwargs: dict[str, Any] | None
+lr_quadratic_warmup: bool | None
+# decay lr to some percentage of the peak lr, e.g. cosine_min_lr_ratio=0.1 for 10% of
+# peak lr
+cosine_min_lr_ratio: float | None
+# freeze lr at some percentage of the step, e.g. cosine_constant_lr_ratio=0.8 means
+# start cosine_min_lr at 80% of training step
+cosine_constant_lr_ratio: float | None
+# Learning rate div factor
+lr_div_factor: float | None
+
+lr_groups: list[LrGroup] | None
+ # For LrGroup:
+ name: str (required)
+ modules: list[str] (required)
+ lr: float (required)
+
+# adamw hyperparams
+adam_epsilon: float | None
+# only used for CAME Optimizer
+adam_epsilon2: float | None
+# adamw hyperparams
+adam_beta1: float | None
+# adamw hyperparams
+adam_beta2: float | None
+# only used for CAME Optimizer
+adam_beta3: float | None
+
+# Dion Optimizer learning rate
+dion_lr: float | None
+# Dion Optimizer momentum
+dion_momentum: float | None
+# Dion Optimizer: r/d fraction for low-rank approximation. Used to compute the low-rank
+# dimension.
+dion_rank_fraction: float | None = 1.0
+# Dion Optimizer: Round up the low-rank dimension to a multiple of this number. This may
+# be useful to ensure even sharding.
+dion_rank_multiple_of: int | None = 1
+
+# Gradient clipping max norm
+max_grad_norm: float | None
+num_epochs: float = 1.0
+
+use_wandb: bool | None
+# Set the name of your wandb run
+wandb_name: str | None
+# Set the ID of your wandb run
+wandb_run_id: str | None
+# "offline" to save run metadata locally and not sync to the server, "disabled" to turn
+# off wandb
+wandb_mode: str | None
+# Your wandb project name
+wandb_project: str | None
+# A wandb Team name if using a Team
+wandb_entity: str | None
+wandb_watch: str | None
+# "checkpoint" to log model to wandb Artifacts every `save_steps` or "end" to log only
+# at the end of training
+wandb_log_model: str | None
+
+use_mlflow: bool | None
+# URI to mlflow
+mlflow_tracking_uri: str | None
+# Your experiment name
+mlflow_experiment_name: str | None
+# Your run name
+mlflow_run_name: str | None
+# set to true to copy each saved checkpoint on each save to mlflow artifact registry
+hf_mlflow_log_artifacts: bool | None
+
+# Enable or disable Comet integration.
+use_comet: bool | None
+# API key for Comet. Recommended to set via `comet login`.
+comet_api_key: str | None
+# Workspace name in Comet. Defaults to the user's default workspace.
+comet_workspace: str | None
+# Project name in Comet. Defaults to Uncategorized.
+comet_project_name: str | None
+# Identifier for the experiment. Used to append data to an existing experiment or
+# control the key of new experiments. Default to a random key.
+comet_experiment_key: str | None
+# Create a new experiment ("create") or log to an existing one ("get"). Default
+# ("get_or_create") auto-selects based on configuration.
+comet_mode: str | None
+# Set to True to log data to Comet server, or False for offline storage. Default is
+# True.
+comet_online: bool | None
+# Dictionary for additional configuration settings, see the doc for more details.
+comet_experiment_config: dict[str, Any] | None
+
+# Enable OpenTelemetry metrics collection and Prometheus export
+use_otel_metrics: bool | None = False
+# Host to bind the OpenTelemetry metrics server to
+otel_metrics_host: str | None = localhost
+# Port for the Prometheus metrics HTTP server
+otel_metrics_port: int | None = 8000
+
+# the number of activate layers in LISA
+lisa_n_layers: int | None
+# how often to switch layers in LISA
+lisa_step_interval: int | None
+# path under the model to access the layers
+lisa_layers_attribute: str | None = model.layers
+
+gradio_title: str | None
+gradio_share: bool | None
+gradio_server_name: str | None
+gradio_server_port: int | None
+gradio_max_new_tokens: int | None
+gradio_temperature: float | None
+
+use_ray: bool = False
+ray_run_name: str | None
+ray_num_workers: int = 1
+resources_per_worker: dict
+
+# The size of the image to resize to. It can be an integer (resized into padded-square
+# image) or a tuple (width, height).If not provided, we will attempt to load from
+# preprocessor.size, otherwise, images won't be resized.
+image_size: int | tuple[int, int] | None
+# The resampling algorithm to use for image resizing. Default is bilinear. Please refer
+# to PIL.Image.Resampling for more details.
+image_resize_algorithm: Literal['bilinear', 'bicubic', 'lanczos'] | Resampling | None
+
+# optional overrides to the base model configuration
+overrides_of_model_config: dict[str, Any] | None
+# optional overrides the base model loading from_pretrained
+overrides_of_model_kwargs: dict[str, Any] | None
+# If you want to specify the type of model to load, AutoModelForCausalLM is a good
+# choice too
+type_of_model: str | None
+# You can specify to choose a specific model revision from huggingface hub
+revision_of_model: str | None
+
+max_packed_sequence_len: int | None
+rope_scaling: Any | None
+noisy_embedding_alpha: float | None
+dpo_beta: float | None
+evaluation_strategy: str | None
+```
+
+---
+
+##
+
+**URL:** https://docs.axolotl.ai
+
+**Contents:**
+- 🎉 Latest Updates
+- ✨ Overview
+- 🚀 Quick Start - LLM Fine-tuning in Minutes
+ - Google Colab
+ - Installation
+ - Using pip
+ - Using Docker
+ - Cloud Providers
+ - Your First Fine-tune
+- 📚 Documentation
+
+A Free and Open Source LLM Fine-tuning Framework
+
+Axolotl is a free and open-source tool designed to streamline post-training and fine-tuning for the latest large language models (LLMs).
+
+Installing with Docker can be less error prone than installing in your own environment.
+
+Other installation approaches are described here.
+
+That’s it! Check out our Getting Started Guide for a more detailed walkthrough.
+
+Contributions are welcome! Please see our Contributing Guide for details.
+
+Interested in sponsoring? Contact us at [email protected]
+
+If you use Axolotl in your research or projects, please cite it as follows:
+
+This project is licensed under the Apache 2.0 License - see the LICENSE file for details.
+
+**Examples:**
+
+Example 1 (bash):
+```bash
+pip3 install -U packaging==23.2 setuptools==75.8.0 wheel ninja
+pip3 install --no-build-isolation axolotl[flash-attn,deepspeed]
+
+# Download example axolotl configs, deepspeed configs
+axolotl fetch examples
+axolotl fetch deepspeed_configs # OPTIONAL
+```
+
+Example 2 (bash):
+```bash
+docker run --gpus '"all"' --rm -it axolotlai/axolotl:main-latest
+```
+
+Example 3 (bash):
+```bash
+# Fetch axolotl examples
+axolotl fetch examples
+
+# Or, specify a custom path
+axolotl fetch examples --dest path/to/folder
+
+# Train a model using LoRA
+axolotl train examples/llama-3/lora-1b.yml
+```
+
+Example 4 (unknown):
+```unknown
+@software{axolotl,
+ title = {Axolotl: Open Source LLM Post-Training},
+ author = {{Axolotl maintainers and contributors}},
+ url = {https://github.com/axolotl-ai-cloud/axolotl},
+ license = {Apache-2.0},
+ year = {2023}
+}
+```
+
+---
+
+## Quickstart
+
+**URL:** https://docs.axolotl.ai/docs/getting-started.html
+
+**Contents:**
+- Quickstart
+- 1 Quick Example
+- 2 Understanding the Process
+ - 2.1 The Configuration File
+ - 2.2 Training
+- 3 Your First Custom Training
+- 4 Common Tasks
+ - 4.1 Testing Your Model
+ - 4.2 Using a UI
+ - 4.3 Preprocessing Data
+
+This guide will walk you through your first model fine-tuning project with Axolotl.
+
+Let’s start by fine-tuning a small language model using LoRA. This example uses a 1B parameter model to ensure it runs on most GPUs. Assuming axolotl is installed (if not, see our Installation Guide)
+
+That’s it! Let’s understand what just happened.
+
+The YAML configuration file controls everything about your training. Here’s what (part of) our example config looks like:
+
+load_in_8bit: true and adapter: lora enables LoRA adapter finetuning.
+
+See our config options for more details.
+
+When you run axolotl train, Axolotl:
+
+Let’s modify the example for your own data:
+
+This specific config is for LoRA fine-tuning a model with instruction tuning data using the alpaca dataset format, which has the following format:
+
+Please see our Dataset Formats for more dataset formats and how to format them.
+
+The same yaml file is used for training, inference, and merging.
+
+After training, test your model:
+
+More details can be found in Inference.
+
+Launch a Gradio interface:
+
+For large datasets, preprocess first:
+
+Please make sure to set dataset_prepared_path: in your config to set the path to save the prepared dataset.
+
+More details can be found in Dataset Preprocessing.
+
+To merge the LoRA weights back into the base model, run:
+
+The merged model will be saved in the {output_dir}/merged directory.
+
+More details can be found in Merging LoRA weights.
+
+Now that you have the basics, you might want to:
+
+Check our other guides for details on these topics:
+
+**Examples:**
+
+Example 1 (bash):
+```bash
+axolotl fetch examples
+```
+
+Example 2 (bash):
+```bash
+axolotl train examples/llama-3/lora-1b.yml
+```
+
+Example 3 (yaml):
+```yaml
+base_model: NousResearch/Llama-3.2-1B
+
+load_in_8bit: true
+adapter: lora
+
+datasets:
+ - path: teknium/GPT4-LLM-Cleaned
+ type: alpaca
+dataset_prepared_path: last_run_prepared
+val_set_size: 0.1
+output_dir: ./outputs/lora-out
+```
+
+Example 4 (yaml):
+```yaml
+base_model: NousResearch/Nous-Hermes-llama-1b-v1
+
+load_in_8bit: true
+adapter: lora
+
+# Training settings
+micro_batch_size: 2
+num_epochs: 3
+learning_rate: 0.0003
+
+# Your dataset
+datasets:
+ - path: my_data.jsonl # Your local data file
+ type: alpaca # Or other format
+```
+
+---
+
+## Multipack (Sample Packing)
+
+**URL:** https://docs.axolotl.ai/docs/multipack.html
+
+**Contents:**
+- Multipack (Sample Packing)
+- Visualization of Multipack with Flash Attention
+- Multipack without Flash Attention
+
+Because Flash Attention simply drops the attention mask, we do not need to construct a 4d attention mask. We only need to concatenate the sequences into a single batch and let flash attention know where each new sequence begins.
+
+4k context, bsz =4, each character represents 256 tokens X represents a padding token
+
+after padding to longest input in each step
+
+w packing ( note it’s the same effective number of tokens per step, but a true bsz of 1)
+
+cu_seqlens: [[ 0, 11, 17, 24, 28, 36, 41 44, 48, 51, 55, 60, 64]]
+
+Multipack can still be achieved without Flash attention, but with lower packing efficiency as we are not able to join multiple batches into a single batch due to context length limits without flash attention. We can use either Pytorch’s Scaled Dot Product Attention implementation or native Pytorch attention implementation along with 4d attention masks to pack sequences together and avoid cross attention.
+
+**Examples:**
+
+Example 1 (unknown):
+```unknown
+0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5
+[[ A A A A A A A A A A A ]
+ B B B B B B ]
+ C C C C C C C ]
+ D D D D ]]
+
+[[ E E E E E E E E ]
+ [ F F F F ]
+ [ G G G ]
+ [ H H H H ]]
+
+[[ I I I ]
+ [ J J J ]
+ [ K K K K K]
+ [ L L L ]]
+```
+
+Example 2 (unknown):
+```unknown
+0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5
+[[ A A A A A A A A A A A ]
+ B B B B B B X X X X X X ]
+ C C C C C C C X X X X ]
+ D D D D X X X X X X X ]]
+
+[[ E E E E E E E E ]
+ [ F F F F X X X X ]
+ [ G G G X X X X X ]
+ [ H H H H X X X X ]]
+
+[[ I I I X X ]
+ [ J J J X X ]
+ [ K K K K K ]
+ [ L L L X X ]]
+```
+
+Example 3 (unknown):
+```unknown
+0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5
+[[ A A A A A A A A A A A B B B B B
+ B C C C C C C C D D D D E E E E
+ E E E E F F F F F G G G H H H H
+ I I I J J J J K K K K K L L L X ]]
+```
+
+---
+
+## Batch size vs Gradient accumulation
+
+**URL:** https://docs.axolotl.ai/docs/batch_vs_grad.html
+
+**Contents:**
+- Batch size vs Gradient accumulation
+
+Gradient accumulation means accumulating gradients over several mini-batches and updating the model weights afterward. When the samples in each batch are diverse, this technique doesn’t significantly impact learning.
+
+This method allows for effective training with larger effective batch sizes without needing proportionally larger memory. Here’s why:
+
+Memory Consumption with Batch Size: The primary reason increasing the batch size impacts memory is due to the storage requirements for intermediate activations. When you forward propagate a batch through a network, you have to store the activations at each layer for each sample in the batch, because these activations are used during backpropagation to compute gradients. Therefore, larger batches mean more activations, leading to greater GPU memory consumption.
+
+Gradient Accumulation: With gradient accumulation, you’re effectively simulating a larger batch size by accumulating gradients over several smaller batches (or micro-batches). However, at any given time, you’re only forward and backward propagating a micro-batch. This means you only store activations for the micro-batch, not the full accumulated batch. As a result, you can simulate the effect of a larger batch size without the memory cost of storing activations for a large batch.
+
+Example 1: Micro batch size: 3 Gradient accumulation steps: 2 Number of GPUs: 3 Total batch size = 3 * 2 * 3 = 18
+
+Example 2: Micro batch size: 2 Gradient accumulation steps: 1 Number of GPUs: 3 Total batch size = 2 * 1 * 3 = 6
+
+**Examples:**
+
+Example 1 (unknown):
+```unknown
+| GPU 1 | GPU 2 | GPU 3 |
+|----------------|----------------|----------------|
+| S1, S2, S3 | S4, S5, S6 | S7, S8, S9 |
+| e1, e2, e3 | e4, e5, e6 | e7, e8, e9 |
+|----------------|----------------|----------------|
+| → (accumulate) | → (accumulate) | → (accumulate) |
+|----------------|----------------|----------------|
+| S10, S11, S12 | S13, S14, S15 | S16, S17, S18 |
+| e10, e11, e12 | e13, e14, e15 | e16, e17, e18 |
+|----------------|----------------|----------------|
+| → (apply) | → (apply) | → (apply) |
+
+Accumulated gradient for the weight w1 after the second iteration (considering all GPUs):
+Total gradient for w1 = e1 + e2 + e3 + e4 + e5 + e6 + e7 + e8 + e9 + e10 + e11 + e12 + e13 + e14 + e15 + e16 + e17 + e18
+
+Weight update for w1:
+w1_new = w1_old - learning rate x (Total gradient for w1 / 18)
+```
+
+Example 2 (unknown):
+```unknown
+| GPU 1 | GPU 2 | GPU 3 |
+|-----------|-----------|-----------|
+| S1, S2 | S3, S4 | S5, S6 |
+| e1, e2 | e3, e4 | e5, e6 |
+|-----------|-----------|-----------|
+| → (apply) | → (apply) | → (apply) |
+
+Accumulated gradient for the weight w1 (considering all GPUs):
+Total gradient for w1 = e1 + e2 + e3 + e4 + e5 + e6
+
+Weight update for w1:
+w1_new = w1_old - learning rate × (Total gradient for w1 / 6)
+```
+
+---
+
+## Debugging
+
+**URL:** https://docs.axolotl.ai/docs/debugging.html
+
+**Contents:**
+- Debugging
+- Table of Contents
+- General Tips
+- Debugging with VSCode
+ - Background
+ - Setup
+ - Remote Hosts
+ - Configuration
+ - Customizing your debugger
+ - Video Tutorial
+
+This document provides some tips and tricks for debugging Axolotl. It also provides an example configuration for debugging with VSCode. A good debugging setup is essential to understanding how Axolotl code works behind the scenes.
+
+While debugging it’s helpful to simplify your test scenario as much as possible. Here are some tips for doing so:
+
+[!Important] All of these tips are incorporated into the example configuration for debugging with VSCode below.
+
+Make sure you are using the latest version of axolotl: This project changes often and bugs get fixed fast. Check your git branch and make sure you have pulled the latest changes from main.
+
+Eliminate concurrency: Restrict the number of processes to 1 for both training and data preprocessing:
+
+Use a small dataset: Construct or use a small dataset from HF Hub. When using a small dataset, you will often have to make sure sample_packing: False and eval_sample_packing: False to avoid errors. If you are in a pinch and don’t have time to construct a small dataset but want to use from the HF Hub, you can shard the data (this will still tokenize the entire dataset, but will only use a fraction of the data for training. For example, to shard the dataset into 20 pieces, add the following to your axolotl config):
+
+Use a small model: A good example of a small model is TinyLlama/TinyLlama-1.1B-Chat-v1.0.
+
+Minimize iteration time: Make sure the training loop finishes as fast as possible, with these settings.
+
+Clear Caches: Axolotl caches certain steps and so does the underlying HuggingFace trainer. You may want to clear some of these caches when debugging.
+
+The below example shows how to configure VSCode to debug data preprocessing of the chat_template format. This is the format used when you have the following in your axolotl config:
+
+[!Important] If you are already familiar with advanced VSCode debugging, you can skip the below explanation and look at the files .vscode/launch.json and .vscode/tasks.json for an example configuration.
+
+[!Tip] If you prefer to watch a video, rather than read, you can skip to the video tutorial below (but doing both is recommended).
+
+Make sure you have an editable install of Axolotl, which ensures that changes you make to the code are reflected at runtime. Run the following commands from the root of this project:
+
+If you developing on a remote host, you can easily use VSCode to debug remotely. To do so, you will need to follow this remote - SSH guide. You can also see the video below on Docker and Remote SSH debugging.
+
+The easiest way to get started is to modify the .vscode/launch.json file in this project. This is just an example configuration, so you may need to modify or copy it to suit your needs.
+
+For example, to mimic the command cd devtools && CUDA_VISIBLE_DEVICES=0 accelerate launch -m axolotl.cli.train dev_chat_template.yml, you would use the below configuration1. Note that we add additional flags that override the axolotl config and incorporate the tips above (see the comments). We also set the working directory to devtools and set the env variable HF_HOME to a temporary folder that is later partially deleted. This is because we want to delete the HF dataset cache before each run in order to ensure that the data preprocessing code is run from scratch.
+
+Additional notes about this configuration:
+
+[!Tip] You may not want to delete these folders. For example, if you are debugging model training instead of data pre-processing, you may NOT want to delete the cache or output folders. You may also need to add additional tasks to the tasks.json file depending on your use case.
+
+Below is the ./vscode/tasks.json file that defines the cleanup-for-dataprep task. This task is run before each debugging session when you use the above configuration. Note how there are two tasks that delete the two folders mentioned above. The third task cleanup-for-dataprep is a composite task that combines the two tasks. A composite task is necessary because VSCode does not allow you to specify multiple tasks in the preLaunchTask argument of the launch.json file.
+
+Your debugging use case may differ from the example above. The easiest thing to do is to put your own axolotl config in the devtools folder and modify the launch.json file to use your config. You may also want to modify the preLaunchTask to delete different folders or not delete anything at all.
+
+The following video tutorial walks through the above configuration and demonstrates how to debug with VSCode, (click the image below to watch):
+
+Using official Axolotl Docker images is a great way to debug your code, and is a very popular way to use Axolotl. Attaching VSCode to Docker takes a few more steps.
+
+On the host that is running axolotl (ex: if you are using a remote host), clone the axolotl repo and change your current directory to the root:
+
+[!Tip] If you already have axolotl cloned on your host, make sure you have the latest changes and change into the root of the project.
+
+Next, run the desired docker image and mount the current directory. Below is a docker command you can run to do this:2
+
+[!Tip] To understand which containers are available, see the Docker section of the README and the DockerHub repo. For details of how the Docker containers are built, see axolotl’s Docker CI builds.
+
+You will now be in the container. Next, perform an editable install of Axolotl:
+
+Next, if you are using a remote host, Remote into this host with VSCode. If you are using a local host, you can skip this step.
+
+Next, select Dev Containers: Attach to Running Container... using the command palette (CMD + SHIFT + P) in VSCode. You will be prompted to select a container to attach to. Select the container you just created. You will now be in the container with a working directory that is at the root of the project. Any changes you make to the code will be reflected both in the container and on the host.
+
+Now you are ready to debug as described above (see Debugging with VSCode).
+
+Here is a short video that demonstrates how to attach to a Docker container on a remote host:
+
+The config actually mimics the command CUDA_VISIBLE_DEVICES=0 python -m accelerate.commands.launch -m axolotl.cli.train devtools/chat_template.yml, but this is the same thing.↩︎
+
+Many of the below flags are recommended best practices by Nvidia when using nvidia-container-toolkit. You can read more about these flags here.↩︎
+
+**Examples:**
+
+Example 1 (yaml):
+```yaml
+datasets:
+ ...
+ shards: 20
+```
+
+Example 2 (yaml):
+```yaml
+datasets:
+ - path: # example on HF Hub: fozziethebeat/alpaca_messages_2k_test
+ type: chat_template
+```
+
+Example 3 (bash):
+```bash
+pip3 install packaging
+pip3 install --no-build-isolation -e '.[flash-attn,deepspeed]'
+```
+
+Example 4 (json):
+```json
+// .vscode/launch.json
+{
+ "version": "0.2.0",
+ "configurations": [
+ {
+ "name": "Debug axolotl prompt - chat_template",
+ "type": "python",
+ "module": "accelerate.commands.launch",
+ "request": "launch",
+ "args": [
+ "-m", "axolotl.cli.train", "dev_chat_template.yml",
+ // The flags below simplify debugging by overriding the axolotl config
+ // with the debugging tips above. Modify as needed.
+ "--dataset_num_proc=1", // limits data preprocessing to one process
+ "--max_steps=1", // limits training to just one step
+ "--batch_size=1", // minimizes batch size
+ "--micro_batch_size=1", // minimizes batch size
+ "--val_set_size=0", // disables validation
+ "--sample_packing=False", // disables sample packing which is necessary for small datasets
+ "--eval_sample_packing=False",// disables sample packing on eval set
+ "--dataset_prepared_path=temp_debug/axolotl_outputs/data", // send data outputs to a temp folder
+ "--output_dir=temp_debug/axolotl_outputs/model" // send model outputs to a temp folder
+ ],
+ "console": "integratedTerminal", // show output in the integrated terminal
+ "cwd": "${workspaceFolder}/devtools", // set working directory to devtools from the root of the project
+ "justMyCode": true, // step through only axolotl code
+ "env": {"CUDA_VISIBLE_DEVICES": "0", // Since we aren't doing distributed training, we need to limit to one GPU
+ "HF_HOME": "${workspaceFolder}/devtools/temp_debug/.hf-cache"}, // send HF cache to a temp folder
+ "preLaunchTask": "cleanup-for-dataprep", // delete temp folders (see below)
+ }
+ ]
+}
+```
+
+---
+
+## Docker
+
+**URL:** https://docs.axolotl.ai/docs/docker.html
+
+**Contents:**
+- Docker
+- Base
+ - Image
+ - Tags format
+- Main
+ - Image
+ - Tags format
+- Cloud
+ - Image
+ - Tags format
+
+This section describes the different Docker images that are released by AxolotlAI at Docker Hub.
+
+For Blackwell GPUs, please use the tags with PyTorch 2.7.1 and CUDA 12.8.
+
+The base image is the most minimal image that can install Axolotl. It is based on the nvidia/cuda image. It includes python, torch, git, git-lfs, awscli, pydantic, and more.
+
+The main image is the image that is used to run Axolotl. It is based on the axolotlai/axolotl-base image and includes the Axolotl codebase, dependencies, and more.
+
+There may be some extra tags appended to the image, like -vllm which installs those packages.
+
+The cloud image is the image that is used to run Axolotl in the cloud. It is based on the axolotlai/axolotl image and sets ENV variables like HuggingFace cache directories for volume mounts, tmux, and more for different cloud providers.
+
+Jupyter lab is run by default. Set JUPYTER_DISABLE=1 in the environment variables to disable it.
+
+This uses the same tags as the main image.
+
+We recommend mounting volumes to /workspace/data for data persistence. /workspace/axolotl contains the source code and is ephemeral.
+
+This is the same as the cloud image but without tmux.
+
+The naming may be a bit confusing as it has -term appended to the end.
+
+This uses the same tags as the cloud image.
+
+**Examples:**
+
+Example 1 (unknown):
+```unknown
+axolotlai/axolotl-base
+```
+
+Example 2 (bash):
+```bash
+main-base-py{python_version}-cu{cuda_version}-{pytorch_version}
+```
+
+Example 3 (unknown):
+```unknown
+axolotlai/axolotl
+```
+
+Example 4 (bash):
+```bash
+# on push to main
+main-py{python_version}-cu{cuda_version}-{pytorch_version}
+
+# latest main (currently torch 2.6.0, python 3.11, cuda 12.4)
+main-latest
+
+# nightly build
+{branch}-{date_in_YYYYMMDD}-py{python_version}-cu{cuda_version}-{pytorch_version}
+
+# tagged release
+{version}
+```
+
+---
diff --git a/src/skills/bundled/baoyu-article-illustrator/PORT_NOTES.md b/src/skills/bundled/baoyu-article-illustrator/PORT_NOTES.md
new file mode 100644
index 0000000..d81dbc9
--- /dev/null
+++ b/src/skills/bundled/baoyu-article-illustrator/PORT_NOTES.md
@@ -0,0 +1,48 @@
+# Port Notes — baoyu-article-illustrator
+
+Ported from [JimLiu/baoyu-skills](https://github.com/JimLiu/baoyu-skills) v1.57.0.
+
+## Changes from upstream
+
+`SKILL.md`, `references/workflow.md`, `references/usage.md`, `references/style-presets.md`, `references/styles.md`, `references/prompt-construction.md`, and `prompts/system.md` were adapted. The 23 style files and 4 palette files are verbatim copies. The `references/config/` directory was removed entirely.
+
+### Adaptations
+
+| Change | Upstream | Hermes |
+|--------|----------|--------|
+| Metadata namespace | `openclaw` | `hermes` |
+| Trigger | `/baoyu-article-illustrator` slash command + CLI flags | Natural language skill matching |
+| User config | EXTEND.md (project/user/XDG paths) + first-time-setup | Removed — not part of Hermes infra |
+| User prompts | `AskUserQuestion` (batched, multi-question) | `clarify` tool (one question at a time) |
+| Image generation | `baoyu-imagine` (Bun/TypeScript, multi-provider, accepts `--ref`, writes to local path) | `image_generate` (returns URL only; agent downloads via `terminal`/`curl`) |
+| Backend selection | User picks provider via CLI flags | Not agent-selectable — `image_generate` uses the user-configured FAL model. Removed hardcoded "nano banana pro" line from `prompts/system.md`. |
+| Reference images | Passed to backend via `--ref`, copied via shell | `vision_analyze` extracts a textual description (binary never touched by `write_file`/`read_file`); description is embedded in prompts. Optional `terminal cp` for a local record. |
+| Platform support | Linux/macOS/Windows/WSL/PowerShell | Linux/macOS only |
+| File operations | Bash commands | Hermes file tools: `write_file`/`read_file` for text, `terminal` for binaries and URL downloads, `vision_analyze` for reading images |
+| Watermark | Driven by EXTEND.md `watermark.enabled` | Optional — user asks for it per-article |
+| Output directory | EXTEND.md `default_output_dir` (imgs-subdir / same-dir / illustrations-subdir / independent) | Defaults based on input type; user overrides in request |
+
+### What was preserved
+
+- Type × Style × Palette three-dimension framework
+- All style definitions (23 files, verbatim)
+- All palette definitions (4 files, verbatim)
+- Core reference files (workflow, prompt-construction, styles, style-presets) — adapted for Hermes tooling
+- Core principles and workflow structure (analyze → confirm → outline → prompts → generate)
+- Prompt-file-as-reproducibility-record discipline
+- Author, version, homepage attribution
+
+## Syncing with upstream
+
+To pull upstream updates:
+
+```bash
+# Compare versions
+curl -sL https://raw.githubusercontent.com/JimLiu/baoyu-skills/main/skills/baoyu-article-illustrator/SKILL.md | head -5
+# Look for version: line
+
+# Diff style/palette files (safe to overwrite — unchanged from upstream)
+diff <(curl -sL https://raw.githubusercontent.com/JimLiu/baoyu-skills/main/skills/baoyu-article-illustrator/references/styles/blueprint.md) references/styles/blueprint.md
+```
+
+`references/styles/*` and `references/palettes/*` can be overwritten directly. `SKILL.md`, `references/workflow.md`, `references/usage.md`, `references/style-presets.md`, `references/styles.md`, `references/prompt-construction.md`, and `prompts/system.md` must be manually merged since they contain Hermes-specific adaptations (tool wiring, backend neutrality, removed EXTEND.md references).
diff --git a/src/skills/bundled/baoyu-article-illustrator/SKILL.md b/src/skills/bundled/baoyu-article-illustrator/SKILL.md
new file mode 100644
index 0000000..6adbebf
--- /dev/null
+++ b/src/skills/bundled/baoyu-article-illustrator/SKILL.md
@@ -0,0 +1,207 @@
+---
+name: baoyu-article-illustrator
+description: "Article illustrations: type × style × palette consistency."
+version: 1.57.0
+author: 宝玉 (JimLiu)
+license: MIT
+platforms: [linux, macos, windows]
+metadata:
+ hermes:
+ tags: [article-illustration, creative, image-generation]
+ category: creative
+ homepage: https://github.com/JimLiu/baoyu-skills#baoyu-article-illustrator
+---
+
+# Article Illustrator
+
+Adapted from [baoyu-article-illustrator](https://github.com/JimLiu/baoyu-skills) for Hermes Agent's tool ecosystem.
+
+Analyze articles, identify illustration positions, generate images with **Type × Style × Palette** consistency.
+
+## When to Use
+
+Trigger this skill when the user asks to illustrate an article, add images to an article, generate illustrations for content, or uses phrases like "为文章配图", "illustrate article", or "add images". The user provides an article (file path or pasted content) and optionally specifies type, style, palette, or density.
+
+## Three Dimensions
+
+| Dimension | Controls | Examples |
+|-----------|----------|----------|
+| **Type** | Information structure | infographic, scene, flowchart, comparison, framework, timeline |
+| **Style** | Rendering approach | notion, warm, minimal, blueprint, watercolor, elegant |
+| **Palette** | Color scheme (optional) | macaron, warm, neon — overrides style's default colors |
+
+Combine freely: `type=infographic, style=vector-illustration, palette=macaron`.
+
+Or use presets: `edu-visual` → type + style + palette in one shot. See [style-presets.md](references/style-presets.md).
+
+## Types
+
+| Type | Best For |
+|------|----------|
+| `infographic` | Data, metrics, technical |
+| `scene` | Narratives, emotional |
+| `flowchart` | Processes, workflows |
+| `comparison` | Side-by-side, options |
+| `framework` | Models, architecture |
+| `timeline` | History, evolution |
+
+## Styles
+
+See [references/styles.md](references/styles.md) for Core Styles, the full gallery, and Type × Style compatibility.
+
+## Output Structure
+
+```
+{output-dir}/
+├── source-{slug}.{ext} # Only for pasted content
+├── outline.md
+├── prompts/
+│ └── NN-{type}-{slug}.md
+└── NN-{type}-{slug}.png
+```
+
+**Default output directory**:
+
+| Input | Output Directory | Markdown Insert Path |
+|-------|------------------|----------------------|
+| Article file path | `{article-dir}/imgs/` | `imgs/NN-{type}-{slug}.png` |
+| Pasted content | `illustrations/{topic-slug}/` (cwd) | `illustrations/{topic-slug}/NN-{type}-{slug}.png` |
+
+If the user asks for a different layout (e.g., images alongside the article, or a `illustrations/` subdirectory), honor that.
+
+**Slug**: 2-4 words, kebab-case. **Conflict**: append `-YYYYMMDD-HHMMSS`.
+
+## Core Principles
+
+- **Visualize concepts, not metaphors** — if the article uses a metaphor (e.g., "电锯切西瓜"), illustrate the underlying concept, not the literal image.
+- **Labels use article data** — actual numbers, terms, and quotes from the article, not generic placeholders.
+- **Prompt files are reproducibility records** — every illustration must have a saved prompt file under `prompts/` before any image is generated.
+- **Strip secrets** — scan source content for API keys, tokens, or credentials before writing anything to disk.
+
+## Workflow
+
+```
+- [ ] Step 1: Detect reference images (if provided)
+- [ ] Step 2: Analyze content
+- [ ] Step 3: Confirm settings (clarify tool, one question at a time)
+- [ ] Step 4: Generate outline
+- [ ] Step 5: Generate prompts
+- [ ] Step 6: Generate images (image_generate)
+- [ ] Step 7: Finalize
+```
+
+### Step 1: Detect Reference Images
+
+If the user supplies reference images (paths pasted inline, attachments, or a URL):
+
+1. For each reference, call `vision_analyze` with the path/URL and a question asking for style, palette, composition, and subject. Record the returned description in `{output-dir}/references/NN-ref-{slug}.md` via `write_file`.
+2. **Do not** try to copy the binary via `write_file` / `read_file` — those are text-only. If you want a local copy for the record, use `terminal` (`cp "$src" "{output-dir}/references/NN-ref-{slug}.{ext}"`). The skill itself never needs to read the binary; it works off the vision description.
+3. Since `image_generate` doesn't take image inputs, the vision description is what gets embedded in prompts during Step 5.
+
+Full procedures: [references/workflow.md](references/workflow.md#step-1-detect-reference-images).
+
+### Step 2: Analyze
+
+| Analysis | Output |
+|----------|--------|
+| Content type | Technical / Tutorial / Methodology / Narrative |
+| Purpose | information / visualization / imagination |
+| Core arguments | 2-5 main points |
+| Positions | Where illustrations add value |
+
+Read source (file path → `read_file`, or pasted text) and write the analysis to `{output-dir}/analysis.md` using `write_file`.
+
+Full procedures: [references/workflow.md](references/workflow.md#step-2-analyze).
+
+### Step 3: Confirm Settings
+
+Use the `clarify` tool. Since `clarify` handles one question at a time, ask the most important question first. Skip any question whose answer is already present in the user's request.
+
+| Order | Question | Options |
+|-------|----------|---------|
+| Q1 | **Preset or Type** | [Recommended preset], [alt preset], or manual: infographic, scene, flowchart, comparison, framework, timeline, mixed |
+| Q2 | **Density** | minimal (1-2), balanced (3-5), per-section (Recommended), rich (6+) |
+| Q3 | **Style** *(skip if preset chosen in Q1)* | [Recommended], minimal-flat, sci-fi, hand-drawn, editorial, scene, poster |
+| Q4 | **Palette** *(optional)* | Default (style colors), macaron, warm, neon |
+| Q5 | **Language** *(only if article language is ambiguous)* | article language / user language |
+
+Don't ask more than 2-3 `clarify` questions in a row. If the user already specified these in their request, skip entirely.
+
+Full procedures: [references/workflow.md](references/workflow.md#step-3-confirm-settings).
+
+### Step 4: Generate Outline → `outline.md`
+
+Save `{output-dir}/outline.md` using `write_file` with frontmatter (type, density, style, palette, image_count) and one entry per illustration:
+
+```yaml
+## Illustration 1
+**Position**: [section/paragraph]
+**Purpose**: [why]
+**Visual Content**: [what to show]
+**Filename**: 01-infographic-concept-name.png
+```
+
+Full template: [references/workflow.md](references/workflow.md#step-4-generate-outline).
+
+### Step 5: Generate Prompts
+
+**BLOCKING**: Every illustration must have a saved prompt file before any image is generated — the prompt file is the reproducibility record.
+
+For each illustration:
+
+1. Create a prompt file per [references/prompt-construction.md](references/prompt-construction.md).
+2. Save to `{output-dir}/prompts/NN-{type}-{slug}.md` using `write_file` with YAML frontmatter.
+3. Prompts MUST use type-specific templates with structured sections (ZONES / LABELS / COLORS / STYLE / ASPECT).
+4. LABELS MUST include article-specific data: actual numbers, terms, metrics, quotes.
+5. Process references (`direct`/`style`/`palette`) per prompt frontmatter — for `direct` usage, embed a textual description of the reference in the prompt (since `image_generate` doesn't take reference-image inputs).
+
+### Step 6: Generate Images
+
+For each prompt file:
+
+1. Call `image_generate(prompt=..., aspect_ratio=...)`. `image_generate` returns a JSON result containing an image URL; it does NOT write to disk and does NOT accept an output path.
+2. Map the prompt's `ASPECT` to `image_generate`'s enum: `16:9` → `landscape`, `9:16` → `portrait`, `1:1` → `square`. Custom ratios → nearest named aspect.
+3. Download the returned URL to `{output-dir}/NN-{type}-{slug}.png` via `terminal` (e.g. `curl -sSL -o "{output-dir}/NN-{type}-{slug}.png" "{url}"`).
+4. On generation failure, auto-retry once.
+
+Note: the underlying image-generation backend is user-configured (default: FAL FLUX 2 Klein 9B) and is NOT agent-selectable via `image_generate`. Do not write model names into prompts expecting them to route.
+
+### Step 7: Finalize
+
+Insert `` after the corresponding paragraph. Alt text: concise description in the article's language.
+
+Report:
+
+```
+Article Illustration Complete!
+Article: [path] | Type: [type] | Density: [level] | Style: [style] | Palette: [palette or default]
+Images: X/N generated
+```
+
+## Modification
+
+| Action | Steps |
+|--------|-------|
+| Edit | Update prompt → Regenerate → Update reference |
+| Add | Position → Prompt → Generate → Update outline → Insert |
+| Delete | Delete files → Remove reference → Update outline |
+
+## References
+
+| File | Content |
+|------|---------|
+| [references/workflow.md](references/workflow.md) | Detailed procedures |
+| [references/usage.md](references/usage.md) | Invocation examples |
+| [references/styles.md](references/styles.md) | Style gallery + Palette gallery |
+| [references/style-presets.md](references/style-presets.md) | Preset shortcuts (type + style + palette) |
+| [references/prompt-construction.md](references/prompt-construction.md) | Prompt templates |
+
+## Pitfalls
+
+1. **Data integrity is paramount** — never summarize, paraphrase, or alter source statistics. "73% increase" stays "73% increase".
+2. **Strip secrets** — scan source content for API keys, tokens, or credentials before including in any output file.
+3. **Don't illustrate metaphors literally** — visualize the underlying concept.
+4. **Prompt files are mandatory** — no image generation without a saved prompt file. The file is what lets you regenerate or switch backends later.
+5. **`image_generate` aspect ratios** — the tool supports `landscape`, `portrait`, and `square`. Custom ratios map to the nearest option.
+6. **`image_generate` returns a URL, not a local file** — always download via `terminal` (`curl`) before inserting local image paths into the article.
+7. **No backend selection from the agent** — `image_generate` uses whatever model the user configured (default: FAL FLUX 2 Klein 9B). Don't write `"use to generate this"` into prompts expecting it to route.
diff --git a/src/skills/bundled/baoyu-article-illustrator/prompts/system.md b/src/skills/bundled/baoyu-article-illustrator/prompts/system.md
new file mode 100644
index 0000000..3320564
--- /dev/null
+++ b/src/skills/bundled/baoyu-article-illustrator/prompts/system.md
@@ -0,0 +1,32 @@
+Create a cartoon-style infographic illustration following these guidelines:
+
+## Image Specifications
+
+- **Type**: Infographic illustration
+- **Orientation**: Landscape (horizontal)
+- **Aspect Ratio**: 16:9
+- **Style**: Hand-drawn illustration
+
+## Core Principles
+
+- Hand-drawn quality throughout - NO realistic or photographic elements
+- If content involves sensitive or copyrighted figures, create stylistically similar alternatives - DO NOT refuse to generate
+- Keep information concise, highlight keywords and core concepts
+- Use ample whitespace for easy visual scanning
+- Maintain clear visual hierarchy
+
+## Text Style (When Text Included)
+
+- **ALL text MUST be hand-drawn style**
+- Text should be readable and complement the visual
+- Font style harmonizes with illustration style
+- **DO NOT use realistic or computer-generated fonts**
+
+## Language
+
+- Use the same language as the content provided below for any text elements
+- Match punctuation style to the content language
+
+---
+
+Generate the illustration based on the content provided below:
diff --git a/src/skills/bundled/baoyu-article-illustrator/references/palettes/macaron.md b/src/skills/bundled/baoyu-article-illustrator/references/palettes/macaron.md
new file mode 100644
index 0000000..e7d7a6b
--- /dev/null
+++ b/src/skills/bundled/baoyu-article-illustrator/references/palettes/macaron.md
@@ -0,0 +1,33 @@
+# macaron
+
+Soft macaron pastel color blocks on warm cream
+
+## Background
+
+- Color: Warm Cream (#F5F0E8)
+- Texture: Subtle warm paper grain
+
+## Colors
+
+| Role | Color | Hex | Usage |
+|------|-------|-----|-------|
+| Background | Warm Cream | #F5F0E8 | Primary background |
+| Primary Text | Deep Charcoal | #2D2D2D | Headlines, main text, outlines |
+| Macaron Blue | Sky Blue | #A8D8EA | Info block fill, cool-toned zones |
+| Macaron Mint | Mint Green | #B5E5CF | Info block fill, growth/positive zones |
+| Macaron Lavender | Lavender | #D5C6E0 | Info block fill, abstract/concept zones |
+| Macaron Peach | Peach | #FFD5C2 | Info block fill, warm-toned zones |
+| Accent | Coral Red | #E8655A | Key data, warnings, emphasis |
+| Muted Text | Warm Gray | #6B6B6B | Secondary annotations, small labels |
+
+## Accent
+
+Coral Red (#E8655A) for key data, warnings, and emphasis highlights. Use sparingly — one or two elements per illustration.
+
+## Semantic Constraint
+
+Soft pastel macaron color palette. Use block colors as rounded card backgrounds for distinct information sections. Accent coral red sparingly for emphasis on key terms only. Do NOT render color names, hex codes, or role labels as visible text in the image.
+
+## Best For
+
+Educational content, knowledge sharing, concept explainers, tutorials, tech summaries, onboarding materials
diff --git a/src/skills/bundled/baoyu-article-illustrator/references/palettes/mono-ink.md b/src/skills/bundled/baoyu-article-illustrator/references/palettes/mono-ink.md
new file mode 100644
index 0000000..88132f9
--- /dev/null
+++ b/src/skills/bundled/baoyu-article-illustrator/references/palettes/mono-ink.md
@@ -0,0 +1,42 @@
+# mono-ink
+
+Black ink on pure white with sparse semantic accent colors
+
+## Background
+
+- Color: Pure White (#FFFFFF)
+- Texture: Clean, no grain, no tint
+
+## Colors
+
+| Role | Color | Hex | Usage |
+|------|-------|-----|-------|
+| Background | Pure White | #FFFFFF | Canvas |
+| Primary | Near Black | #1A1A1A | All lines, text, figures, arrows |
+| Accent (risk/emphasis) | Coral Red | #E8655A | Risk, problem, gap, key emphasis |
+| Accent (positive) | Muted Teal | #5FA8A8 | Positive, solution, "after" state |
+| Accent (neutral tag) | Dusty Lavender | #9B8AB5 | Neutral tags, category labels |
+| Soft Fill | Pale Gray | #F0F0F0 | Subtle zone backgrounds (optional) |
+
+## Accent
+
+Use black ink for all structural elements — lines, text, figures. Accent colors appear only for semantic highlighting: coral red for risks/gaps/problems, muted teal for positive/solution/after-states, dusty lavender for neutral category tags. Total colored pixels must remain under 10% of canvas. Pale gray may back a subtle zone but must never dominate.
+
+## Semantic Constraint
+
+Black ink on white canvas. Accent colors for semantic highlighting only — total colored pixels under 10% of canvas. Do NOT render color names, hex codes, or role labels as visible text in the image.
+
+## Compatible With
+
+- `ink-notes` (primary, default pairing)
+- `minimal` (strict monochrome variation, drops the style's built-in accent)
+- `sketch` (pencil + ink hybrid look)
+
+## Not Recommended With
+
+- `sketch-notes` — its "no pure white backgrounds" rule conflicts
+- `warm`, `elegant`, `watercolor`, `fantasy-animation` — color-heavy by design, mono-ink strips their identity
+
+## Best For
+
+Professional visual notes, Before/After essays, tech manifestos, framework analogies, whiteboard-presentation explainers
diff --git a/src/skills/bundled/baoyu-article-illustrator/references/palettes/neon.md b/src/skills/bundled/baoyu-article-illustrator/references/palettes/neon.md
new file mode 100644
index 0000000..d863d67
--- /dev/null
+++ b/src/skills/bundled/baoyu-article-illustrator/references/palettes/neon.md
@@ -0,0 +1,33 @@
+# neon
+
+Vibrant neon colors on dark backgrounds
+
+## Background
+
+- Color: Deep Purple (#2D1B4E)
+- Texture: Subtle grid pattern or solid dark
+
+## Colors
+
+| Role | Color | Hex | Usage |
+|------|-------|-----|-------|
+| Background | Deep Purple | #2D1B4E | Primary background |
+| Alt Background | Dark Teal | #0F4C5C | Alternative sections |
+| Primary | Hot Pink | #FF1493 | Main accent |
+| Secondary | Electric Cyan | #00FFFF | Supporting elements |
+| Tertiary | Neon Yellow | #FFFF00 | Highlights |
+| Accent 1 | Lime Green | #32CD32 | Energy, success |
+| Accent 2 | Orange | #FF6B35 | Warmth |
+| Text | White | #FFFFFF | Text elements |
+
+## Accent
+
+Hot Pink (#FF1493) for primary emphasis. High contrast neon-on-dark creates immediate visual impact.
+
+## Semantic Constraint
+
+Vibrant neon-on-dark palette. High contrast, immediate visual impact. Do NOT render color names, hex codes, or role labels as visible text in the image.
+
+## Best For
+
+Gaming, retro tech, 80s/90s nostalgic content, bold editorial, trend and pop culture
diff --git a/src/skills/bundled/baoyu-article-illustrator/references/palettes/warm.md b/src/skills/bundled/baoyu-article-illustrator/references/palettes/warm.md
new file mode 100644
index 0000000..c2e7afa
--- /dev/null
+++ b/src/skills/bundled/baoyu-article-illustrator/references/palettes/warm.md
@@ -0,0 +1,32 @@
+# warm
+
+Warm earth tones on soft peach, no cool colors
+
+## Background
+
+- Color: Soft Peach (#FFECD2)
+- Texture: Warm paper texture
+
+## Colors
+
+| Role | Color | Hex | Usage |
+|------|-------|-----|-------|
+| Background | Soft Peach | #FFECD2 | Primary background |
+| Outlines | Deep Charcoal | #2D2D2D | All element outlines |
+| Primary | Warm Orange | #ED8936 | Main accent color |
+| Secondary | Terracotta | #C05621 | Warm depth |
+| Tertiary | Golden Yellow | #F6AD55 | Highlights, energy |
+| Accent | Deep Brown | #744210 | Grounding, anchoring |
+| Text | Warm Charcoal | #4A4A4A | Text elements |
+
+## Accent
+
+Warm Orange (#ED8936) for primary emphasis. Warm-only palette — no cool colors (no green, blue, purple). Modern-retro feel.
+
+## Semantic Constraint
+
+Warm earth tone palette. Warm-only — no cool colors (no green, blue, purple). Do NOT render color names, hex codes, or role labels as visible text in the image.
+
+## Best For
+
+Product showcases, team introductions, feature grids, brand content, personal growth, lifestyle
diff --git a/src/skills/bundled/baoyu-article-illustrator/references/prompt-construction.md b/src/skills/bundled/baoyu-article-illustrator/references/prompt-construction.md
new file mode 100644
index 0000000..611359e
--- /dev/null
+++ b/src/skills/bundled/baoyu-article-illustrator/references/prompt-construction.md
@@ -0,0 +1,426 @@
+# Prompt Construction
+
+## Prompt File Format
+
+Each prompt file uses YAML frontmatter + content:
+
+```yaml
+---
+illustration_id: 01
+type: infographic
+style: blueprint
+references: # ⚠️ ONLY if files EXIST in references/ directory
+ - ref_id: 01
+ filename: 01-ref-diagram.png
+ usage: direct # direct | style | palette
+---
+
+[Type-specific template content below...]
+```
+
+**⚠️ CRITICAL - When to include `references` field**:
+
+| Situation | Action |
+|-----------|--------|
+| Reference file saved to `references/` | Include in frontmatter ✓ |
+| Style extracted verbally (no file) | DO NOT include in frontmatter, append to prompt body instead |
+| File path in frontmatter but file doesn't exist | ERROR - remove references field |
+
+**Reference Usage Types** (only when file exists):
+
+| Usage | Description | Generation Action |
+|-------|-------------|-------------------|
+| `direct` | Primary visual reference | Describe the reference (composition, subject, style, palette) in prompt text — `image_generate` does not accept reference-image inputs |
+| `style` | Style characteristics only | Describe style in prompt text |
+| `palette` | Color palette extraction | Include colors in prompt |
+
+**If no reference file but style/palette extracted verbally**, append directly to prompt body:
+```
+COLORS (from reference):
+- Primary: #E8756D coral
+- Secondary: #7ECFC0 mint
+...
+
+STYLE (from reference):
+- Clean lines, minimal shadows
+- Gradient backgrounds
+...
+```
+
+---
+
+## Default Composition Requirements
+
+**Apply to ALL prompts by default**:
+
+| Requirement | Description |
+|-------------|-------------|
+| **Clean composition** | Simple layouts, no visual clutter |
+| **White space** | Generous margins, breathing room around elements |
+| **No complex backgrounds** | Solid colors or subtle gradients only, avoid busy textures |
+| **Centered or content-appropriate** | Main visual elements centered or positioned by content needs |
+| **Matching graphics** | Use graphic elements that align with content theme |
+| **Highlight core info** | White space draws attention to key information |
+
+**Add to ALL prompts**:
+> Clean composition with generous white space. Simple or no background. Main elements centered or positioned by content needs.
+
+---
+
+## Color Specification Rules
+
+Colors in prompts use hex codes for **rendering guidance only** — they tell the model which colors to use, NOT what text to display.
+
+**⚠️ CRITICAL**: Image generation models sometimes render color names and hex values as visible text labels in the image (e.g., painting "Macaron Blue #A8D8EA" as a label). This must be prevented.
+
+**Add to ALL prompts that contain a COLORS section**:
+> Color values (#hex) and color names are rendering guidance only — do NOT display color names, hex codes, or palette labels as visible text in the image.
+
+---
+
+## Character Rendering
+
+When depicting people:
+
+| Guideline | Description |
+|-----------|-------------|
+| **Style** | Simplified cartoon silhouettes or symbolic expressions |
+| **Avoid** | Realistic human portrayals, detailed faces |
+| **Diversity** | Varied body types when showing multiple people |
+| **Emotion** | Express through posture and simple gestures |
+
+**Add to ALL prompts with human figures**:
+> Human figures: simplified stylized silhouettes or symbolic representations, not photorealistic.
+
+---
+
+## Text in Illustrations
+
+| Element | Guideline |
+|---------|-----------|
+| **Size** | Large, prominent, immediately readable |
+| **Style** | Handwritten fonts preferred for warmth |
+| **Content** | Concise keywords and core concepts only |
+| **Language** | Match article language |
+
+**Add to prompts with text**:
+> Text should be large and prominent with handwritten-style fonts. Keep minimal, focus on keywords.
+
+---
+
+## Principles
+
+Good prompts must include:
+
+1. **Layout Structure First**: Describe composition, zones, flow direction
+2. **Specific Data/Labels**: Use actual numbers, terms from article
+3. **Visual Relationships**: How elements connect
+4. **Semantic Colors**: Meaning-based color choices (red=warning, green=efficient)
+5. **Style Characteristics**: Line treatment, texture, mood
+6. **Aspect Ratio**: End with ratio and complexity level
+
+## Type-Specific Templates
+
+### Infographic
+
+```
+[Title] - Data Visualization
+
+Layout: [grid/radial/hierarchical]
+
+ZONES:
+- Zone 1: [data point with specific values]
+- Zone 2: [comparison with metrics]
+- Zone 3: [summary/conclusion]
+
+LABELS: [specific numbers, percentages, terms from article]
+COLORS: [semantic color mapping]
+STYLE: [style characteristics]
+ASPECT: 16:9
+```
+
+**Infographic + vector-illustration**:
+```
+Flat vector illustration infographic. Clean black outlines on all elements.
+COLORS: Cream background (#F5F0E6), Coral Red (#E07A5F), Mint Green (#81B29A), Mustard Yellow (#F2CC8F)
+ELEMENTS: Geometric simplified icons, no gradients, playful decorative elements (dots, stars)
+```
+
+**Infographic + vector-illustration + warm palette**:
+```
+Flat vector illustration infographic. Clean black outlines on all elements.
+PALETTE OVERRIDE (warm): Warm-only color palette, no cool colors.
+COLORS: Soft Peach background (#FFECD2), Warm Orange (#ED8936),
+ Terracotta (#C05621), Golden Yellow (#F6AD55), Deep Brown (#744210)
+ELEMENTS: Geometric simplified icons, no gradients, rounded corners,
+ modular card layout, consistent icon style
+```
+
+### Scene
+
+```
+[Title] - Atmospheric Scene
+
+FOCAL POINT: [main subject]
+ATMOSPHERE: [lighting, mood, environment]
+MOOD: [emotion to convey]
+COLOR TEMPERATURE: [warm/cool/neutral]
+STYLE: [style characteristics]
+ASPECT: 16:9
+```
+
+### Flowchart
+
+```
+[Title] - Process Flow
+
+Layout: [left-right/top-down/circular]
+
+STEPS:
+1. [Step name] - [brief description]
+2. [Step name] - [brief description]
+...
+
+CONNECTIONS: [arrow types, decision points]
+STYLE: [style characteristics]
+ASPECT: 16:9
+```
+
+**Flowchart + vector-illustration**:
+```
+Flat vector flowchart with bold arrows and geometric step containers.
+COLORS: Cream background (#F5F0E6), steps in Coral/Mint/Mustard, black outlines
+ELEMENTS: Rounded rectangles, thick arrows, simple icons per step
+```
+
+**Flowchart + sketch-notes + macaron palette**:
+```
+Hand-drawn educational flowchart on warm cream paper. Slight wobble on all lines.
+PALETTE: macaron — soft pastel color blocks
+COLORS: Warm Cream background (#F5F0E8), zone fills in Macaron Blue (#A8D8EA),
+ Lavender (#D5C6E0), Mint (#B5E5CF), Coral Red (#E8655A) for emphasis
+ELEMENTS: Rounded cards with dashed/solid borders, wavy hand-drawn arrows with labels,
+ simple stick-figure characters, doodle decorations (stars, underlines)
+STYLE: Color fills don't completely fill outlines, hand-drawn lettering, generous white space
+```
+
+**Flowchart + ink-notes + mono-ink palette**:
+```
+Professional hand-drawn visual-note flowchart on pure white. Black ink line work
+with slight wobble, à la Mike Rohde sketchnoting.
+PALETTE: mono-ink — black ink dominant, sparse semantic accents
+COLORS: Pure White background (#FFFFFF), Near Black (#1A1A1A) for all lines,
+ text, and figures; Coral Red (#E8655A) only for risk/emphasis,
+ Muted Teal (#5FA8A8) only for positive/solution states
+ELEMENTS: Left-to-right stage boxes with rounded-rect frames, wavy hand-drawn
+ arrows between stages, simple stick-figure characters with role
+ labels above (e.g., "ML Engineer", "Team Lead"), dashed-border box
+ for future/empty stage, small doodle icons per stage
+STYLE: Hand-lettered titles (bold, oversized), handwritten stage labels and
+ annotations, generous white space, bottom tagline summarizing takeaway
+```
+
+### Comparison
+
+```
+[Title] - Comparison View
+
+LEFT SIDE - [Option A]:
+- [Point 1]
+- [Point 2]
+
+RIGHT SIDE - [Option B]:
+- [Point 1]
+- [Point 2]
+
+DIVIDER: [visual separator]
+STYLE: [style characteristics]
+ASPECT: 16:9
+```
+
+**Comparison + vector-illustration**:
+```
+Flat vector comparison with split layout. Clear visual separation.
+COLORS: Left side Coral (#E07A5F), Right side Mint (#81B29A), cream background
+ELEMENTS: Bold icons, black outlines, centered divider line
+```
+
+**Comparison + vector-illustration + warm palette**:
+```
+Flat vector comparison with split layout. Clear visual separation.
+PALETTE OVERRIDE (warm): Warm-only color palette, no cool colors.
+COLORS: Left side Warm Orange (#ED8936), Right side Terracotta (#C05621),
+ Soft Peach background (#FFECD2), Deep Brown (#744210) accents
+ELEMENTS: Bold icons, black outlines, centered divider line
+```
+
+**Comparison + ink-notes + mono-ink palette** (Before/After, Traditional vs New):
+```
+Professional hand-drawn sketchnote comparison on pure white. Black ink line work
+with slight wobble, à la Mike Rohde sketchnoting.
+PALETTE: mono-ink — black ink dominant, sparse semantic accents
+COLORS: Pure White background (#FFFFFF), Near Black (#1A1A1A) for all outlines,
+ text, figures, arrows; Coral Red (#E8655A) reserved for risks/gaps
+ (left/Before side); Muted Teal (#5FA8A8) reserved for positives
+ (right/After side). Color accents under 10% of canvas.
+LAYOUT: Left | Right split with vertical hand-drawn divider. Hand-lettered
+ "Before" label (top-left) and "After" label (top-right).
+LEFT SIDE: Stick figure(s) with role label above, speech bubble showing the
+ pain point, bulleted pain-point list in handwritten text.
+RIGHT SIDE: Stick figure(s) showing the new state, bulleted improvement list,
+ small positive-action icons.
+BRIDGE: Curved hand-drawn "mindset shift" arrow bridging left → right with
+ small inline label describing the shift.
+BOTTOM: Single-line hand-lettered tagline summarizing the takeaway.
+STYLE: Hand-lettered headings (bold, oversized), handwritten body annotations,
+ generous white space, no computer fonts, no gradients, no shadows.
+```
+
+### Framework
+
+```
+[Title] - Conceptual Framework
+
+STRUCTURE: [hierarchical/network/matrix]
+
+NODES:
+- [Concept 1] - [role]
+- [Concept 2] - [role]
+
+RELATIONSHIPS: [how nodes connect]
+STYLE: [style characteristics]
+ASPECT: 16:9
+```
+
+**Framework + vector-illustration**:
+```
+Flat vector framework diagram with geometric nodes and bold connectors.
+COLORS: Cream background (#F5F0E6), nodes in Coral/Mint/Mustard/Blue, black outlines
+ELEMENTS: Rounded rectangles or circles for nodes, thick connecting lines
+```
+
+**Framework + vector-illustration + warm palette**:
+```
+Flat vector framework diagram with geometric nodes and bold connectors.
+PALETTE OVERRIDE (warm): Warm-only color palette, no cool colors.
+COLORS: Soft Peach background (#FFECD2), nodes in Warm Orange (#ED8936),
+ Terracotta (#C05621), Golden Yellow (#F6AD55), black outlines
+ELEMENTS: Rounded rectangles or circles for nodes, thick connecting lines
+```
+
+**Framework + ink-notes + mono-ink palette** (command center, OS analogy):
+```
+Professional hand-drawn sketchnote framework on pure white. Black ink line work
+with slight wobble, à la Mike Rohde sketchnoting.
+PALETTE: mono-ink — black ink dominant, sparse semantic accents
+COLORS: Pure White background (#FFFFFF), Near Black (#1A1A1A) for all lines,
+ text, figures; Dusty Lavender (#9B8AB5) for neutral category tags only;
+ Coral Red (#E8655A) for emphasis sparingly. Color accents under 10%.
+STRUCTURE: Central rounded-rectangle frame as "the system" with hand-lettered
+ title inside. Inner layer of labeled sub-components (node labels
+ above each). Outer layer of feeder arrows from stick-figure
+ operators/users with role labels.
+ELEMENTS: Stick figures at the edges with role tags ("Team Lead", "Operator"),
+ wavy hand-drawn connector arrows with small inline labels, small
+ doodle icons per component, dashed-border placeholder(s) for
+ future/empty capabilities.
+BOTTOM: Single-line hand-lettered tagline.
+STYLE: Hand-lettered headings, handwritten annotations, generous white space,
+ no computer fonts, no gradients.
+```
+
+### Timeline
+
+```
+[Title] - Chronological View
+
+DIRECTION: [horizontal/vertical]
+
+EVENTS:
+- [Date/Period 1]: [milestone]
+- [Date/Period 2]: [milestone]
+
+MARKERS: [visual indicators]
+STYLE: [style characteristics]
+ASPECT: 16:9
+```
+
+### Screen-Print Style Override
+
+When `style: screen-print`, replace standard style instructions with:
+
+```
+Screen print / silkscreen poster art. Flat color blocks, NO gradients.
+COLORS: 2-5 colors maximum. [Choose from style palette or duotone pair]
+TEXTURE: Halftone dot patterns, slight color layer misregistration, paper grain
+COMPOSITION: Bold silhouettes, geometric framing, negative space as storytelling element
+FIGURES: Silhouettes only, no detailed faces, stencil-cut edges
+TYPOGRAPHY: Bold condensed sans-serif integrated into composition (not overlaid)
+```
+
+**Scene + screen-print**:
+```
+Conceptual poster scene. Single symbolic focal point, NOT literal illustration.
+COLORS: Duotone pair (e.g., Burnt Orange #E8751A + Deep Teal #0A6E6E) on Off-Black #121212
+COMPOSITION: Centered silhouette or geometric frame, 60%+ negative space
+TEXTURE: Halftone dots, paper grain, slight print misregistration
+```
+
+**Comparison + screen-print**:
+```
+Split poster composition. Each side dominated by one color from duotone pair.
+LEFT: [Color A] side with silhouette/icon for [Option A]
+RIGHT: [Color B] side with silhouette/icon for [Option B]
+DIVIDER: Geometric shape or negative space boundary
+TEXTURE: Halftone transitions between sides
+```
+
+---
+
+## Palette Override
+
+When a palette is specified (via `--palette` or preset), it overrides the style's default colors:
+
+1. Read style file → get rendering rules (Visual Elements, Style Rules, line treatment)
+2. Read palette file (`palettes/.md`) → get Colors + Background
+3. Palette Colors **replace** style's default Color Palette in prompt
+4. Palette Background **replaces** style's Background color (keep style's texture description)
+5. Build prompt: style rendering instructions + palette colors
+
+**Prompt frontmatter** includes palette when specified:
+```yaml
+---
+illustration_id: 01
+type: infographic
+style: vector-illustration
+palette: macaron
+---
+```
+
+**Example**: `vector-illustration` + `macaron` palette:
+```
+Flat vector illustration infographic. Clean black outlines on all elements.
+PALETTE: macaron — soft pastel color blocks
+COLORS: Warm Cream background (#F5F0E8), Macaron Blue (#A8D8EA), Mint (#B5E5CF),
+ Lavender (#D5C6E0), Peach (#FFD5C2), Coral Red (#E8655A) for emphasis
+ELEMENTS: Geometric simplified icons, no gradients, playful decorative elements
+```
+
+When no palette is specified, use the style's built-in Color Palette as before.
+
+---
+
+## What to Avoid
+
+- Vague descriptions ("a nice image")
+- Literal metaphor illustrations
+- Missing concrete labels/annotations
+- Generic decorative elements
+
+## Watermark Integration (optional)
+
+If the user asks for a watermark, append:
+
+```
+Include a subtle watermark "[content]" positioned at [position].
+```
diff --git a/src/skills/bundled/baoyu-article-illustrator/references/style-presets.md b/src/skills/bundled/baoyu-article-illustrator/references/style-presets.md
new file mode 100644
index 0000000..5e0777f
--- /dev/null
+++ b/src/skills/bundled/baoyu-article-illustrator/references/style-presets.md
@@ -0,0 +1,80 @@
+# Style Presets
+
+A preset expands to a type + style + optional palette combination. Users can override any dimension in their request.
+
+## By Category
+
+### Technical & Engineering
+
+| Preset | Type | Style | Palette | Best For |
+|----------|------|-------|---------|----------|
+| `tech-explainer` | `infographic` | `blueprint` | — | API docs, system metrics, technical deep-dives |
+| `system-design` | `framework` | `blueprint` | — | Architecture diagrams, system design |
+| `architecture` | `framework` | `vector-illustration` | — | Component relationships, module structure |
+| `science-paper` | `infographic` | `scientific` | — | Research findings, lab results, academic |
+
+### Knowledge & Education
+
+| Preset | Type | Style | Palette | Best For |
+|----------|------|-------|---------|----------|
+| `knowledge-base` | `infographic` | `vector-illustration` | — | Concept explainers, tutorials, how-to |
+| `saas-guide` | `infographic` | `notion` | — | Product guides, SaaS docs, tool walkthroughs |
+| `tutorial` | `flowchart` | `vector-illustration` | — | Step-by-step tutorials, setup guides |
+| `process-flow` | `flowchart` | `notion` | — | Workflow documentation, onboarding flows |
+| `warm-knowledge` | `infographic` | `vector-illustration` | `warm` | Product showcases, team intros, feature cards, brand content |
+| `edu-visual` | `infographic` | `vector-illustration` | `macaron` | Knowledge summaries, concept explainers, educational articles |
+| `hand-drawn-edu` | `flowchart` | `sketch-notes` | `macaron` | Hand-drawn educational diagrams, process explainers, onboarding visuals |
+| `ink-notes-compare` | `comparison` | `ink-notes` | `mono-ink` | Before/After essays, Traditional vs New, OS-style comparisons, mindset-shift narratives |
+| `ink-notes-flow` | `flowchart` | `ink-notes` | `mono-ink` | Professional process explainers, workforce pipelines, hand-drawn technical walkthroughs |
+| `ink-notes-framework` | `framework` | `ink-notes` | `mono-ink` | System analogies, command-center diagrams, architecture-as-metaphor, tech manifestos |
+
+### Data & Analysis
+
+| Preset | Type | Style | Palette | Best For |
+|----------|------|-------|---------|----------|
+| `data-report` | `infographic` | `editorial` | — | Data journalism, metrics reports, dashboards |
+| `versus` | `comparison` | `vector-illustration` | — | Tech comparisons, framework shootouts |
+| `business-compare` | `comparison` | `elegant` | — | Product evaluations, strategy options |
+
+### Narrative & Creative
+
+| Preset | Type | Style | Palette | Best For |
+|----------|------|-------|---------|----------|
+| `storytelling` | `scene` | `warm` | — | Personal essays, reflections, growth stories |
+| `lifestyle` | `scene` | `watercolor` | — | Travel, wellness, lifestyle, creative |
+| `history` | `timeline` | `elegant` | — | Historical overviews, milestones |
+| `evolution` | `timeline` | `warm` | — | Progress narratives, growth journeys |
+
+### Editorial & Opinion
+
+| Preset | Type | Style | Palette | Best For |
+|----------|------|-------|---------|----------|
+| `opinion-piece` | `scene` | `screen-print` | — | Op-eds, commentary, critical essays |
+| `editorial-poster` | `comparison` | `screen-print` | — | Debate, contrasting viewpoints |
+| `cinematic` | `scene` | `screen-print` | — | Dramatic narratives, cultural essays |
+
+## Content Type → Preset Recommendations
+
+Use this table during Step 3 to recommend presets based on Step 2 content analysis:
+
+| Content Type (Step 2) | Primary Preset | Alternatives |
+|------------------------|----------------|--------------|
+| Technical | `tech-explainer` | `system-design`, `architecture` |
+| Tutorial | `tutorial` | `process-flow`, `knowledge-base`, `edu-visual` |
+| Methodology / Framework | `system-design` | `architecture`, `process-flow` |
+| Data / Metrics | `data-report` | `versus`, `tech-explainer` |
+| Comparison / Review | `versus` | `business-compare`, `editorial-poster`, `ink-notes-compare` |
+| Manifesto / Mindset shift / Professional visual note | `ink-notes-compare` | `ink-notes-framework`, `ink-notes-flow` |
+| Narrative / Personal | `storytelling` | `lifestyle`, `evolution` |
+| Opinion / Editorial | `opinion-piece` | `cinematic`, `editorial-poster` |
+| Historical / Timeline | `history` | `evolution` |
+| Academic / Research | `science-paper` | `tech-explainer`, `data-report` |
+| SaaS / Product | `saas-guide` | `knowledge-base`, `process-flow`, `warm-knowledge` |
+| Education / Knowledge | `edu-visual` | `knowledge-base`, `tutorial`, `hand-drawn-edu` |
+
+## Override Examples
+
+- "use the tech-explainer preset but swap the style for notion" = infographic type with notion style
+- "storytelling preset with timeline type" = timeline type with warm style
+
+Explicit type/style/palette mentions in the user's request always override preset values.
diff --git a/src/skills/bundled/baoyu-article-illustrator/references/styles.md b/src/skills/bundled/baoyu-article-illustrator/references/styles.md
new file mode 100644
index 0000000..75631e9
--- /dev/null
+++ b/src/skills/bundled/baoyu-article-illustrator/references/styles.md
@@ -0,0 +1,224 @@
+# Style Reference
+
+## Core Styles
+
+Simplified style tier for quick selection:
+
+| Core Style | Maps To | Best For |
+|------------|---------|----------|
+| `vector` | vector-illustration | Knowledge articles, tutorials, tech content |
+| `minimal-flat` | notion | General, knowledge sharing, SaaS |
+| `sci-fi` | blueprint | AI, frontier tech, system design |
+| `hand-drawn` | sketch/warm | Relaxed, reflective, casual content |
+| `editorial` | editorial | Processes, data, journalism |
+| `scene` | warm/watercolor | Narratives, emotional, lifestyle |
+| `poster` | screen-print | Opinion, editorial, cultural, cinematic |
+
+Use Core Styles for most cases. See full Style Gallery below for granular control.
+
+---
+
+## Style Gallery
+
+| Style | Description | Best For |
+|-------|-------------|----------|
+| `vector-illustration` | Clean flat vector art with bold shapes | Knowledge articles, tutorials, tech content |
+| `notion` | Minimalist hand-drawn line art | Knowledge sharing, SaaS, productivity |
+| `elegant` | Refined, sophisticated | Business, thought leadership |
+| `warm` | Friendly, approachable | Personal growth, lifestyle, education |
+| `minimal` | Ultra-clean, zen-like | Philosophy, minimalism, core concepts |
+| `blueprint` | Technical schematics | Architecture, system design, engineering |
+| `watercolor` | Soft artistic with natural warmth | Lifestyle, travel, creative |
+| `editorial` | Magazine-style infographic | Tech explainers, journalism |
+| `scientific` | Academic precise diagrams | Biology, chemistry, technical research |
+| `chalkboard` | Classroom chalk drawing style | Education, teaching, explanations |
+| `fantasy-animation` | Ghibli/Disney-inspired hand-drawn | Storybook, magical, emotional |
+| `flat` | Modern bold geometric shapes | Modern digital, contemporary |
+| `flat-doodle` | Cute flat with bold outlines | Cute, friendly, approachable |
+| `intuition-machine` | Technical briefing with aged paper | Technical briefings, academic |
+| `nature` | Organic earthy illustration | Environmental, wellness |
+| `pixel-art` | Retro 8-bit gaming aesthetic | Gaming, retro tech |
+| `playful` | Whimsical pastel doodles | Fun, casual, educational |
+| `retro` | 80s/90s neon geometric | 80s/90s nostalgic, bold |
+| `sketch` | Raw pencil notebook style | Brainstorming, creative exploration |
+| `screen-print` | Bold poster art, halftone textures, limited colors | Opinion, editorial, cultural, cinematic |
+| `sketch-notes` | Soft hand-drawn warm notes | Educational, warm notes |
+| `ink-notes` | Black ink on pure white, sparse semantic accents, hand-lettered (à la Mike Rohde's sketchnoting) | Before/After essays, tech manifestos, framework analogies |
+| `vintage` | Aged parchment historical | Historical, heritage |
+
+Full specifications: `references/styles/
+
+
+