Improve built-in theme pipeline/format

.agents/skills/theme-definition/SKILL.md

@@ -0,0 +1,109 @@
+---
+name: theme-definition
+description: Create or update a bundled YAML theme source in the repository `themes/` directory when asked for a specific color theme such as Gruvbox, Nord, or a custom dark or light palette. These YAML files are compiled into TextMate `.tmTheme` files during the build.
+---
+
+# Theme Definition
+
+Use this skill when asked to create or update a bundled theme definition for a named color theme or palette.
+
+## Goal
+
+Create or update exactly one bundled theme source file in the repository top-level `themes/` directory:
+
+- `themes/<theme-name>.yml`
+
+Examples:
+
+- `Gruvbox Dark` -> `themes/gruvbox_dark.yml`
+- `Nord Light` -> `themes/nord_light.yml`
+- `My Theme` -> `themes/my_theme.yml`
+
+If a matching file already exists, update it instead of creating a duplicate.
+
+The YAML source is the authored artifact. Amp's build compiles it into a generated `.tmTheme` file with the same key during `build.rs`.
+
+## Required Constraints
+
+- Only author bundled YAML theme sources in `themes/`.
+- Do not manually create or edit generated `.tmTheme` files for bundled themes.
+- Do not write the theme anywhere except the top-level `themes/` directory unless the user explicitly asks for something else.
+- Use the semantic token-family standard in [references/token-color-standard.md](references/token-color-standard.md) whenever the user does not provide a custom scope map.
+- Treat `palette` as the single source of truth for authored colors:
+  - define hex literals in `palette`
+  - reference palette keys from `settings` and `rules`
+  - do not place inline hex literals directly in `settings` or `rules`
+- Match the actual compiler schema in `build/theme_compiler/`:
+  - required top-level keys: `name`, `settings`, `rules`
+  - optional top-level key: `palette`
+  - `palette` values must be hex colors
+  - required `settings` keys: `foreground`, `background`, `line_highlight`
+  - `settings.foreground`, `settings.background`, and `settings.line_highlight` must reference palette keys
+  - `rules` must not be empty
+  - each rule may contain `name`, `scope`, `foreground`, `background`, `font_style`
+  - `rules[*].foreground` and `rules[*].background`, when present, must reference palette keys
+  - each rule must define at least one of `foreground`, `background`, or `font_style`
+  - `scope` must be a single valid TextMate selector string
+  - `font_style` values are `bold`, `italic`, and `underline`
+- Treat unknown YAML keys as invalid rather than inventing extra structure.
+
+Amp requires these base theme settings:
+
+- `settings.foreground`
+- `settings.background`
+- `settings.line_highlight`
+
+When the user does not provide a complete scope list, do not stop at a small "practical first pass". Map the requested palette onto the canonical token families in the reference file so common code tokens do not collapse to the default foreground.
+
+## Workflow
+
+1. Identify the requested theme name and whether it is dark, light, or otherwise palette-driven.
+2. Inspect nearby files in `themes/` for filename and YAML rule-shape conventions.
+3. Read [references/token-color-standard.md](references/token-color-standard.md) and choose colors for each required family.
+4. Create or update a YAML theme source with:
+   - top-level `name`
+   - a `palette` containing the authored hex colors
+   - `settings.foreground`, `settings.background`, and `settings.line_highlight` as palette-key references
+   - `rules` covering the canonical token families in the reference, using the requested palette
+5. Prefer broadly useful TextMate scopes and stable fallback tiers instead of copying an existing bundled theme's exact rule list.
+6. Validate through the repository build path before finishing.
+
+## Output Guidelines
+
+### For Generated Or Updated Themes
+
+- Use a filename stem that matches the theme key Amp will expose in theme selection.
+- Keep the YAML concise and readable rather than encoding generated `.tmTheme` structure by hand.
+- Put literal hex values in `palette`, not directly in `settings` or per-rule color fields.
+- If the requested palette is underspecified, make the smallest reasonable set of assumptions and state them briefly.
+- Prefer semantically complete coverage over an artificially tiny rule set.
+- Ensure the base settings are sufficient for Amp's UI color mapping:
+  - `settings.foreground` is used for default text
+  - `settings.background` is used for inverted background mappings
+  - `settings.line_highlight` is used for the focused or current-line background
+- Use the default foreground as a last-resort fallback, not as the intended color for common code tokens.
+- Keep at least two punctuation tiers when the palette allows it:
+  - structural punctuation can stay muted
+  - semantic operators should remain clearly visible
+- Distinguish the following families whenever the palette allows it:
+  - comments
+  - strings
+  - numbers and constants
+  - keywords and storage
+  - functions and methods
+  - types and namespaces
+  - local variables
+  - parameters
+  - support or builtin symbols
+  - annotations or attributes
+  - semantic operators
+  - structural punctuation
+- Rust is a useful stress case for validation, but the output must remain cross-language and useful for markup and configuration formats too.
+
+### For Validation
+
+- Prefer `cargo check` as the main smoke test. It runs `build.rs`, compiles `themes/*.yml`, and loads the generated bundled themes.
+- Use `cargo test --test theme_compiler` when validating compiler-schema behavior or debugging theme-source failures.
+- Treat inline color literals outside `palette` as source-format errors.
+- If there is no dedicated theme test for a specific change, still perform at least a build-path smoke check before finishing.
+- When updating a bundled example theme, prefer adding or running a test that confirms the compiled theme loads and that key token families are represented by explicit scope rules.

.agents/skills/theme-definition/references/token-color-standard.md

@@ -0,0 +1,95 @@
+# Token Color Standard
+
+Use this reference when generating or updating a bundled YAML theme source in `themes/` and the user has not provided a detailed scope map. The authored file is YAML, but the rule scopes should still target stable TextMate token families because Amp compiles the source into a `.tmTheme` during the build.
+
+For authored theme sources in this repo, `palette` is the only place for literal hex colors. `settings` and `rules` must reference palette keys so color intent stays named and reviewable.
+
+## Goal
+
+Map a palette onto stable semantic token families so common syntax elements stay differentiated across languages. Rust is the stress case: if a verbose Rust file still looks mostly like plain foreground text, coverage is too weak.
+
+## Core Rules
+
+- Start with broad TextMate scope families, not language-specific one-offs.
+- Use the base foreground only for truly unclassified text and low-signal fallback.
+- Prefer distinct hues for major semantic families over minimalist sameness.
+- Keep structural punctuation readable but quieter than semantic operators.
+- Parameters must not share the plain-text color when local variables are also colored.
+- Support or builtin scopes should usually differ from user-defined names.
+
+## Canonical Families
+
+Every generated theme should include explicit selectors for these families unless the palette is intentionally constrained.
+
+| Family | Purpose | Typical selectors |
+| --- | --- | --- |
+| Plain text | Unclassified fallback text | `variable`, `source`, `text` fallback only |
+| Comments | Comments and docs | `comment`, `comment.block.documentation`, `punctuation.definition.comment` |
+| Strings | String bodies | `string`, `string.quoted`, `string.unquoted` |
+| String escapes | Escape sequences and regex escapes | `constant.character.escape`, `constant.other.escape`, `string.regexp` |
+| Numbers | Numeric literals | `constant.numeric` |
+| Constants | Language and user constants | `constant.language`, `constant.character`, `constant.other`, `support.constant` |
+| Keywords | Control flow and declarations | `keyword`, `keyword.control`, `keyword.declaration`, `keyword.other` |
+| Storage/modifiers | Types, modifiers, mutability, ownership-like markers | `storage`, `storage.type`, `storage.modifier` |
+| Functions and methods | Declared and invoked callables | `entity.name.function`, `meta.function-call`, `support.function`, `variable.function` |
+| Types | Structs, enums, classes, traits, primitive types | `entity.name.type`, `entity.name.class`, `entity.name.struct`, `entity.name.enum`, `entity.name.trait`, `support.type` |
+| Namespaces/modules | Paths, modules, packages | `entity.name.namespace`, `entity.name.module`, `support.module` |
+| Macros/preprocessor | Macro-like or generated forms | `entity.name.macro`, `support.macro`, `meta.preprocessor` |
+| Local variables | Normal bindings and fields when exposed as variables | `variable.other`, `variable.object`, `variable.other.member` |
+| Parameters | Function and closure parameters | `variable.parameter` |
+| Language/self variables | `self`, `this`, shell vars, special bindings | `variable.language`, `support.variable` |
+| Attributes/annotations | Rust attributes, decorators, annotation names | `entity.other.attribute-name`, `storage.annotation`, `punctuation.definition.annotation` |
+| Semantic operators | Accessors, assignment, ranges, arrows, logical ops | `keyword.operator`, `keyword.operator.assignment`, `keyword.operator.accessor`, `keyword.operator.range` |
+| Structural punctuation | Braces, commas, delimiters, separators | `punctuation.separator`, `punctuation.terminator`, `meta.brace`, `meta.delimiter`, `punctuation.section` |
+| Markup/config extras | Tags, headings, emphasis, diffs | `entity.name.tag`, `markup.heading`, `markup.bold`, `markup.italic`, `markup.inserted`, `markup.deleted` |
+| Invalid/deprecated | Errors and deprecated syntax | `invalid`, `invalid.deprecated` |
+
+## Fallback Order
+
+Use this fallback order when a palette is underspecified:
+
+1. Comments
+2. Strings
+3. Numbers/constants
+4. Keywords/storage
+5. Functions
+6. Types/namespaces
+7. Variables/parameters
+8. Support or builtin symbols
+9. Attributes/annotations
+10. Operators
+11. Structural punctuation
+12. Plain text fallback
+
+Do not skip from a specialized family straight to plain text if a nearby family already expresses similar semantics. For example:
+
+- `variable.parameter` should fall back to the variable family, not the base foreground.
+- `support.function` should fall back to function or support coloring, not plain text.
+- `entity.name.namespace` should fall back to types or support, not plain text.
+
+## Palette Mapping Guidance
+
+- Comments: muted and lower-contrast than code.
+- Strings: warm or otherwise clearly distinct from comments and keywords.
+- Numbers/constants: usually share a family, but language constants may be slightly stronger.
+- Keywords/storage: strong and consistent; these anchor the syntax.
+- Functions: distinct from types and variables.
+- Types/namespaces: related but not identical when the palette has enough room.
+- Local variables: subtle but still distinguishable from plain text.
+- Parameters: warmer or otherwise more specific than local variables.
+- Support/builtins: separate from user-defined names when possible.
+- Attributes/annotations: visible but not louder than keywords.
+- Semantic operators: visible enough to show expression structure.
+- Structural punctuation: muted but never invisible.
+
+## Validation Checklist
+
+Before finishing, confirm:
+
+- The theme parses through Syntect.
+- The theme contains explicit selectors for parameters, support or builtins, annotations or attributes, operators, and structural punctuation.
+- A Rust-heavy file such as `build.rs` would show visible differences between:
+  - function names and type names
+  - parameters and local variables
+  - namespace paths and plain text
+  - semantic operators and structural delimiters

AGENTS.md

@@ -3,7 +3,7 @@
 ## Project Structure & Module Organization
 - `src/` contains the Rust source: `main.rs` bootstraps the app, `lib.rs` exposes core logic.
 - Key modules live under `src/commands/`, `src/models/`, `src/view/`, `src/input/`, `src/presenters/`, and `src/util/`.
-- Themes and defaults are stored under `src/themes/` and `src/models/application/preferences/`.
+- Bundled theme sources and defaults are stored under `themes/` and `src/models/application/preferences/`.
 - Documentation sources live in `documentation/` (Zensical/MkDocs content).
 - Benchmarks live in `benches/`.
 - For a deeper walkthrough of module roles, the event loop, and command/keymap mechanics, see `docs/architecture.md`.

Cargo.toml

@@ -11,9 +11,12 @@ readme = "README.md"
 license-file = "LICENSE"
 keywords = ["text", "editor", "terminal", "modal"]
 edition="2021"
+build = "build/main.rs"
 
 [build-dependencies]
 regex = "1.10"
+serde = { version = "1.0", features = ["derive"] }
+serde_yaml = "0.9"
 syntect = "5.1"
 
 [dependencies]
@@ -50,6 +53,8 @@ default-features = false # removes unused openssl dependency
 
 [dev-dependencies]
 criterion = "0.5"
+serde = { version = "1.0", features = ["derive"] }
+serde_yaml = "0.9"
 
 [[bench]]
 name = "draw_buffer"

build.rs → build/main.rs

@@ -1,3 +1,5 @@
+mod theme_compiler;
+
 use regex::Regex;
 use std::env;
 use std::fs::{self, read_to_string, File};
@@ -6,15 +8,19 @@ use std::path::{Path, PathBuf};
 use std::process::Command;
 use std::result::Result;
 use syntect::dumps::dump_to_uncompressed_file;
+use syntect::highlighting::ThemeSet;
 use syntect::parsing::SyntaxSet;
 
 const COMMAND_REGEX: &str = r"pub fn (.*)\(app: &mut Application\) -> Result";
 const APP_SYNTAX_DIR: &str = "syntaxes";
 const APP_SYNTAX_SOURCE: &str = "app_syntaxes.packdump";
+const APP_THEME_DIR: &str = "themes";
+const APP_THEME_SOURCE: &str = "app_themes.packdump";
 
 fn main() {
     generate_commands();
     bake_app_syntaxes();
+    bake_app_themes();
     set_build_revision();
 }
 
@@ -129,3 +135,15 @@ fn bake_app_syntaxes() {
     dump_to_uncompressed_file(&builder.build(), output_path)
         .expect("Failed to write bundled syntax dump");
 }
+
+fn bake_app_themes() {
+    let out_dir = env::var("OUT_DIR").expect("The compiler did not provide $OUT_DIR");
+    let output_path = PathBuf::from(out_dir).join(APP_THEME_SOURCE);
+    let compiled_theme_dir = PathBuf::from(env::var("OUT_DIR").unwrap()).join("compiled_themes");
+    theme_compiler::compile_themes(Path::new(APP_THEME_DIR), &compiled_theme_dir)
+        .expect("Failed to compile bundled theme sources");
+    let theme_set = ThemeSet::load_from_folder(&compiled_theme_dir)
+        .expect("Failed to load compiled bundled themes");
+
+    dump_to_uncompressed_file(&theme_set, output_path).expect("Failed to write bundled theme dump");
+}

build/theme_compiler/mod.rs

@@ -0,0 +1,67 @@
+mod parsed;
+mod textmate;
+mod validated;
+
+use std::collections::HashSet;
+use std::fs;
+use std::path::{Path, PathBuf};
+
+pub use validated::Theme;
+
+pub fn compile_themes(source_dir: &Path, output_dir: &Path) -> Result<Vec<PathBuf>, String> {
+    if output_dir.exists() {
+        fs::remove_dir_all(output_dir)
+            .map_err(|error| format!("Failed to clear generated theme directory: {error}"))?;
+    }
+
+    fs::create_dir_all(output_dir)
+        .map_err(|error| format!("Failed to create generated theme directory: {error}"))?;
+
+    let mut themes = Vec::new();
+    let mut seen_keys = HashSet::new();
+    for entry in fs::read_dir(source_dir)
+        .map_err(|error| format!("Failed to read theme source directory: {error}"))?
+    {
+        let path = entry
+            .map_err(|error| format!("Failed to read theme source entry: {error}"))?
+            .path();
+        if !path.is_file() || path.extension().and_then(|ext| ext.to_str()) != Some("yml") {
+            continue; // skip non-yaml files
+        }
+
+        let key = path
+            .file_stem()
+            .and_then(|stem| stem.to_str())
+            .ok_or_else(|| format!("Invalid theme source filename: {}", path.display()))?
+            .to_string();
+
+        if !seen_keys.insert(key.clone()) {
+            return Err(format!("Duplicate theme key: {key}"));
+        }
+
+        let content = fs::read_to_string(&path)
+            .map_err(|error| format!("Failed to read {}: {error}", path.display()))?;
+        let theme = parse_theme(&content)
+            .map_err(|error| format!("Failed to compile {key}.yml: {error}"))?;
+        themes.push((key.clone(), theme));
+    }
+
+    let mut outputs = Vec::new();
+    for (key, theme) in themes {
+        let output_path = output_dir.join(format!("{key}.tmTheme"));
+        fs::write(&output_path, render_tmtheme(&theme))
+            .map_err(|error| format!("Failed to write {}: {error}", output_path.display()))?;
+        outputs.push(output_path);
+    }
+
+    Ok(outputs)
+}
+
+pub fn parse_theme(content: &str) -> Result<Theme, String> {
+    let parsed_theme = parsed::parse(content)?;
+    validated::Theme::try_from_parsed(parsed_theme)
+}
+
+pub fn render_tmtheme(theme: &Theme) -> String {
+    textmate::render(theme)
+}