diff --git a/skills/spec/SKILL.md b/skills/spec/SKILL.md index 59da171..964d0b1 100644 --- a/skills/spec/SKILL.md +++ b/skills/spec/SKILL.md @@ -3,7 +3,7 @@ description: 'Specification-driven development. Triggers on: "spec new", "spec g name: spec license: MIT compatibility: 'Designed for Claude Code. On OpenCode and VS Code Copilot, AskUserQuestion falls back to numbered list. Core functionality preserved on all platforms.' -argument-hint: ' [plan-name] [--flags]' +argument-hint: ' [plan-name] [--branch] [--usage] [--flags]' --- > **Compatibility**: AskUserQuestion falls back to numbered list on non-Claude-Code platforms. @@ -20,6 +20,7 @@ command -v npx >/dev/null 2>&1 || echo "MISSING: npx" ``` /spec new add OAuth login to the settings page # inline objective → plans now +/spec new --usage add OAuth login to settings # also record planning AI usage → ai-usage.md /spec new auth-refactor # bare name → writes intent.md to fill in /spec new auth-refactor # re-run after filling it → plans /spec go my-plan diff --git a/skills/spec/references/ai-usage-template.md b/skills/spec/references/ai-usage-template.md new file mode 100644 index 0000000..361f152 --- /dev/null +++ b/skills/spec/references/ai-usage-template.md @@ -0,0 +1,66 @@ +# ai-usage.md Template + +Write this structure to `{PLAN_DIR}/ai-usage.md` when `/spec new` is run with `--usage`. Substitute all +`{...}` placeholders with computed values. This file is separate from `plan.md` — usage telemetry lives +only here. + +--- + +```markdown +# AI Usage — {plan-name} + +## Session + +- **Plan**: {plan-name} +- **Branch**: {branch} +- **Agent**: {agent-name} ({agent-version}) +- **Session file**: {session-file-path} +- **Session id**: {session-id} +- **Planning window**: {start-iso} → {end-iso} ({duration}) +- **Captured by**: {subagent | current-agent (fallback)} + +## Token Usage + +| Metric | Tokens | +| ------------------------- | -------- | +| Input | {n} | +| Output | {n} | +| Cache read | {n} | +| Cache creation | {n} | +| **Total (all classes)** | **{n}** | + +> **Total (all classes)** = input + output + cache read + cache creation — every token class the agent +> processed, summed. It is *not* a cost figure: cache reads are typically cheaper or free versus fresh +> input, so this over-states spend. Read it as "tokens processed", not "tokens billed". Values are what +> the agent's session file reports for assistant turns inside the planning window; this is a usage +> record, not a bill. + +## Tool Calls + +| Tool | Count | +| -------- | ----- | +| {name} | {n} | +| ... | | +| **Total**| **{n}** | + +## Notes + +{One or two lines of context — e.g. web searches/fetches performed, or any caveats.} +``` + +--- + +**Telemetry-unavailable variant** — when no session file could be located or parsed for the current +agent, write this instead of the tables above (keep the `## Session` block with whatever is known): + +```markdown +## Token Usage + +_Telemetry unavailable — could not locate or parse a session file for agent "{agent-name}". Planning +completed normally; only usage measurement is missing._ +``` + +**Rules:** +- This file is written only when `--usage` was passed. Never create it otherwise. +- Never mix this content into `plan.md`, `user-guide.md`, or implementation files. +- Always write the file when `--usage` is set — even on the unavailable path — so the record exists. diff --git a/skills/spec/references/usage-monitor.md b/skills/spec/references/usage-monitor.md new file mode 100644 index 0000000..047c31c --- /dev/null +++ b/skills/spec/references/usage-monitor.md @@ -0,0 +1,167 @@ +# Session-file usage monitor + +Operational spec for `/spec new --usage`. It locates the executing agent's session/transcript file, +aggregates token and tool telemetry, and writes `{PLAN_DIR}/ai-usage.md` using +`references/ai-usage-template.md`. Preferred execution is a background subagent; a current-agent +fallback is defined for when a subagent cannot be launched. + +This is agent-agnostic: Claude Code is parsed first-class; other agents are discovered best-effort and +degrade to the "telemetry unavailable" template variant rather than failing the plan. + +## Inputs (provided by `new.md`) + +- `PLAN_DIR` — absolute or repo-relative plan directory. +- `PLAN_NAME`, `BRANCH` — for the report header. +- `USAGE_START_EPOCH` — `date +%s` captured when `--usage` tracking began. +- `CWD` — the working directory `/spec new` is running in (`pwd`). + +## Step A — Locate the session file (agent-agnostic) + +Try providers in order; stop at the first that yields a readable file. + +1. **Claude Code.** Session JSONL lives at + `~/.claude/projects/{SLUG}/{sessionId}.jsonl`, where `{SLUG}` is `CWD` with every `/` replaced by + `-` (and a leading `-` retained). The active session is the most recently modified `.jsonl` whose + lines carry `"cwd":"{CWD}"`. + + Select the most recently modified `.jsonl` whose lines actually carry `"cwd":"{CWD}"` — do + **not** just grab the newest file. When this monitor runs as a background subagent, the + subagent's own transcript is typically the newest `.jsonl` in the same project dir; filtering by + `cwd` (and skipping the running subagent's own session id, when it is known via `$CLAUDE_SESSION_ID`) + keeps the measurement on the parent planning session. + + ```bash + SLUG=$(echo "$CWD" | sed 's#/#-#g') + CC_DIR="$HOME/.claude/projects/$SLUG" + SESSION_FILE="" + if [ -d "$CC_DIR" ]; then + SESSION_FILE=$(ls -t "$CC_DIR"/*.jsonl 2>/dev/null \ + | while read -r f; do + # skip the subagent's own transcript if we can identify it + [ -n "$CLAUDE_SESSION_ID" ] && [ "$(basename "$f" .jsonl)" = "$CLAUDE_SESSION_ID" ] && continue + grep -ql "\"cwd\":\"$CWD\"" "$f" && { echo "$f"; break; } + done) + fi + AGENT_NAME="claude-code" + ``` + +2. **Other agents (best-effort).** Check known transcript locations; if any exists and is JSON/JSONL, + set `SESSION_FILE` and `AGENT_NAME` accordingly. Probe (extend as agents are added): + + `nullglob` makes unmatched patterns expand to nothing (instead of staying literal) and `globstar` + makes `**` recurse; without them the loop silently no-ops on dead placeholder paths. + + ```bash + if [ -z "$SESSION_FILE" ]; then + shopt -s nullglob globstar 2>/dev/null || true + for cand in \ + "$HOME/.cursor/"*session*.jsonl \ + "$HOME/.config/opencode/"**/*.jsonl \ + "$HOME/.aider"*.jsonl ; do + [ -e "$cand" ] && { SESSION_FILE="$cand"; AGENT_NAME="other"; break; } + done + shopt -u nullglob globstar 2>/dev/null || true + fi + ``` + +3. **None found.** Leave `SESSION_FILE` empty → take the **unavailable path** in Step C. + +Derive, when a Claude Code file was found: + +```bash +SESSION_ID=$(basename "$SESSION_FILE" .jsonl) +AGENT_VERSION=$(grep -m1 '"version"' "$SESSION_FILE" | jq -r '.version' 2>/dev/null || echo "unknown") +``` + +## Step B — Aggregate telemetry (Claude Code JSONL) + +Each assistant line has `.message.usage` with `input_tokens`, `output_tokens`, +`cache_read_input_tokens`, `cache_creation_input_tokens`, and `server_tool_use.{web_search_requests, +web_fetch_requests}`. Tool invocations appear as `.message.content[] | select(.type=="tool_use") | +.name`. + +Bound the aggregation to the **planning window** the header advertises: each line carries an ISO-8601 +`.timestamp`; only assistant turns at or after `USAGE_START_EPOCH` belong to this `/spec new` run. On a +long-lived session, summing the whole file would over-count turns from before planning began. Pass the +start epoch into `jq` and filter on it (turns with no parseable timestamp are kept, so a session that +predates timestamped lines still measures rather than silently dropping to zero): + +```bash +read -r IN OUT CR CC WS WF <= $start) + | .message.usage ] as $u + | [ ([$u[]?.input_tokens // 0] | add // 0), + ([$u[]?.output_tokens // 0] | add // 0), + ([$u[]?.cache_read_input_tokens // 0] | add // 0), + ([$u[]?.cache_creation_input_tokens // 0] | add // 0), + ([$u[]?.server_tool_use.web_search_requests // 0] | add // 0), + ([$u[]?.server_tool_use.web_fetch_requests // 0] | add // 0) ] + | @tsv' "$SESSION_FILE" 2>/dev/null) +EOF +TOTAL=$(( ${IN:-0} + ${OUT:-0} + ${CR:-0} + ${CC:-0} )) + +# Tool-call breakdown (name -> count), and total tool calls — same window bound: +WINDOW_FILTER='select(.type=="assistant") + | select((.timestamp // "" | if . == "" then '"${USAGE_START_EPOCH:-0}"' else (fromdateiso8601? // '"${USAGE_START_EPOCH:-0}"') end) >= '"${USAGE_START_EPOCH:-0}"') + | .message.content[]? | select(.type=="tool_use") | .name' +TOOL_TABLE=$(jq -rc "$WINDOW_FILTER" "$SESSION_FILE" 2>/dev/null | sort | uniq -c | awk '{printf "| %s | %s |\n", $2, $1}') +TOOL_TOTAL=$(jq -rc "$WINDOW_FILTER" "$SESSION_FILE" 2>/dev/null | wc -l | tr -d ' ') +``` + +If `jq` errors or the file is not the expected schema (empty `IN/OUT` and no tool rows), treat as +**unavailable** (Step C, unavailable variant) — do not emit zeros as if measured. + +Compute the window: + +```bash +END_EPOCH=$(date +%s) +START_ISO=$(date -u -r "$USAGE_START_EPOCH" +"%Y-%m-%dT%H:%M:%SZ" 2>/dev/null || date -u +"%Y-%m-%dT%H:%M:%SZ") +END_ISO=$(date -u +"%Y-%m-%dT%H:%M:%SZ") +DUR=$(( END_EPOCH - USAGE_START_EPOCH ))s +``` + +## Step C — Write `ai-usage.md` + +Read `references/ai-usage-template.md`. Substitute the header fields +(`{plan-name}`, `{branch}`, `{agent-name}`, `{agent-version}`, `{session-file-path}`, `{session-id}`, +`{start-iso}`, `{end-iso}`, `{duration}`, `{Captured by}`). + +- **Measured path:** fill the Token Usage table (`IN/OUT/CR/CC/TOTAL`), the Tool Calls table + (`TOOL_TABLE` rows + `TOOL_TOTAL`), and a Notes line mentioning web search/fetch counts + (`WS`/`WF`) if non-zero. +- **Unavailable path:** use the template's "telemetry-unavailable variant" for the Token Usage / + Tool Calls sections; still fill whatever `## Session` fields are known. + +Write to `{PLAN_DIR}/ai-usage.md`. Verify: `test -s "{PLAN_DIR}/ai-usage.md"`. + +## Subagent prompt (preferred) + +`new.md` launches this as `subagent_type: general-purpose`, `run_in_background: true`, and stores the +Task id as `USAGE_TASK_ID`. Prompt handed to the subagent: + + You are an AI-usage monitor for a `/spec new` planning run. Do NOT plan or edit any plan files. + + Inputs: + - PLAN_DIR = {PLAN_DIR} + - PLAN_NAME = {PLAN_NAME} + - BRANCH = {BRANCH} + - CWD = {CWD} + - USAGE_START_EPOCH = {USAGE_START_EPOCH} + + Follow `~/.claude/skills/spec/references/usage-monitor.md` Steps A–C exactly: + locate the current agent's session file (agent-agnostic), aggregate token and tool telemetry, + and write {PLAN_DIR}/ai-usage.md using ~/.claude/skills/spec/references/ai-usage-template.md. + If no session file can be located or parsed, write the "telemetry unavailable" variant — never + fail. Print the final path and the total token count when done. + +The subagent runs to completion in the background; `new.md` blocks on its `TaskOutput` only at Step 6, +after planning is otherwise finished, so tracking never delays the plan. + +## Current-agent fallback + +If a subagent cannot be launched (tool unavailable, or `TaskCreate` fails), the planning agent itself +runs Steps A–C at Step 6 — after the plan files are written — and writes `ai-usage.md`. Set +`Captured by = current-agent (fallback)` in the header. diff --git a/skills/spec/references/workflows/new.md b/skills/spec/references/workflows/new.md index 073f747..d2759ae 100644 --- a/skills/spec/references/workflows/new.md +++ b/skills/spec/references/workflows/new.md @@ -33,6 +33,9 @@ Received from dispatcher: ``` VALIDATE_MODE=false [[ "$*" =~ --validate|-v ]] && VALIDATE_MODE=true + +USAGE_MODE=false +[[ "$*" =~ --usage ]] && USAGE_MODE=true ``` ## Step 0.5: Detect Branch Context @@ -239,6 +242,26 @@ If `RESEARCH_CONTEXT` is set, copy or link the explore artifacts into `$PLAN_DIR Report: `✓ Plan directory created at: $PLAN_DIR` +### 5.2u: Start AI Usage Monitor (only if `--usage`) + +Runs **only when `USAGE_MODE=true`**. Skip this subsection entirely otherwise. + +```bash +if [[ "$USAGE_MODE" == "true" ]]; then + USAGE_START_EPOCH=$(date +%s) + CWD=$(pwd) +fi +``` + +Then launch the monitor as a background subagent (`subagent_type: general-purpose`, +`run_in_background: true`) using the prompt in `references/usage-monitor.md` ("Subagent prompt"), +substituting `{PLAN_DIR}`, `{PLAN_NAME}`, `{BRANCH}` (=`$TARGET_BRANCH` or `$CURRENT_BRANCH`), `{CWD}`, +and `{USAGE_START_EPOCH}`. Store the Task id as `USAGE_TASK_ID`. + +If the subagent cannot be launched (no `TaskCreate` tool, or launch fails), set `USAGE_FALLBACK=true` +and continue — the current agent will do the tally itself in Step 6. The monitor never blocks planning: +you do not wait for it here. + ### 5.3: Create Plan Files **a. plan.md** at `$PLAN_DIR/plan.md` @@ -393,6 +416,15 @@ If `SUGGESTED_ALLOW` is non-empty, present permission suggestions as information Then automatically apply them: read `.claude/settings.json` (start from `{}` if absent), union `permissions.allow` array with `SUGGESTED_ALLOW` (deduplicate, sort), write back. Report: `✓ Added {N} allow entries to .claude/settings.json`. +**Finalize AI usage tracking (only if `USAGE_MODE=true`):** Skip entirely otherwise. If a monitor +subagent was launched (`USAGE_TASK_ID` set), block on it: `TaskOutput(id: USAGE_TASK_ID, block: true)`. +If it did not write the file, or `USAGE_FALLBACK=true`, or no subagent was launched, run +`references/usage-monitor.md` Steps A–C yourself now (the plan files already exist) and write +`{PLAN_DIR}/ai-usage.md` — including the "telemetry unavailable" variant if no session file is found. +Confirm `test -s "$PLAN_DIR/ai-usage.md"`, then add one line to the completion report: +`📊 AI usage recorded: {PLAN_DIR}/ai-usage.md`. This file is separate from plan.md and is written only +in this mode. + Do **not** ask whether the plan covers everything. Finish immediately: ⛔ **STOP HERE. Do not implement anything.** Your job is complete. If `BG_MODE=true`, launch `/spec go {plan-name}` (pass `--silent` if `SILENT=true`) and report launch. If `BG_MODE=false`, report completion: