diff --git a/plugins/claude-code/README.md b/plugins/claude-code/README.md index a7c39e4..0423e56 100644 --- a/plugins/claude-code/README.md +++ b/plugins/claude-code/README.md @@ -1,20 +1,22 @@ # Supercompact — Claude Code Plugin -Entity-preservation conversation compaction for Claude Code. Replaces the built-in LLM-based `/compact` with EITF scoring — **~400x faster** and **2x better entity retention**. +Entity-preservation conversation compaction for Claude Code. **~400x faster** and **2x better entity retention** than the built-in LLM-based `/compact`. ## Quick Install ```bash -git clone https://github.com/yourusername/supercompact.git +git clone https://github.com/heiervang-technologies/supercompact.git cd supercompact/plugins/claude-code ./install.sh ``` -**Prerequisites:** Python 3.11+, [uv](https://github.com/astral-sh/uv) +The installer automatically registers the plugin in `~/.claude/settings.json`. Restart Claude Code, then use `/supercompact`. + +**Prerequisites:** Python 3.11+, [uv](https://github.com/astral-sh/uv), jq ## What It Does -When Claude Code compacts your conversation (either automatically or via `/compact`), it normally calls an LLM to summarize — slow (~30s) and lossy. Supercompact replaces this with **EITF** (Entity-frequency Inverse Turn Frequency), a zero-model algorithm that: +When Claude Code compacts your conversation (either automatically or via `/compact`), it normally calls an LLM to summarize — slow (~30s) and lossy. Supercompact uses **EITF** (Entity-frequency Inverse Turn Frequency), a zero-model algorithm that: 1. Extracts structured entities (file paths, errors, functions, URLs, etc.) 2. Scores each conversation turn by entity importance × rarity @@ -25,13 +27,13 @@ Result: compaction in **~0.2 seconds** with **2x better retention** of file path ## How It Works -The installer sets up three integration points: +The plugin provides three integration points: -1. **cli.js patch** — Replaces the LLM API call in Claude Code's main compaction function with a subprocess call to supercompact. Falls back to the original LLM on error. +1. **`/supercompact` command** — On-demand compaction. Replaces the session with a compacted version and restarts. This is the primary interface. -2. **PreCompact hook** — Backs up the full transcript before any compaction runs, and produces a supercompact alternative alongside Claude's built-in result. +2. **PreCompact hook** — When Claude's built-in compaction triggers, the hook backs up the full transcript before it's lost. The backup is saved as `*.pre-compact-full` alongside the session JSONL. -3. **`/supercompact` command** — Manual on-demand compaction with configurable method and budget. +3. **cli.js patch** *(npm installations only)* — Replaces the LLM API call in Claude Code's compaction function with supercompact. Falls back to the original LLM on error. Not available on standalone binary installations. ## Configuration @@ -70,6 +72,16 @@ Manual compaction. Examples: ./install.sh --patch-only # Patch cli.js only (plugin must be installed first) ``` +## Update + +```bash +cd supercompact +git pull +./plugins/claude-code/install.sh +``` + +Re-running the installer is safe — it replaces all files and is fully idempotent. + ## Uninstall ```bash @@ -95,9 +107,10 @@ Manual compaction. Examples: ├── hooks/ │ └── hooks.json # PreCompact hook registration ├── hooks-handlers/ - │ └── supercompact-precompact.sh + │ └── supercompact-precompact.sh # Backup-only hook └── scripts/ - ├── patcher.py # cli.js patching logic + ├── compact-session.sh # Main compaction script + ├── patcher.py # cli.js patching logic └── patch-compaction.sh ``` @@ -105,6 +118,15 @@ Manual compaction. Examples: Hook activity is logged to `~/.cache/supercompact/hook.log`. +## Standalone Binary Installation + +If Claude Code is installed as a standalone binary (not via npm), the cli.js patch cannot be applied. The installer detects this automatically, skips patching, and configures `settings.json` for you. + +In standalone mode: +- **`/supercompact`** — Works fully. This is the primary way to compact. +- **`/compact`** — Still uses Claude's built-in LLM compaction (cannot be replaced without cli.js patch). +- **PreCompact hook** — Backs up the full transcript before Claude's built-in compaction runs. + ## Troubleshooting **Compaction not working after Claude Code update:** @@ -118,7 +140,7 @@ Check `~/.cache/supercompact/hook.log` for errors. Common causes: - Python/uv not in PATH during compaction - Supercompact directory removed or corrupted -**Verify patch status:** +**Verify patch status (npm installations only):** ```bash grep -c "SUPERCOMPACT_EITF" "$(readlink -f "$(which claude)" | sed 's|[^/]*$|cli.js|')" # 1 = patched, 0 = not patched diff --git a/plugins/claude-code/commands/supercompact.md b/plugins/claude-code/commands/supercompact.md index 5785357..01d6154 100644 --- a/plugins/claude-code/commands/supercompact.md +++ b/plugins/claude-code/commands/supercompact.md @@ -1,73 +1,26 @@ --- description: EITF entity-preservation compaction (~400x faster than /compact, 2x better entity retention) argument-hint: "[budget] [--method eitf|setcover|dedup]" -allowed-tools: Bash(cd *), Bash(uv *), Bash(PROJECT_DIR*), Bash(JSONL_FILE*), Bash(ls *), Bash(wc *), Bash(cp *), Bash(mv *), Bash(restart-claude*), Bash(echo *) +allowed-tools: Bash(*/compact-session.sh*), Bash(restart-claude*) --- # Supercompact — Entity-Preservation Compaction -**CRITICAL: Do NOT use the built-in /compact command. You must follow the exact steps below using Bash tool calls.** - -You are running the supercompact algorithm. This is completely separate from Claude Code's built-in /compact. You must execute the bash commands below, not delegate to any built-in compaction. - -## Configuration - -Settings come from environment variables (set via plugin config): -- `PLUGIN_SETTING_METHOD` — scoring method (default: `eitf`). Options: `eitf`, `setcover`, `dedup` -- `PLUGIN_SETTING_BUDGET` — token budget (default: `80000`) - -The user can override these via arguments: `/supercompact 120000 --method setcover` - -## Step 1: Find the conversation JSONL and supercompact directory +Run the compaction script. It will find the session JSONL automatically, compact it, and report results. ```bash -PROJECT_DIR=$(echo "$PWD" | sed 's|/|-|g; s|^|'"$HOME"'/.claude/projects/|') -JSONL_FILE=$(ls -t "$PROJECT_DIR"/*.jsonl 2>/dev/null | head -1) -echo "JSONL: $JSONL_FILE" -wc -l "$JSONL_FILE" +SCRIPT="${CLAUDE_PLUGIN_ROOT:-${HOME}/.local/share/supercompact/claude-code/plugin}/scripts/compact-session.sh" +"$SCRIPT" $ARGUMENTS ``` -## Step 2: Run compaction - -Parse $ARGUMENTS for an optional numeric budget and `--method `. Fall back to env vars, then defaults. +If the script succeeds and reports compaction was performed (not "already within budget"), restart to load the compacted context: ```bash -METHOD="${PLUGIN_SETTING_METHOD:-eitf}" -BUDGET="${PLUGIN_SETTING_BUDGET:-80000}" -# Override from arguments if provided (e.g. "/supercompact 120000 --method setcover") -for arg in $ARGUMENTS; do - if [[ "$arg" =~ ^[0-9]+$ ]]; then BUDGET="$arg"; fi - if [[ "$prev" == "--method" ]]; then METHOD="$arg"; fi - prev="$arg" -done -# Find supercompact installation -SUPERCOMPACT_DIR="$HOME/.local/share/supercompact/claude-code/supercompact" -if [[ ! -f "$SUPERCOMPACT_DIR/compact.py" ]]; then - echo "ERROR: supercompact not found at $SUPERCOMPACT_DIR. Run install.sh first." - exit 1 -fi -echo "Method: $METHOD, Budget: $BUDGET" -cd "$SUPERCOMPACT_DIR" && uv run python compact.py "$JSONL_FILE" --method "$METHOD" --budget "$BUDGET" --output /tmp/supercompact-output.jsonl --verbose +restart-claude "Session compacted with supercompact. Restarting to load compacted context." ``` -## Step 3: Replace the session JSONL - -```bash -cp "$JSONL_FILE" "${JSONL_FILE}.pre-supercompact" -mv /tmp/supercompact-output.jsonl "$JSONL_FILE" -echo "Replaced session JSONL (backup: ${JSONL_FILE}.pre-supercompact)" -``` - -## Step 4: Report results briefly - -Report: method used, turns kept vs dropped, compression ratio, wall clock time. - -## Step 5: Restart to reload compacted context - -The JSONL on disk is now compacted, but the live session still has old context in memory. Restart to load the compacted version: +If `restart-claude` is not available, tell the user: "Run `/quit` then `claude --resume` to load the compacted context." -```bash -restart-claude "Session compacted with supercompact ($METHOD). Restarting to load compacted context." -``` +If the script reports "already within budget", tell the user and do NOT restart. -If `restart-claude` is not available, tell the user: "Run `/quit` then `claude --resume` to load the compacted context." +If the script fails, show the error output to the user and do not restart. diff --git a/plugins/claude-code/hooks-handlers/supercompact-precompact.sh b/plugins/claude-code/hooks-handlers/supercompact-precompact.sh index d0c6dd4..f4ff67b 100755 --- a/plugins/claude-code/hooks-handlers/supercompact-precompact.sh +++ b/plugins/claude-code/hooks-handlers/supercompact-precompact.sh @@ -1,42 +1,18 @@ #!/usr/bin/env bash -# supercompact-precompact.sh - PreCompact hook for entity-preservation compaction +# supercompact-precompact.sh - PreCompact hook (backup-only) # -# Triggered when Claude Code is about to compact the conversation. # The PreCompact hook CANNOT block or replace Claude's built-in compaction — -# it is notification-only. So we use it to: +# it is notification-only. Running supercompact here is wasted work since +# Claude's LLM compaction overwrites the result anyway. # +# Instead, we just: # 1. Back up the full transcript before Claude's summarization loses detail -# 2. Run compaction (configurable method) to produce a superior alternative -# 3. The user can later resume from the supercompact version instead of Claude's -# -# Configuration via environment variables: -# PLUGIN_SETTING_METHOD Scoring method (default: eitf) -# PLUGIN_SETTING_BUDGET Token budget (default: 80000) +# 2. Log the event +# 3. Clean up old backups set -euo pipefail -# Resolve supercompact installation root -# Layout: ~/.local/share/supercompact/claude-code/plugin/hooks-handlers/THIS_SCRIPT -# ~/.local/share/supercompact/claude-code/supercompact/compact.py -SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" -PLUGIN_ROOT="$(cd "${SCRIPT_DIR}/.." && pwd)" -INSTALL_ROOT="$(cd "${PLUGIN_ROOT}/.." && pwd)" -SUPERCOMPACT_DIR="${INSTALL_ROOT}/supercompact" - -if [[ ! -f "${SUPERCOMPACT_DIR}/compact.py" ]]; then - # Fallback: check if supercompact is bundled inside the plugin dir (dev mode) - if [[ -f "${PLUGIN_ROOT}/supercompact/compact.py" ]]; then - SUPERCOMPACT_DIR="${PLUGIN_ROOT}/supercompact" - else - echo "$(date -Iseconds) ERROR: supercompact not found at ${SUPERCOMPACT_DIR}" >> "${HOME}/.cache/supercompact/hook.log" 2>/dev/null - exit 0 - fi -fi - -METHOD="${PLUGIN_SETTING_METHOD:-eitf}" -BUDGET="${PLUGIN_SETTING_BUDGET:-80000}" LOG_DIR="${HOME}/.cache/supercompact" - mkdir -p "${LOG_DIR}" # Read hook input from stdin (JSON with transcript_path, session_id, trigger, etc.) @@ -48,42 +24,22 @@ JSONL_FILE=$(echo "${HOOK_INPUT}" | jq -r '.transcript_path // empty') echo "$(date -Iseconds) PreCompact hook triggered (trigger=${TRIGGER})" >> "${LOG_DIR}/hook.log" if [[ -z "${JSONL_FILE}" || ! -f "${JSONL_FILE}" ]]; then - echo "$(date -Iseconds) ERROR: No transcript_path in hook input or file missing" >> "${LOG_DIR}/hook.log" - exit 0 + echo "$(date -Iseconds) ERROR: No transcript_path in hook input or file missing" >> "${LOG_DIR}/hook.log" + exit 0 fi JSONL_SIZE=$(wc -l < "${JSONL_FILE}") echo "$(date -Iseconds) Transcript: ${JSONL_FILE} (${JSONL_SIZE} lines)" >> "${LOG_DIR}/hook.log" -# 1. Back up the full transcript before Claude's compaction destroys detail +# Back up the full transcript before Claude's compaction destroys detail BACKUP_FILE="${JSONL_FILE}.pre-compact-full" cp "${JSONL_FILE}" "${BACKUP_FILE}" echo "$(date -Iseconds) Full backup saved: ${BACKUP_FILE}" >> "${LOG_DIR}/hook.log" -# 2. Run supercompact to produce a superior alternative -SC_OUTPUT="${JSONL_FILE}.supercompact" - -echo "$(date -Iseconds) Running supercompact (method=${METHOD}, budget=${BUDGET})" >> "${LOG_DIR}/hook.log" - -cd "${SUPERCOMPACT_DIR}" -if uv run python compact.py "${JSONL_FILE}" \ - --method "${METHOD}" \ - --budget "${BUDGET}" \ - --output "${SC_OUTPUT}" 2>> "${LOG_DIR}/hook.log"; then +# Clean up old backups (keep last 3) +ls -t "${JSONL_FILE}.pre-compact-full"* 2>/dev/null | tail -n +4 | xargs rm -f 2>/dev/null || true +ls -t "${JSONL_FILE}.pre-supercompact"* 2>/dev/null | tail -n +4 | xargs rm -f 2>/dev/null || true - SC_SIZE=$(wc -l < "${SC_OUTPUT}") - echo "$(date -Iseconds) Supercompact (${METHOD}): ${JSONL_SIZE} -> ${SC_SIZE} lines (saved as .supercompact)" >> "${LOG_DIR}/hook.log" - - # Clean up old backups (keep last 3 of each type) - ls -t "${JSONL_FILE}.pre-compact-full"* 2>/dev/null | tail -n +4 | xargs rm -f 2>/dev/null || true - ls -t "${JSONL_FILE}.supercompact"* 2>/dev/null | tail -n +4 | xargs rm -f 2>/dev/null || true - - echo "$(date -Iseconds) SUCCESS: Supercompact alternative ready at ${SC_OUTPUT}" >> "${LOG_DIR}/hook.log" - echo "$(date -Iseconds) NOTE: Claude's built-in compaction will still run (hook cannot block it)" >> "${LOG_DIR}/hook.log" - echo "$(date -Iseconds) To use supercompact version: cp '${SC_OUTPUT}' '${JSONL_FILE}'" >> "${LOG_DIR}/hook.log" -else - echo "$(date -Iseconds) ERROR: Supercompact (${METHOD}) failed (Claude's compaction will proceed)" >> "${LOG_DIR}/hook.log" - rm -f "${SC_OUTPUT}" 2>/dev/null || true -fi +echo "$(date -Iseconds) Backup-only hook complete (use /supercompact for manual compaction)" >> "${LOG_DIR}/hook.log" exit 0 diff --git a/plugins/claude-code/install.sh b/plugins/claude-code/install.sh index a361c7f..4369801 100755 --- a/plugins/claude-code/install.sh +++ b/plugins/claude-code/install.sh @@ -168,19 +168,39 @@ if [[ "$DO_INSTALL" == true ]]; then ok "Plugin installed to ${INSTALL_DIR}" - # Print plugin-dir usage - echo "" - info "To load the plugin, use one of:" - echo " claude --plugin-dir ${PLUGIN_DEST}" - echo "" - echo " Or add to ~/.claude/settings.json:" - echo " { \"pluginDirs\": [\"${PLUGIN_DEST}\"] }" - echo "" + # Auto-configure settings.json to load the plugin + SETTINGS_FILE="${HOME}/.claude/settings.json" + info "Configuring Claude Code to load plugin..." + mkdir -p "$(dirname "${SETTINGS_FILE}")" + + if [[ ! -f "${SETTINGS_FILE}" ]]; then + # Create settings.json with pluginDirs + echo '{"pluginDirs":["'"${PLUGIN_DEST}"'"]}' | jq . > "${SETTINGS_FILE}" + ok "Created ${SETTINGS_FILE} with pluginDirs" + elif jq -e '.pluginDirs' "${SETTINGS_FILE}" >/dev/null 2>&1; then + # pluginDirs exists — check if our path is already there + if jq -e --arg p "${PLUGIN_DEST}" '.pluginDirs | index($p)' "${SETTINGS_FILE}" >/dev/null 2>&1; then + ok "Plugin already registered in settings.json" + else + # Add our path to existing pluginDirs array + jq --arg p "${PLUGIN_DEST}" '.pluginDirs += [$p]' "${SETTINGS_FILE}" > "${SETTINGS_FILE}.tmp" \ + && mv "${SETTINGS_FILE}.tmp" "${SETTINGS_FILE}" + ok "Added plugin to existing pluginDirs in settings.json" + fi + else + # settings.json exists but no pluginDirs key — add it + jq --arg p "${PLUGIN_DEST}" '. + {pluginDirs: [$p]}' "${SETTINGS_FILE}" > "${SETTINGS_FILE}.tmp" \ + && mv "${SETTINGS_FILE}.tmp" "${SETTINGS_FILE}" + ok "Added pluginDirs to settings.json" + fi fi # ------------------------------------------------------------------ # Patch cli.js # ------------------------------------------------------------------ +PATCH_APPLIED=false +STANDALONE_BINARY=false + if [[ "$DO_PATCH" == true ]]; then SUPERCOMPACT_DEST="${INSTALL_DIR}/supercompact" @@ -188,16 +208,33 @@ if [[ "$DO_PATCH" == true ]]; then fatal "Supercompact not installed at ${SUPERCOMPACT_DEST}. Run install first (without --patch-only)." fi - echo "" - info "Patching Claude Code cli.js..." - bash "${INSTALL_DIR}/plugin/scripts/patch-compaction.sh" "${SUPERCOMPACT_DEST}" - EXIT_CODE=$? + # Detect standalone binary vs npm installation + CLAUDE_BIN="${CLAUDE_BIN:-$(which claude 2>/dev/null || echo "")}" + CLAUDE_REAL="" + if [[ -n "$CLAUDE_BIN" ]]; then + CLAUDE_REAL="$(readlink -f "$CLAUDE_BIN" 2>/dev/null || echo "$CLAUDE_BIN")" + fi - if [[ $EXIT_CODE -eq 0 ]]; then - ok "cli.js patched — compaction now uses supercompact" + if [[ -n "$CLAUDE_REAL" ]] && head -c 4 "$CLAUDE_REAL" 2>/dev/null | grep -q "ELF\|MZ"; then + STANDALONE_BINARY=true + echo "" + warn "Claude Code is installed as a standalone binary (not via npm)" + warn "cli.js patching is not available for standalone installations" + info "The /supercompact slash command and PreCompact hook will still work" + info "Use '/supercompact' for on-demand compaction" else - err "Patching failed (exit code $EXIT_CODE)" - exit $EXIT_CODE + echo "" + info "Patching Claude Code cli.js..." + bash "${INSTALL_DIR}/plugin/scripts/patch-compaction.sh" "${SUPERCOMPACT_DEST}" + EXIT_CODE=$? + + if [[ $EXIT_CODE -eq 0 ]]; then + ok "cli.js patched — compaction now uses supercompact" + PATCH_APPLIED=true + else + warn "cli.js patching failed (exit code $EXIT_CODE)" + warn "The /supercompact slash command and PreCompact hook will still work" + fi fi fi @@ -210,15 +247,19 @@ echo "" echo "What's installed:" echo " • Supercompact library at ${INSTALL_DIR}/supercompact/" echo " • Plugin at ${INSTALL_DIR}/plugin/" -if [[ "$DO_PATCH" == true ]]; then +echo " • Plugin registered in ~/.claude/settings.json" +if [[ "$PATCH_APPLIED" == true ]]; then echo " • cli.js patched for automatic compaction replacement" fi echo "" -echo "Configuration (via environment variables or plugin settings):" -echo " PLUGIN_SETTING_METHOD=eitf # eitf, setcover, dedup" -echo " PLUGIN_SETTING_BUDGET=80000 # token budget" -echo " PLUGIN_SETTING_FALLBACK_TO_BUILTIN=true # fall back to LLM on error" -echo "" -if [[ "$DO_PATCH" == true ]]; then - echo "Restart Claude Code to activate the patch." +echo "Usage:" +if [[ "$PATCH_APPLIED" == true ]]; then + echo " /compact and /supercompact both use supercompact now." + echo " Restart Claude Code to activate." +else + echo " /supercompact # On-demand entity-preservation compaction" + echo " /supercompact 120000 # Custom token budget" fi +echo "" +echo "To update later: git pull && ./install.sh" +echo "To uninstall: ./uninstall.sh" diff --git a/plugins/claude-code/scripts/compact-session.sh b/plugins/claude-code/scripts/compact-session.sh new file mode 100755 index 0000000..8432872 --- /dev/null +++ b/plugins/claude-code/scripts/compact-session.sh @@ -0,0 +1,132 @@ +#!/usr/bin/env bash +# compact-session.sh - Self-contained supercompact session compaction +# +# Finds the current Claude Code session JSONL, runs supercompact, +# backs up the original, and replaces it with the compacted version. +# +# Usage: compact-session.sh [budget] [--method name] +# +# Environment: +# CLAUDE_PROJECT_DIR Project dir set by Claude Code (preferred for JSONL lookup) +# PLUGIN_SETTING_METHOD Scoring method (default: eitf) +# PLUGIN_SETTING_BUDGET Token budget (default: 80000) + +set -euo pipefail + +# --- Resolve supercompact installation --- +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +PLUGIN_ROOT="$(cd "${SCRIPT_DIR}/.." && pwd)" +INSTALL_ROOT="$(cd "${PLUGIN_ROOT}/.." && pwd)" +SUPERCOMPACT_DIR="${INSTALL_ROOT}/supercompact" + +if [[ ! -f "${SUPERCOMPACT_DIR}/compact.py" ]]; then + # Dev mode: repo layout + if [[ -f "${PLUGIN_ROOT}/../../supercompact/compact.py" ]]; then + SUPERCOMPACT_DIR="$(cd "${PLUGIN_ROOT}/../../supercompact" && pwd)" + else + echo "ERROR: supercompact not found at ${SUPERCOMPACT_DIR}" + exit 1 + fi +fi + +# --- Parse arguments --- +METHOD="${PLUGIN_SETTING_METHOD:-eitf}" +BUDGET="${PLUGIN_SETTING_BUDGET:-80000}" +prev="" +for arg in "$@"; do + if [[ "$arg" =~ ^[0-9]+$ ]]; then BUDGET="$arg"; fi + if [[ "$prev" == "--method" ]]; then METHOD="$arg"; fi + prev="$arg" +done + +# --- Find the session JSONL --- +find_project_dir() { + # Prefer CLAUDE_PROJECT_DIR if set + if [[ -n "${CLAUDE_PROJECT_DIR:-}" && -d "${CLAUDE_PROJECT_DIR}" ]]; then + echo "${CLAUDE_PROJECT_DIR}" + return + fi + # Fallback: derive from PWD (same logic Claude Code uses) + local derived + derived="${HOME}/.claude/projects/$(echo "${PWD}" | sed 's|/|-|g')" + if [[ -d "${derived}" ]]; then + echo "${derived}" + return + fi + echo "" +} + +PROJECT_DIR="$(find_project_dir)" +if [[ -z "${PROJECT_DIR}" ]]; then + echo "ERROR: Could not find Claude project directory" + echo " Tried CLAUDE_PROJECT_DIR=${CLAUDE_PROJECT_DIR:-}" + echo " Tried PWD-derived=${HOME}/.claude/projects/$(echo "${PWD}" | sed 's|/|-|g')" + exit 1 +fi + +JSONL_FILE="$(ls -t "${PROJECT_DIR}"/*.jsonl 2>/dev/null | head -1)" +if [[ -z "${JSONL_FILE}" || ! -f "${JSONL_FILE}" ]]; then + echo "ERROR: No .jsonl files found in ${PROJECT_DIR}" + exit 1 +fi + +LINES_BEFORE=$(wc -l < "${JSONL_FILE}") +echo "Session JSONL: ${JSONL_FILE}" +echo "Lines before: ${LINES_BEFORE}" +echo "Method: ${METHOD}" +echo "Budget: ${BUDGET}" +echo "" + +# --- Run supercompact --- +SC_OUTPUT="/tmp/supercompact-output-$$.jsonl" +trap 'rm -f "${SC_OUTPUT}"' EXIT + +START_TIME=$(date +%s%N) + +cd "${SUPERCOMPACT_DIR}" +SC_STDOUT=$(uv run python compact.py compact "${JSONL_FILE}" \ + --method "${METHOD}" \ + --budget "${BUDGET}" \ + --output "${SC_OUTPUT}" \ + --verbose 2>&1) || { + echo "" + echo "${SC_STDOUT}" + echo "" + echo "ERROR: supercompact failed" + exit 1 +} + +END_TIME=$(date +%s%N) +ELAPSED_MS=$(( (END_TIME - START_TIME) / 1000000 )) + +echo "${SC_STDOUT}" + +if [[ ! -f "${SC_OUTPUT}" ]]; then + # Already within budget — not an error + echo "" + echo "Session is already within budget. No compaction needed." + exit 0 +fi + +# --- Backup and replace --- +BACKUP_FILE="${JSONL_FILE}.pre-supercompact" +cp "${JSONL_FILE}" "${BACKUP_FILE}" +mv "${SC_OUTPUT}" "${JSONL_FILE}" +trap - EXIT # output file moved, no cleanup needed + +LINES_AFTER=$(wc -l < "${JSONL_FILE}") + +# --- Report --- +if [[ ${LINES_BEFORE} -gt 0 ]]; then + REDUCTION=$(( (LINES_BEFORE - LINES_AFTER) * 100 / LINES_BEFORE )) +else + REDUCTION=0 +fi + +echo "" +echo "=== Compaction Complete ===" +echo "Lines before: ${LINES_BEFORE}" +echo "Lines after: ${LINES_AFTER}" +echo "Reduction: ${REDUCTION}%" +echo "Time: ${ELAPSED_MS}ms" +echo "Backup: ${BACKUP_FILE}" diff --git a/plugins/claude-code/uninstall.sh b/plugins/claude-code/uninstall.sh index e065ea2..fde814f 100755 --- a/plugins/claude-code/uninstall.sh +++ b/plugins/claude-code/uninstall.sh @@ -131,6 +131,18 @@ if [[ "$DO_REMOVE" == true ]]; then ok "No plugin files found at ${INSTALL_DIR}" fi + # Remove pluginDirs entry from settings.json + SETTINGS_FILE="${HOME}/.claude/settings.json" + PLUGIN_PATH="${INSTALL_DIR}/plugin" + if [[ -f "${SETTINGS_FILE}" ]] && command -v jq &>/dev/null; then + if jq -e --arg p "${PLUGIN_PATH}" '.pluginDirs | index($p)' "${SETTINGS_FILE}" >/dev/null 2>&1; then + info "Removing plugin from settings.json..." + jq --arg p "${PLUGIN_PATH}" '.pluginDirs = [.pluginDirs[] | select(. != $p)]' "${SETTINGS_FILE}" > "${SETTINGS_FILE}.tmp" \ + && mv "${SETTINGS_FILE}.tmp" "${SETTINGS_FILE}" + ok "Plugin removed from settings.json" + fi + fi + # Remove log directory LOG_DIR="${HOME}/.cache/supercompact" if [[ -d "$LOG_DIR" ]]; then diff --git a/tests/test_pareto_pure.py b/tests/test_pareto_pure.py new file mode 100644 index 0000000..ee7d99f --- /dev/null +++ b/tests/test_pareto_pure.py @@ -0,0 +1,93 @@ +"""Pure-function tests for pareto.py pareto_frontier. + +Covers the Pareto frontier extraction: minimize x, maximize y. +No file system, network, or GPU access. +""" + +from __future__ import annotations + +import sys +from pathlib import Path + +import pytest + +sys.path.insert(0, str(Path(__file__).resolve().parent.parent)) + +from pareto import pareto_frontier + + +class TestParetoFrontier: + def test_single_point(self): + result = pareto_frontier([(1.0, 0.5)]) + assert result == [0] + + def test_two_dominated(self): + # (1, 0.8) dominates (2, 0.5) — lower x, higher y + points = [(1.0, 0.8), (2.0, 0.5)] + result = pareto_frontier(points) + assert result == [0] + + def test_two_non_dominated(self): + # (1, 0.5) and (2, 0.9) — tradeoff: lower x vs higher y + points = [(1.0, 0.5), (2.0, 0.9)] + result = pareto_frontier(points) + assert 0 in result + assert 1 in result + + def test_three_points_one_dominated(self): + # (1, 0.8), (2, 0.9), (3, 0.7) + # (3, 0.7) is dominated by (2, 0.9) — higher x AND lower y + points = [(1.0, 0.8), (2.0, 0.9), (3.0, 0.7)] + result = pareto_frontier(points) + assert 0 in result + assert 1 in result + assert 2 not in result + + def test_all_same_x(self): + # Same x: only the one with max y is Pareto-optimal + points = [(1.0, 0.3), (1.0, 0.9), (1.0, 0.5)] + result = pareto_frontier(points) + # The first sorted point will be one of the x=1.0 points + # Only the one with y=0.9 matters — but depends on sort stability + # At minimum, index 1 (y=0.9) should be on the frontier + assert 1 in result + + def test_all_same_y(self): + # Same y: only the leftmost (min x) is kept since no subsequent improves y + points = [(3.0, 0.5), (1.0, 0.5), (2.0, 0.5)] + result = pareto_frontier(points) + # First sorted by x: index 1 (x=1.0) is first, y=0.5 becomes best_y + # No subsequent point exceeds 0.5, so only first is on frontier + assert len(result) == 1 + assert 1 in result # (1.0, 0.5) is the leftmost + + def test_staircase_all_pareto(self): + # Perfect staircase: each point has lower x but also lower y + points = [(1.0, 0.1), (2.0, 0.4), (3.0, 0.7), (4.0, 0.9)] + result = pareto_frontier(points) + assert result == [0, 1, 2, 3] + + def test_empty_input(self): + result = pareto_frontier([]) + assert result == [] + + def test_returns_list(self): + result = pareto_frontier([(0.0, 1.0)]) + assert isinstance(result, list) + + def test_indices_are_original(self): + # Verify returned indices refer to original positions, not sorted + points = [(5.0, 0.9), (1.0, 0.3), (3.0, 0.7)] + result = pareto_frontier(points) + # Sorted by x: index 1 (1.0, 0.3), index 2 (3.0, 0.7), index 0 (5.0, 0.9) + # All form ascending y staircase → all on frontier + assert 0 in result + assert 1 in result + assert 2 in result + + def test_large_dominated_set(self): + # One clearly dominant point + many dominated + points = [(1.0, 1.0)] + [(float(i), 0.1) for i in range(2, 20)] + result = pareto_frontier(points) + assert 0 in result + assert len(result) == 1 # all others have higher x and lower y diff --git a/tests/test_pure_helpers.py b/tests/test_pure_helpers.py new file mode 100644 index 0000000..0e03948 --- /dev/null +++ b/tests/test_pure_helpers.py @@ -0,0 +1,511 @@ +"""Pure-function tests for supercompact lib helpers. + +Covers: + lib/parser.py — _is_user_message, extract_text, Turn + lib/fitness.py — _extract_vocab, _idf, FitnessResult.f1 + lib/scorer.py — _format_instruct, _last_token_pool + lib/types.py — build_query + lib/dedup.py — SuffixAutomaton + +All pure functions — no file system, network, or GPU access. +""" + +from __future__ import annotations + +import sys +from collections import Counter +from pathlib import Path + +import pytest +import torch + +# Ensure lib is importable when running from repo root +sys.path.insert(0, str(Path(__file__).resolve().parent.parent)) + +from lib.parser import Turn, _is_user_message, extract_text +from lib.fitness import FitnessResult, _extract_vocab, _idf +from lib.scorer import _format_instruct, _last_token_pool +from lib.types import build_query, ScoredTurn +from lib.dedup import SuffixAutomaton + + +# --------------------------------------------------------------------------- +# Turn dataclass +# --------------------------------------------------------------------------- + +class TestTurn: + def test_default_lines_empty(self): + t = Turn(kind="user") + assert t.lines == [] + + def test_append_adds_record(self): + t = Turn(kind="system") + t.append({"type": "assistant"}) + assert len(t.lines) == 1 + + def test_append_multiple(self): + t = Turn(kind="user") + t.append({"a": 1}) + t.append({"b": 2}) + assert len(t.lines) == 2 + + def test_kind_stored(self): + t = Turn(kind="system", index=3) + assert t.kind == "system" + assert t.index == 3 + + +# --------------------------------------------------------------------------- +# _is_user_message +# --------------------------------------------------------------------------- + +class TestIsUserMessage: + def test_plain_string_content(self): + record = {"type": "user", "message": {"content": "hello"}} + assert _is_user_message(record) is True + + def test_wrong_type_returns_false(self): + record = {"type": "assistant", "message": {"content": "hi"}} + assert _is_user_message(record) is False + + def test_tool_result_block_returns_false(self): + record = { + "type": "user", + "message": { + "content": [{"type": "tool_result", "content": "output"}] + }, + } + assert _is_user_message(record) is False + + def test_text_block_returns_true(self): + record = { + "type": "user", + "message": { + "content": [{"type": "text", "text": "hi there"}] + }, + } + assert _is_user_message(record) is True + + def test_source_tool_assistant_uuid_returns_false(self): + record = { + "type": "user", + "sourceToolAssistantUUID": "abc-123", + "message": {"content": "injected"}, + } + assert _is_user_message(record) is False + + def test_missing_type_returns_false(self): + assert _is_user_message({}) is False + + def test_empty_content_list_returns_true(self): + # An empty list has no tool_result blocks + record = {"type": "user", "message": {"content": []}} + assert _is_user_message(record) is True + + def test_mixed_blocks_without_tool_result_returns_true(self): + record = { + "type": "user", + "message": { + "content": [ + {"type": "text", "text": "question"}, + {"type": "image", "source": {}}, + ] + }, + } + assert _is_user_message(record) is True + + +# --------------------------------------------------------------------------- +# extract_text +# --------------------------------------------------------------------------- + +def _make_turn(records: list[dict]) -> Turn: + t = Turn(kind="system") + for r in records: + t.append(r) + return t + + +class TestExtractText: + def test_string_content(self): + t = _make_turn([{"message": {"content": "hello world"}}]) + assert "hello world" in extract_text(t) + + def test_text_block(self): + t = _make_turn([ + {"message": {"content": [{"type": "text", "text": "foo bar"}]}} + ]) + assert "foo bar" in extract_text(t) + + def test_thinking_block(self): + t = _make_turn([ + {"message": {"content": [{"type": "thinking", "thinking": "I think..."}]}} + ]) + assert "I think..." in extract_text(t) + + def test_tool_use_block(self): + t = _make_turn([ + { + "message": { + "content": [ + { + "type": "tool_use", + "name": "Bash", + "input": {"command": "ls -la"}, + } + ] + } + } + ]) + result = extract_text(t) + assert "tool_use: Bash" in result + assert "command" in result + assert "ls -la" in result + + def test_tool_result_string_content(self): + t = _make_turn([ + { + "message": { + "content": [ + {"type": "tool_result", "content": "output text"} + ] + } + } + ]) + assert "output text" in extract_text(t) + + def test_tool_result_list_content(self): + t = _make_turn([ + { + "message": { + "content": [ + { + "type": "tool_result", + "content": [{"type": "text", "text": "nested text"}], + } + ] + } + } + ]) + assert "nested text" in extract_text(t) + + def test_empty_turn(self): + t = Turn(kind="system") + assert extract_text(t) == "" + + def test_multiple_records_joined(self): + t = _make_turn([ + {"message": {"content": "first"}}, + {"message": {"content": "second"}}, + ]) + result = extract_text(t) + assert "first" in result + assert "second" in result + + def test_long_tool_input_truncated(self): + long_val = "x" * 600 + t = _make_turn([ + { + "message": { + "content": [ + { + "type": "tool_use", + "name": "Write", + "input": {"content": long_val}, + } + ] + } + } + ]) + result = extract_text(t) + # Value is truncated to 500 chars + "..." + assert "..." in result + assert len(result) < 600 + + +# --------------------------------------------------------------------------- +# _extract_vocab +# --------------------------------------------------------------------------- + +class TestExtractVocab: + def test_basic_word(self): + vocab = _extract_vocab("hello world") + # 'hello' and 'world' are 5 chars — both >=4 so both included + assert "hello" in vocab + assert "world" in vocab + + def test_short_words_excluded(self): + vocab = _extract_vocab("a the is it") + assert len(vocab) == 0 + + def test_lowercased(self): + vocab = _extract_vocab("FooBar") + # "FooBar" matches _WORD_RE since len>=4, stored lowercased + assert "foobar" in vocab + + def test_file_path_extracted(self): + vocab = _extract_vocab("/home/user/projects/myfile.py") + path_keys = [k for k in vocab if "/" in k] + assert len(path_keys) > 0 + + def test_returns_counter(self): + result = _extract_vocab("python python") + assert isinstance(result, Counter) + + def test_empty_string(self): + assert _extract_vocab("") == Counter() + + def test_word_count_correct(self): + vocab = _extract_vocab("pytest pytest pytest") + assert vocab["pytest"] == 3 + + +# --------------------------------------------------------------------------- +# _idf +# --------------------------------------------------------------------------- + +class TestIdf: + def test_term_in_all_docs(self): + docs = [Counter({"word": 1}), Counter({"word": 2})] + result = _idf("word", docs, 2) + # df=2, total=2 → log(1 + 2/2) = log(2) + import math + assert abs(result - math.log(2)) < 1e-9 + + def test_term_in_no_docs(self): + docs = [Counter({"other": 1})] + assert _idf("missing", docs, 1) == 0.0 + + def test_term_in_one_of_two_docs(self): + import math + docs = [Counter({"rare": 1}), Counter({"other": 1})] + result = _idf("rare", docs, 2) + # df=1, total=2 → log(1 + 2) = log(3) + assert abs(result - math.log(3)) < 1e-9 + + def test_idf_increases_with_rarity(self): + docs = [Counter({"common": 1})] * 5 + [Counter({"rare": 1})] + common_idf = _idf("common", docs, 6) + rare_idf = _idf("rare", docs, 6) + assert rare_idf > common_idf + + def test_total_docs_zero_guard(self): + # df=0 path returns 0, even with total_docs=0 + assert _idf("anything", [], 0) == 0.0 + + +# --------------------------------------------------------------------------- +# FitnessResult.f1 +# --------------------------------------------------------------------------- + +class TestFitnessResultF1: + def _make_result(self, recall: float, compression: float) -> FitnessResult: + return FitnessResult( + method="test", + recall=recall, + speed_s=0.0, + compression=compression, + budget=1000, + total_tokens=1000, + kept_tokens=int(compression * 1000), + prefix_turns=10, + suffix_turns=5, + suffix_vocab_size=50, + scored_count=8, + kept_scored=4, + dropped_scored=4, + ) + + def test_perfect_recall_zero_compression_gives_zero_f1(self): + r = self._make_result(recall=1.0, compression=1.0) + # compression_eff = 0, recall = 1 → harmonic mean = 0 + assert r.f1 == 0.0 + + def test_zero_recall_perfect_compression_gives_zero_f1(self): + r = self._make_result(recall=0.0, compression=0.0) + # compression_eff = 1, recall = 0 → harmonic mean = 0 + assert r.f1 == 0.0 + + def test_balanced_gives_nonzero_f1(self): + r = self._make_result(recall=0.8, compression=0.5) + # compression_eff = 0.5 + expected = 2 * 0.8 * 0.5 / (0.8 + 0.5) + assert abs(r.f1 - expected) < 1e-9 + + def test_both_zero_gives_zero(self): + r = self._make_result(recall=0.0, compression=1.0) + # recall=0, compression_eff=0 → denom=0 → returns 0 + assert r.f1 == 0.0 + + def test_f1_between_zero_and_one(self): + for recall in [0.2, 0.5, 0.9]: + for comp in [0.1, 0.5, 0.8]: + r = self._make_result(recall=recall, compression=comp) + assert 0.0 <= r.f1 <= 1.0 + + +# --------------------------------------------------------------------------- +# _format_instruct +# --------------------------------------------------------------------------- + +class TestFormatInstruct: + def test_basic_format(self): + result = _format_instruct("Find relevant docs", "some text") + assert result == "Instruct: Find relevant docs\nQuery: some text" + + def test_empty_instruction(self): + result = _format_instruct("", "text") + assert result == "Instruct: \nQuery: text" + + def test_empty_text(self): + result = _format_instruct("instruction", "") + assert result == "Instruct: instruction\nQuery: " + + def test_newline_preserved_in_text(self): + result = _format_instruct("instr", "line1\nline2") + assert "line1\nline2" in result + + def test_returns_string(self): + assert isinstance(_format_instruct("a", "b"), str) + + +# --------------------------------------------------------------------------- +# _last_token_pool +# --------------------------------------------------------------------------- + +class TestLastTokenPool: + def test_right_padded_single(self): + # Sequence [token, pad]: last real token is at index 0 + hidden = torch.tensor([[[1.0, 2.0], [0.0, 0.0]]]) # (1, 2, 2) + mask = torch.tensor([[1, 0]]) # attend to index 0 only + result = _last_token_pool(hidden, mask) + assert result.shape == (1, 2) + assert torch.allclose(result, torch.tensor([[1.0, 2.0]])) + + def test_right_padded_both_real(self): + # Two real tokens, no padding + hidden = torch.tensor([[[1.0, 0.0], [3.0, 4.0]]]) # (1, 2, 2) + mask = torch.tensor([[1, 1]]) + result = _last_token_pool(hidden, mask) + # Last real token is at index 1 + assert torch.allclose(result, torch.tensor([[3.0, 4.0]])) + + def test_left_padded_single(self): + # Left padding: [pad, token] → left_padding condition triggers → use last position + hidden = torch.tensor([[[0.0, 0.0], [5.0, 6.0]]]) # (1, 2, 2) + mask = torch.tensor([[0, 1]]) + # left_padding: mask[:, -1].sum() == batch_size → True + result = _last_token_pool(hidden, mask) + assert torch.allclose(result, torch.tensor([[5.0, 6.0]])) + + def test_batch_right_padded(self): + # Batch of 2, different sequence lengths + hidden = torch.tensor([ + [[1.0, 0.0], [2.0, 0.0], [0.0, 0.0]], # len=2, last real at idx 1 + [[3.0, 0.0], [4.0, 0.0], [5.0, 0.0]], # len=3, last real at idx 2 + ]) # (2, 3, 2) + mask = torch.tensor([[1, 1, 0], [1, 1, 1]]) + result = _last_token_pool(hidden, mask) + assert result.shape == (2, 2) + assert torch.allclose(result[0], torch.tensor([2.0, 0.0])) + assert torch.allclose(result[1], torch.tensor([5.0, 0.0])) + + def test_output_shape(self): + hidden = torch.randn(4, 8, 16) + mask = torch.ones(4, 8, dtype=torch.long) + result = _last_token_pool(hidden, mask) + assert result.shape == (4, 16) + + +# --------------------------------------------------------------------------- +# build_query +# --------------------------------------------------------------------------- + +class TestBuildQuery: + def _user_turn(self, text: str) -> Turn: + t = Turn(kind="user") + t.append({"message": {"content": text}}) + return t + + def test_single_turn(self): + turns = [self._user_turn("what is this?")] + result = build_query(turns) + assert "what is this?" in result + + def test_uses_last_three_turns(self): + turns = [self._user_turn(f"msg{i}") for i in range(5)] + result = build_query(turns) + # Should contain msgs 2, 3, 4 but not 0 or 1 + assert "msg4" in result + assert "msg3" in result + assert "msg2" in result + + def test_truncated_at_max_chars(self): + long_text = "x" * 5000 + turns = [self._user_turn(long_text)] + result = build_query(turns, max_chars=4000) + assert len(result) <= 4000 + + def test_empty_turns(self): + result = build_query([]) + assert result == "" + + def test_returns_string(self): + turns = [self._user_turn("hello")] + assert isinstance(build_query(turns), str) + + +# --------------------------------------------------------------------------- +# SuffixAutomaton +# --------------------------------------------------------------------------- + +class TestSuffixAutomaton: + def test_build_and_propagate(self): + sa = SuffixAutomaton() + for i, c in enumerate("abcabc"): + sa.extend(c, i) + sa.propagate_counts() + # Should have states + assert len(sa.states) > 1 + + def test_match_repeated_length_all_repeated(self): + sa = SuffixAutomaton() + text = "aaaa" + for i, c in enumerate(text): + sa.extend(c, i) + sa.propagate_counts() + lengths = sa.match_repeated_length(text) + assert len(lengths) == len(text) + + def test_match_repeated_length_no_repeats(self): + sa = SuffixAutomaton() + text = "abcd" + for i, c in enumerate(text): + sa.extend(c, i) + sa.propagate_counts() + lengths = sa.match_repeated_length(text) + # Unique chars — nothing repeats more than once + assert all(l == 0 for l in lengths) + + def test_match_repeated_length_partial_repeat(self): + sa = SuffixAutomaton() + # "xyzxyz" — "xyz" appears twice + text = "xyzxyz" + for i, c in enumerate(text): + sa.extend(c, i) + sa.propagate_counts() + lengths = sa.match_repeated_length(text) + # The second "xyz" (positions 3-5) should have repeat lengths > 0 + assert any(l > 0 for l in lengths[3:]) + + def test_empty_string(self): + sa = SuffixAutomaton() + sa.propagate_counts() + lengths = sa.match_repeated_length("") + assert lengths == [] + + def test_initial_state_exists(self): + sa = SuffixAutomaton() + assert len(sa.states) == 1 + assert sa.states[0].len == 0 + assert sa.states[0].link == -1