Skip to content

Add repeatable prompt and diagnostic benchmarks - #208

Merged
jiminu merged 1 commit into
mainfrom
perf/prompt-diagnostic-benchmarks
Sep 15, 2026
Merged

jiminu merged 1 commit into
mainfrom
perf/prompt-diagnostic-benchmarks

Conversation

@jiminu

@jiminu jiminu commented Sep 15, 2026

Copy link
Copy Markdown
Owner

The existing benchmark measured shell initialization but stopped before a prompt was displayed, and did not cover status or doctor. Add optional --prompt and --diagnostics measurements to the existing base/full modes and TSV output.

  • Measure first-prompt and command-to-prompt latency in an empty directory and the source repository using Python's standard-library PTY support. Numbered markers reject editing redraws; graceful shutdown releases history locks before the next sample.
  • Install configuration in a temporary HOME before timing diagnostics, report the observed installation and exit codes, and fail if a sampled exit code changes. Preserve WSL platform context and bound mise configuration discovery to avoid ancestor settings affecting results.
  • Include the missing history module in the benchmark fixture and document measurement scope, dependencies, and limits.

Validation:

  • bash scripts/check.sh passed locally on macOS: all 21 suites, Bash/Zsh syntax, ShellCheck, and shell formatting.
  • Regression checks cover numeric metric output, prompt sequencing, timeout/closed-shell behavior, isolated environment settings, and normal exit hooks. Code review completed.
  • Full-mode local run: 30 first-prompt samples per scenario, 90 command samples per scenario, and 30 samples per diagnostic command. p50: first prompt 90–109 ms, command-to-prompt 31–49 ms, status 625 ms, doctor 640 ms. These are machine-specific observations, not performance budgets.
  • The diagnostic fixture uses temporary pinned shell tools plus host package/PATH inventories. Its cask inventory encountered missing tap trust metadata in the isolated HOME; timings include that error path and do not represent a freshly provisioned complete environment.

@jiminu
jiminu merged commit b5841d5 into main Sep 15, 2026
7 checks passed
@jiminu
jiminu deleted the perf/prompt-diagnostic-benchmarks branch September 15, 2026 08:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant