From fafddfe4c146b970ed4cba7b7b7839b28615348f Mon Sep 17 00:00:00 2001 From: Dmitriy Date: Sat, 30 May 2026 16:36:02 +0300 Subject: [PATCH 1/3] docs: comprehensive documentation audit and enhancement MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Closes 20 of 22 documentation-audit items, restructuring the docs around the Diátaxis framework and adding the standard open-source community and project-health files. Content & structure (Diátaxis): - Add Explanation quadrant: 15 site/explanation/*.md analyzer concept pages - Add How-to quadrant: 5 task-oriented guides (bare repo, monorepo, CI, MCP client, reading a burndown chart) - Add Reference glossary (site/glossary.md) - Add Release notes section (index + unreleased) wired into nav - Migrate page titles to sentence case across the site Community & project health: - CONTRIBUTING.md, CODE_OF_CONDUCT.md, SECURITY.md, SUPPORT.md, GOVERNANCE.md, THIRD_PARTY_NOTICES.md - GitHub issue templates + pull request template - README Maintainers / Contributing / License / Security / Support sections Architecture decisions: - ADRs (MADR): record-architecture-decisions, two-binary-split, libgit2-via-cgo; exposed in nav via site/adr -> ../docs/adr symlink Changelog: - Keep a Changelog 1.1.0 proposal as CHANGELOG.md.proposed.md (original CHANGELOG.md intentionally left untouched) - Thin site/contributing/changelog.md to a pointer AI-agent discoverability: - llms.txt and llms-full.txt Docs CI & linting: - .github/workflows/docs.yml jobs plus markdownlint, Vale, cspell, and lychee link-check configs mkdocs build --strict passes; no Go source touched. Co-Authored-By: Claude Opus 4.8 (1M context) --- .github/ISSUE_TEMPLATE/bug_report.md | 36 + .github/ISSUE_TEMPLATE/feature_request.md | 20 + .github/PULL_REQUEST_TEMPLATE.md | 29 + .github/workflows/docs.yml | 79 + .lycheeignore | 32 + .markdownlint.jsonc | 35 + .vale.ini | 30 + CHANGELOG.md.proposed.md | 165 + CODE_OF_CONDUCT.md | 26 + CONTRIBUTING.md | 76 + GOVERNANCE.md | 112 + README.md | 46 +- SECURITY.md | 45 + SUPPORT.md | 41 + THIRD_PARTY_NOTICES.md | 30 + cspell.json | 88 + .../adr/0001-record-architecture-decisions.md | 68 + .../0002-two-binary-split-uast-codefang.md | 63 + docs/adr/0003-libgit2-via-cgo.md | 65 + llms-full.txt | 2974 +++++++++++++++++ llms.txt | 44 + mkdocs.yml | 74 +- site/adr | 1 + site/analyzers/anomaly.md | 107 +- site/analyzers/burndown.md | 136 +- site/analyzers/clones.md | 116 +- site/analyzers/cohesion.md | 115 +- site/analyzers/comments.md | 66 +- site/analyzers/complexity.md | 97 +- site/analyzers/couples.md | 104 +- site/analyzers/developers.md | 116 +- site/analyzers/file-history.md | 70 +- site/analyzers/halstead.md | 78 +- site/analyzers/imports.md | 82 +- site/analyzers/index.md | 12 +- site/analyzers/quality.md | 64 +- site/analyzers/sentiment.md | 185 +- site/analyzers/shotness.md | 200 +- site/analyzers/typos.md | 85 +- site/architecture/overview.md | 30 +- site/architecture/streaming-pipeline.md | 54 +- site/architecture/uast.md | 38 +- site/contributing/changelog.md | 99 +- site/contributing/code-of-conduct.md | 23 +- site/contributing/index.md | 52 +- site/examples/index.md | 12 +- site/explanation/anomaly.md | 93 + site/explanation/burndown.md | 97 + site/explanation/clones.md | 107 + site/explanation/cohesion.md | 108 + site/explanation/comments.md | 59 + site/explanation/complexity.md | 84 + site/explanation/couples.md | 99 + site/explanation/developers.md | 102 + site/explanation/file-history.md | 68 + site/explanation/halstead.md | 69 + site/explanation/imports.md | 67 + site/explanation/quality.md | 54 + site/explanation/sentiment.md | 161 + site/explanation/shotness.md | 113 + site/explanation/typos.md | 73 + site/getting-started/first-analysis.md | 30 +- site/getting-started/installation.md | 12 +- site/getting-started/quickstart.md | 18 +- site/glossary.md | 161 + site/guide/cli-reference.md | 22 +- site/guide/configuration.md | 16 +- site/guide/data-analytics.md | 64 +- site/guide/output-formats.md | 10 +- site/how-to/analyze-a-bare-repo.md | 70 + site/how-to/analyze-a-monorepo.md | 79 + site/how-to/attach-an-mcp-client.md | 88 + site/how-to/read-a-burndown-chart.md | 79 + site/how-to/run-in-ci.md | 95 + site/index.md | 14 +- site/integrations/ai-agents.md | 58 +- site/integrations/docker-and-actions.md | 48 +- site/integrations/mcp.md | 22 +- site/operations/large-scale-scanning.md | 28 +- site/operations/observability.md | 62 +- site/release-notes/index.md | 18 + site/release-notes/unreleased.md | 99 + 82 files changed, 6763 insertions(+), 1804 deletions(-) create mode 100644 .github/ISSUE_TEMPLATE/bug_report.md create mode 100644 .github/ISSUE_TEMPLATE/feature_request.md create mode 100644 .github/PULL_REQUEST_TEMPLATE.md create mode 100644 .lycheeignore create mode 100644 .markdownlint.jsonc create mode 100644 .vale.ini create mode 100644 CHANGELOG.md.proposed.md create mode 100644 CODE_OF_CONDUCT.md create mode 100644 CONTRIBUTING.md create mode 100644 GOVERNANCE.md create mode 100644 SECURITY.md create mode 100644 SUPPORT.md create mode 100644 THIRD_PARTY_NOTICES.md create mode 100644 cspell.json create mode 100644 docs/adr/0001-record-architecture-decisions.md create mode 100644 docs/adr/0002-two-binary-split-uast-codefang.md create mode 100644 docs/adr/0003-libgit2-via-cgo.md create mode 100644 llms-full.txt create mode 100644 llms.txt create mode 120000 site/adr create mode 100644 site/explanation/anomaly.md create mode 100644 site/explanation/burndown.md create mode 100644 site/explanation/clones.md create mode 100644 site/explanation/cohesion.md create mode 100644 site/explanation/comments.md create mode 100644 site/explanation/complexity.md create mode 100644 site/explanation/couples.md create mode 100644 site/explanation/developers.md create mode 100644 site/explanation/file-history.md create mode 100644 site/explanation/halstead.md create mode 100644 site/explanation/imports.md create mode 100644 site/explanation/quality.md create mode 100644 site/explanation/sentiment.md create mode 100644 site/explanation/shotness.md create mode 100644 site/explanation/typos.md create mode 100644 site/glossary.md create mode 100644 site/how-to/analyze-a-bare-repo.md create mode 100644 site/how-to/analyze-a-monorepo.md create mode 100644 site/how-to/attach-an-mcp-client.md create mode 100644 site/how-to/read-a-burndown-chart.md create mode 100644 site/how-to/run-in-ci.md create mode 100644 site/release-notes/index.md create mode 100644 site/release-notes/unreleased.md diff --git a/.github/ISSUE_TEMPLATE/bug_report.md b/.github/ISSUE_TEMPLATE/bug_report.md new file mode 100644 index 00000000..a1aac28e --- /dev/null +++ b/.github/ISSUE_TEMPLATE/bug_report.md @@ -0,0 +1,36 @@ +--- +name: Bug report +about: Report a defect in Codefang +title: 'bug: ' +labels: bug +--- + + + +### Codefang version + + + +### Operating system and architecture + + + +### Steps to reproduce + + + +1. +2. +3. + +### Expected behavior + + + +### Actual behavior + + + +### Repository + + diff --git a/.github/ISSUE_TEMPLATE/feature_request.md b/.github/ISSUE_TEMPLATE/feature_request.md new file mode 100644 index 00000000..71929f81 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/feature_request.md @@ -0,0 +1,20 @@ +--- +name: Feature request +about: Propose a new analyzer, output format, or integration +title: 'feat: ' +labels: enhancement +--- + + + +### Use case + + + +### Expected behavior + + + +### Alternatives considered + + diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md new file mode 100644 index 00000000..662c547a --- /dev/null +++ b/.github/PULL_REQUEST_TEMPLATE.md @@ -0,0 +1,29 @@ + + +## Scope + + + +## Test matrix + + + +## Link to FRD/BUG + + + +Closes # + +## Breaking changes + + + +--- + +### Before opening this PR + +- [ ] All tests pass locally (`make test`) +- [ ] Linting is clean (`make lint`) +- [ ] Dead code analysis passes (`make deadcode`) +- [ ] Commit messages follow conventional commits +- [ ] Documentation is updated if needed diff --git a/.github/workflows/docs.yml b/.github/workflows/docs.yml index 60653fc0..c40e28e8 100644 --- a/.github/workflows/docs.yml +++ b/.github/workflows/docs.yml @@ -11,7 +11,13 @@ on: branches: [main] paths: - 'site/**' + - '**/*.md' - 'mkdocs.yml' + - '.markdownlint.jsonc' + - '.vale.ini' + - 'cspell.json' + - '.lycheeignore' + - '.github/workflows/docs.yml' workflow_dispatch: permissions: @@ -68,3 +74,76 @@ jobs: - name: Deploy to GitHub Pages id: deployment uses: actions/deploy-pages@v4 + + markdownlint: + name: Lint Markdown (markdownlint) + runs-on: ubuntu-latest + steps: + - name: Checkout code + uses: actions/checkout@v4 + + - name: Run markdownlint-cli2 + uses: DavidAnson/markdownlint-cli2-action@v16 + with: + globs: | + **/*.md + !build/** + !vendor/** + !third_party/** + !node_modules/** + + vale: + name: Prose style (Vale, advisory) + runs-on: ubuntu-latest + # Advisory for the first iteration: surface findings without blocking the + # workflow until the R-STYLE-02 sentence-case migration lands. + continue-on-error: true + steps: + - name: Checkout code + uses: actions/checkout@v4 + + - name: Run Vale (Microsoft + Google packs) + uses: errata-ai/vale-action@reviewdog + with: + fail_on_error: false + reporter: github-pr-check + files: '["site", "README.md", "CONTRIBUTING.md", "docs"]' + env: + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + + cspell: + name: Spell check (cspell, advisory) + runs-on: ubuntu-latest + # Advisory for the first iteration: project jargon is seeded in cspell.json, + # but the dictionary is still maturing, so do not block the workflow yet. + continue-on-error: true + steps: + - name: Checkout code + uses: actions/checkout@v4 + + - name: Run cspell + uses: streetsidesoftware/cspell-action@v6 + with: + config: cspell.json + files: '**/*.md' + incremental_files_only: false + + lychee: + name: Check links (lychee) + runs-on: ubuntu-latest + steps: + - name: Checkout code + uses: actions/checkout@v4 + + - name: Run lychee link checker + uses: lycheeverse/lychee-action@v2 + with: + args: >- + --no-progress + --verbose + --max-concurrency 8 + --exclude-mail + './**/*.md' + fail: true + env: + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} diff --git a/.lycheeignore b/.lycheeignore new file mode 100644 index 00000000..1aeb000e --- /dev/null +++ b/.lycheeignore @@ -0,0 +1,32 @@ +# lychee link-checker ignore list (one regex per line). +# Reference: https://github.com/lycheeverse/lychee#ignoring-links +# Patterns here are known-unreachable or context-dependent links that would +# otherwise produce false negatives in CI. + +# Localhost / loopback endpoints used in MCP and observability examples. +^https?://localhost +^https?://127\.0\.0\.1 +^https?://0\.0\.0\.0 +^https?://\[::1\] + +# Private / link-local address ranges from large-scale-scanning examples. +^https?://10\. +^https?://192\.168\. +^https?://172\.(1[6-9]|2[0-9]|3[01])\. + +# Example/placeholder hosts. +example\.com +example\.org +your-org +<.*> + +# The project's own GitHub Pages site is built and published by this same +# workflow; on a PR the new pages may 404 until deploy completes. +sumatoshi-tech\.github\.io + +# pkg.go.dev rate-limits aggressive crawlers and 404s modules until first fetch. +pkg\.go\.dev + +# mailto and tel links — checked separately via --exclude-mail in the job. +^mailto: +^tel: diff --git a/.markdownlint.jsonc b/.markdownlint.jsonc new file mode 100644 index 00000000..a9f2f764 --- /dev/null +++ b/.markdownlint.jsonc @@ -0,0 +1,35 @@ +// markdownlint-cli2 configuration for codefang docs. +// Rule reference: https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md +// Tuned for the existing site/ + root *.md + docs/adr/ corpus so the first CI run +// is signal, not a wall of false positives. +{ + "default": true, + + // MD013 line-length: docs use long prose lines and wide tables/links. + // Disable the hard wrap requirement rather than enforce an arbitrary column. + "MD013": false, + + // MD033 inline HTML: analyzer pages and the README use raw HTML + // (

, , admonition blocks, grid cards). + "MD033": false, + + // MD041 first-line-h1: MkDocs pages sometimes open with front matter or + // an attr_list/admonition block instead of a top-level heading. + "MD041": false, + + // MD024 duplicate headings: Keep a Changelog repeats Added/Changed/Fixed + // across versions, and ADRs repeat section names. Allow when not siblings. + "MD024": { "siblings_only": true }, + + // MD046 code-block-style: allow both fenced and indented blocks. + "MD046": false, + + // MD007 unordered-list indentation: MkDocs material renders 4-space nesting. + "MD007": { "indent": 4 }, + + // MD036 emphasis-as-heading: the README uses bold lead-ins as pseudo-headings. + "MD036": false, + + // MD034 bare URLs: badges and reference links occasionally use bare URLs. + "MD034": false +} diff --git a/.vale.ini b/.vale.ini new file mode 100644 index 00000000..b3355ed7 --- /dev/null +++ b/.vale.ini @@ -0,0 +1,30 @@ +; Vale configuration for codefang docs. +; Reference: https://vale.sh/docs/topics/config/ +; The Microsoft and Google style packs are pulled by the CI action's +; `vale sync` step (PackageStyles). Advisory for the first iteration: +; the CI job runs with continue-on-error so findings are visible but do +; not block the workflow until the R-STYLE-02 sentence-case migration lands. + +StylesPath = .github/styles + +MinAlertLevel = suggestion + +; Pull the Microsoft and Google rule packs. +Packages = Microsoft, Google + +[*.md] +BasedOnStyles = Vale, Microsoft, Google + +; Project jargon and proper nouns Vale's spelling rule would otherwise flag. +Vale.Terms = NO + +; Sentence-case headings are enforced after the R-STYLE-02 migration. +Google.Headings = YES +Microsoft.Headings = suggestion + +; Allow contractions and first-person — the README voice is intentionally casual. +Google.Contractions = NO +Google.FirstPerson = NO + +; Allow exclamation points used sparingly in the marketing-flavoured README. +Google.Exclamation = NO diff --git a/CHANGELOG.md.proposed.md b/CHANGELOG.md.proposed.md new file mode 100644 index 00000000..e33e53f2 --- /dev/null +++ b/CHANGELOG.md.proposed.md @@ -0,0 +1,165 @@ + + + + + + + + +# Changelog + +All notable changes to the Codefang project are documented in this file. + +The format is based on [Keep a Changelog 1.1.0](https://keepachangelog.com/en/1.1.0/), +and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). + +## [Unreleased] + +### Added + +- Temporal anomaly detection analyzer (`history/anomaly`) using Z-score + statistical analysis over sliding windows for detecting sudden quality + degradation in commit history. +- TimeSeries output format (`--format timeseries`) that merges all analyzer + outputs into a single chronologically ordered JSON array. +- Plot output format (`--format plot`) generating interactive HTML charts via + go-echarts for visual analysis. +- NDJSON output format (`--format ndjson`) emitting one JSON line per analyzer + result (with an optional metadata line) for streaming ingestion into + warehouses such as ClickHouse. +- MCP server (`codefang mcp`) exposing analysis capabilities as tools for AI + agents via the Model Context Protocol (stdio transport, JSON-RPC 2.0). Three + tools: `codefang_analyze`, `uast_parse`, `codefang_history`. +- Docker support with a multi-stage Dockerfile and a `debian:bookworm-slim` + runtime image. Runs as a non-root `codefang` user by default. +- GitHub Actions integration (`action.yml`) for automated code-quality checks + in CI pipelines with configurable analyzers, formats, and quality gates. +- OpenTelemetry observability with distributed tracing, RED metrics (rate, + errors, duration), and structured logging with trace-context injection. + Supports Jaeger, Prometheus, and OTLP collectors. +- Streaming pipeline for bounded-memory processing of large repositories via + chunk-based processing with hibernate/boot cycles. +- Double-buffered chunk pipelining that overlaps pipeline prefetch with + analyzer consumption for higher throughput on multi-chunk workloads. +- Checkpointing with automatic save after each processed chunk and crash + recovery via the `--resume` flag. Checkpoint format v2 preserves aggregator + spill state so resumed runs produce identical output. +- Configuration system (`.codefang.yaml`) with file, environment-variable, and + CLI-flag support. Merge priority: CLI > env > file > defaults. +- Large-scale scanning support for fleet analysis of thousands of repositories, + with bare-repo support, GNU Parallel and Kubernetes orchestration patterns, + and data-warehouse loading guides (Athena, Snowflake, Spark). +- Deep context propagation that eliminates `context.Background()` calls in + production hot paths for end-to-end tracing. +- Attribute-filter span processor that enforces an allow-list of attribute key + prefixes to prevent PII leakage to collectors. +- Health-check endpoints (`/healthz`, `/readyz`, `/metrics`) for server-mode + deployments. +- Incremental scanning with the `--since` flag, supporting Go durations, date + strings, and RFC 3339 timestamps. +- Memory-budget auto-tuning via the `--memory-budget` flag with automatic chunk + size calculation. +- Watchdog stall detection with a configurable `worker_timeout` for identifying + hung workers. +- `--include-vendored` flag (bool, default `false`) to re-include paths detected + as vendored by enry/Linguist (`vendor/`, `node_modules/`, `third_party/`, + `testdata/`, `dist/`, minified bundles, and more). +- `--include-generated` flag (bool, default `false`) to re-include + auto-generated files (`*.pb.go`, `zz_generated_*.go`, `*_pb2.py`, `*.min.js`, + and content-header markers such as `DO NOT EDIT`, `Code generated`, + `@generated`). +- `--extra-excluded-prefixes` flag (strings, default `[]`) for additional UNIX + path prefixes to exclude for ecosystems enry does not know (for example + `.venv/`, `target/`, `.gradle/`). +- `source_file` field on every function record (relative path, for example + `pkg/kubelet/kubelet.go`) for `static/complexity`, `static/halstead`, + `static/cohesion`, and `static/comments`. +- `language` field on every function record (for example `go`, `bash`) for the + same static analyzers. +- `directory` field on every function record (for example `pkg/kubelet`) for the + same static analyzers. +- `start_time` and `end_time` fields (RFC 3339) on every history time-series + tick for `history/sentiment`, `history/anomaly`, `history/quality`, + `history/devs`, and `history/file-history`. +- `email` field on developer records, plus `primary_dev_email` / + `secondary_dev_email` on bus-factor records and `developer1_email` / + `developer2_email` on developer-coupling records, via a new + `SplitIdentity` helper. +- Top-level `metadata` section in the report envelope with `repo_path`, + `repo_name`, `analyzed_at` (RFC 3339), and `codefang_version`. +- Per-analyzer `schema` manifest in each analyzer result with field `type`, + `grain`, and `description` for automated ETL generation across all 17 + analyzers. +- `clone_type_distribution` computed from the full pair population rather than + the capped 1,000-pair sample, so Type-1/2/3 percentages are accurate on large + codebases. + +### Changed + +- Complete rewrite from the original `src-d/hercules` project, into a modern Go + codebase with idiomatic patterns, clean architecture, and comprehensive test + coverage. +- Split the tool into two binaries: `uast` (Universal AST parser using + Tree-sitter for 60+ languages) and `codefang` (analysis engine for static and + history analysis). Unix philosophy: small tools joined by pipes. +- Replaced the original Babelfish/bblfsh parser with a Tree-sitter-based UAST + that is faster, more reliable, and locally compiled, supporting 60+ + programming languages. +- Added a DSL-based UAST mapping layer with a custom domain-specific language + for transforming Tree-sitter ASTs into standardized UAST nodes. +- Vendored libgit2 (`third_party/libgit2`), compiled as a static library for + reproducible builds without external dependencies. +- Replaced all `log.Printf` and `fmt.Printf` calls with structured logging via + `log/slog`. Instance loggers are enforced by `sloglint`. +- Removed all backward-compatibility fallbacks in the quality, anomaly, + sentiment, and devs `ParseReportData` paths; formalized shotness shallow + extraction; and integrated VADER (GoVader) for real sentiment analysis. +- Default analysis output across both phases now excludes vendor and generated + files, matching the convention of mature multi-language analysers. Pass + `--include-vendored --include-generated` to restore the previous behaviour. + **Breaking change.** +- Made `--languages` cross-phase. Static analysis previously ignored the flag; + it now narrows both static and history phases through a single + `internal/analyzers/plumbing/langpath` source of truth, with fail-fast errors + on unknown language tokens. +- Pushed the `--languages` filter down into libgit2 via a new `cf_tree_diff_v2` + C ABI that forwards a pathspec to `git_diff_options.pathspec`, cutting tree-diff + work on polyglot repositories with a narrow filter (about 34% wall-time and + 36% `cgocall` CPU reduction on a synthetic fixture). +- Replaced the `FileContentAnalyzer` + `WalksAllFiles` marker-interface pattern + with explicit `StaticAnalyzer` (UAST) and `RawFileAnalyzer` (raw file) + hierarchies sharing a `FormattableAnalyzer` base, driven by explicit pipeline + stages. This enabled relative source paths and per-record language stamping. +- Changed `developers[].languages` from a map keyed by language name to a sorted + `[]LanguageStatsEntry` array so it can be UNNEST'd in columnar warehouses. + Empty language strings become `Other`. **Breaking change to output schema.** +- Changed `activity[].by_developer` from a `map[int]int` to a sorted + `[]DeveloperCommits` array of `{dev_id, commits}`. **Breaking change to output + schema.** +- Changed `file_contributors[].contributors` from a `map[int]LineStats` to a + sorted `[]ContributorEntry` array of `{dev_id, added, removed, changed}`. + **Breaking change to output schema.** +- Changed clone-pair `func_a` / `func_b` paths from absolute to relative. + +### Deprecated + +- `--skip-blacklist` is now a no-op (the new default already excludes vendor and + generated files); a Cobra deprecation warning fires when it is passed. +- `--blacklisted-prefixes` is superseded by `--extra-excluded-prefixes` + (identical semantics); a Cobra deprecation warning fires when it is passed. + +### Fixed + +- Fixed a data race in `internal/framework.PipelineSampler`: `t1Captured` was a + plain `bool` read by the sampler goroutine and written by the caller, causing + intermittent races under `go test -race`. It is now a `sync/atomic.Bool` using + `CompareAndSwap`, so at most one t1 heap profile is captured. Removed the + unused `t0Captured` field. + +### Removed + +- Removed `// FRD: specs/frds/FRD-...md` comments from all `.go` files; + `specs/` is gitignored, so those references broke for anyone cloning the repo. + Traceability now lives in FRDs and PR descriptions. + +[Unreleased]: https://github.com/Sumatoshi-tech/codefang/commits/main diff --git a/CODE_OF_CONDUCT.md b/CODE_OF_CONDUCT.md new file mode 100644 index 00000000..600972b3 --- /dev/null +++ b/CODE_OF_CONDUCT.md @@ -0,0 +1,26 @@ + + +# Code of conduct + +Codefang adopts the [Contributor Covenant 2.1](https://www.contributor-covenant.org/version/2/1/code_of_conduct/) +as its code of conduct. By participating in this project, you agree to uphold it. + +## Read the full text + +The complete, canonical code of conduct lives on the documentation site: + +**** + +The source for that page is +[`site/contributing/code-of-conduct.md`](site/contributing/code-of-conduct.md), +which is the canonical version. Edit it there. + +## Reporting and enforcement + +Instances of abusive, harassing, or otherwise unacceptable behavior may be +reported **privately** to the maintainer at ****. Do not +open a public GitHub issue, discussion, or pull request to report a violation — +a private report protects everyone involved while the matter is reviewed. + +All complaints will be reviewed and investigated promptly and fairly, and the +maintainer will respect the privacy and security of the reporter. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 00000000..6f812774 --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,76 @@ + + +# Contributing to Codefang + +Thanks for your interest in contributing. Whether you are fixing a bug, adding a +new analyzer, improving documentation, or proposing a feature, every +contribution makes Codefang better for the whole community. + +## Read the full guide + +The complete contributing guide is maintained on the documentation site: + +**** + +It covers everything you need to get started: + +- **Getting started** — prerequisites, fork, clone, and build. +- **Development workflow** — branch, write tests first, lint, run the race detector. +- **Code standards** — idiomatic Go, context propagation, structured logging, error handling. +- **Commit conventions** — [Conventional Commits](https://www.conventionalcommits.org/). +- **Pull request process** — the pre-PR checklist and the PR description template. +- **Reporting bugs** and **feature requests** — what to include so maintainers can act. + +The source for that page lives at +[`site/contributing/index.md`](site/contributing/index.md), which is the +canonical version. Edit it there. + +## Quick reference + +To build, test, and lint locally: + +```bash +make build # build all binaries (compiles vendored libgit2) +make test # run the full test suite +make lint # run golangci-lint and dead code analysis +``` + +## Versioning + +Codefang follows [Semantic Versioning](https://semver.org/) (`MAJOR.MINOR.PATCH`). +A release is cut by the GoReleaser pipeline in +[`.github/workflows/ci.yml`](.github/workflows/ci.yml), which runs on any pushed +`v*` tag (the workflow's `tags: ["v*"]` trigger), so the tag you choose is the +contract you ship. + +- **MAJOR** — a breaking change to the public surface (see below). +- **MINOR** — a backward-compatible, additive feature. +- **PATCH** — a bug fix, performance improvement, or internal refactor that does + not change the public surface. + +The **public surface** is the exported Go API (anything reachable through +`pkg/`), the CLI flags and commands, the output schemas of those commands, and +the configuration keys; changing or removing any of these in an incompatible way +is a breaking change and requires a MAJOR bump. + +## Contribution agreement + +Codefang has **no separate Contributor License Agreement (CLA)** and **does not +require a Developer Certificate of Origin (DCO) sign-off** (`git commit -s`) on +commits. There is no CLA bot or DCO check in the +[`.github/workflows/`](.github/workflows/) pipelines, and none is planned. + +Instead, contributions are accepted on an **inbound=outbound** basis under the +project's [Apache License 2.0](LICENSE). This follows the Apache-2.0 +contribution clause (§5): unless you state otherwise, any contribution you +intentionally submit for inclusion is provided under the same Apache-2.0 terms +that cover the project, with no additional conditions. By opening a pull +request, you confirm you have the right to submit the work under that license. + +If a maintainer ever adopts a CLA or starts requiring DCO sign-off, this section +and the relevant pipeline will be updated to say so. + +## Code of conduct + +This project follows a [Code of Conduct](https://sumatoshi-tech.github.io/codefang/contributing/code-of-conduct/). +By participating, you agree to uphold it. diff --git a/GOVERNANCE.md b/GOVERNANCE.md new file mode 100644 index 00000000..6f9e6859 --- /dev/null +++ b/GOVERNANCE.md @@ -0,0 +1,112 @@ + + +# Governance + +This document describes who decides what in Codefang, how those decisions are +made, and how the set of maintainers changes over time. It is an honest +description of the project as it stands: Codefang is led by a single +benevolent maintainer, with a clear path to growing the maintainer team as the +community grows. + +## Project leadership + +Codefang follows a benevolent-maintainer model. One maintainer holds final +authority over the project's direction, the code that ships, and the releases +that are cut. As of this writing, that maintainer is: + +- Dmytro Gajewski (`d.y.gaevskiy@gmail.com`) + +The current maintainer list is always the source of truth in the +[`MAINTAINERS`](MAINTAINERS) file at the repository root. When that file and +this document disagree, `MAINTAINERS` wins, and the disagreement is a bug to +fix here. + +## Roles + +- **Maintainer.** Has write access to the repository. Reviews and merges pull + requests, cuts releases, sets the roadmap, and has the final say on disputes. + Maintainers are listed in [`MAINTAINERS`](MAINTAINERS). +- **Contributor.** Anyone who has had a pull request merged. Contributors propose + changes, file issues, review other contributions, and help triage. No special + access is required to contribute. + +There is no separate committee or steering group. While the project has a single +maintainer, "the maintainers" and "the maintainer" mean the same person. + +## Decision process + +The maintainer is the decision-maker. The process below is what to expect in +practice, scaled to the size of the change. + +- **Small changes** (bug fixes, documentation, internal refactors that do not + change the public surface): the maintainer reviews the pull request and merges + it once it passes review and the checks in [`CONTRIBUTING.md`](CONTRIBUTING.md) + are green. +- **Significant changes** (new public Go API, breaking changes, new CLI flags or + output schemas, new dependencies): open an issue first to discuss the change + before opening a pull request. The maintainer decides whether to accept the + direction, then reviews the implementation. Record the rationale for a + notable decision as an ADR under [`docs/adr/`](docs/adr/). +- **Releases:** the maintainer decides when to cut a release and which version to + tag, following the Semantic Versioning policy documented in + [`CONTRIBUTING.md`](CONTRIBUTING.md). + +All change proposals go through a pull request and a review. The maintainer does +not merge their own substantial changes without giving the community a chance to +comment through the issue and pull-request process. + +## Disputes + +If you disagree with a decision, open an issue (or comment on the relevant pull +request) and make your case with evidence. The maintainer will respond and, +where reasonable, revisit the decision. While the project has a single +maintainer, that maintainer resolves disputes; once there is more than one +maintainer, disputes that cannot be settled by discussion are resolved by a +simple majority vote of the maintainers, with the longest-serving maintainer +breaking a tie. + +## Adding a maintainer + +The project actively wants to grow its maintainer team. A contributor becomes a +candidate for maintainer through: + +1. **Sustained contribution.** A track record of merged pull requests, helpful + reviews, and issue triage that shows good judgment about the codebase. +2. **Nomination.** An existing maintainer nominates the candidate by opening an + issue or pull request that proposes adding them to [`MAINTAINERS`](MAINTAINERS). +3. **Consensus.** The existing maintainers agree. While there is a single + maintainer, that maintainer's agreement is sufficient. Once there are several, + adding a maintainer uses lazy consensus: the nomination is approved unless an + existing maintainer objects with a reason within a reasonable review window. + +A new maintainer is added by a pull request that updates [`MAINTAINERS`](MAINTAINERS) +and grants repository write access. + +## Removing a maintainer + +A maintainer is removed from [`MAINTAINERS`](MAINTAINERS) in any of these cases: + +- **By their own request.** A maintainer may step down at any time. +- **By inactivity.** After a sustained period with no reviews, merges, or + participation, the remaining maintainers may remove the inactive maintainer by + the same consensus process used to add one. An emeritus note acknowledging + their contributions is encouraged. +- **By consensus.** In the rare case of conduct incompatible with the + [Code of Conduct](https://sumatoshi-tech.github.io/codefang/contributing/code-of-conduct/), + the remaining maintainers may remove a maintainer by consensus. + +Removal is recorded by a pull request that updates [`MAINTAINERS`](MAINTAINERS) +and revokes repository write access. + +## Changing this document + +This document is itself governed by the decision process above: propose changes +through a pull request, and the maintainers decide. Keep it honest — it must +always describe how the project actually operates, not how we wish it did. + +## See also + +- [`MAINTAINERS`](MAINTAINERS) — the authoritative maintainer list. +- [`CONTRIBUTING.md`](CONTRIBUTING.md) — how to file issues, propose changes, and + the versioning policy. +- [Code of Conduct](https://sumatoshi-tech.github.io/codefang/contributing/code-of-conduct/) — the behavior expected of everyone who participates. diff --git a/README.md b/README.md index 328765a3..22ce53ca 100644 --- a/README.md +++ b/README.md @@ -27,7 +27,7 @@ This is a **reincarnation**. We took the core philosophy, stripped out the obsol --- -## 2. Historical Context +## 2. Historical context Once upon a time, there was a company called **source{d}**. They pioneered "Mining Software Repositories" on a massive scale. Their crown jewel was `hercules`, a tool that could chew through git logs faster than you can say `git blame`. @@ -58,7 +58,7 @@ go install github.com/Sumatoshi-tech/codefang/cmd/codefang@latest go install github.com/Sumatoshi-tech/codefang/cmd/uast@latest ``` -### Let's Flex +### Let's flex Codefang follows the UNIX philosophy: small tools, joined by pipes. @@ -90,11 +90,11 @@ codefang history -a devs --head . --- -## 4. Architecture & Design Decisions +## 4. Architecture & design decisions We made a few bold choices in the rewrite. -### The Split: `uast` vs `codefang` +### The split: `uast` vs `codefang` The original `hercules` was a monolith. We split it in two: @@ -105,13 +105,13 @@ The original `hercules` was a monolith. We split it in two: Most linters are language-specific. `eslint` for JS, `golangci-lint` for Go. Codefang uses a **Universal** AST. This means we can write a single "Complexity Analyzer" and it immediately works for Python, Go, and TypeScript. -### Pluggable Analysis +### Pluggable analysis Analyzers are modular. You want to measure "Sentiment of Code Comments"? There's an analyzer for that (`sentiment`). You want to find "Typos in Variable Names"? There's one for that (`typos`). --- -## 5. Codefang as an AI Agent Tool +## 5. Codefang as an AI agent tool Don't just use AI to write code. Use Codefang to verify it. By giving `codefang` and `uast` to your AI agent as tools (via MCP or shell), you create a self-correcting quality loop. @@ -139,6 +139,40 @@ Don't just use AI to write code. Use Codefang to verify it. By giving `codefang` --- +## 6. Maintainers + +The dumbbells don't rack themselves. This project is kept in fighting shape by: + +- **Dmytro Gajewski** + +--- + +## 7. Contributing + +Spotted a bug? Want to teach Codefang a new trick? PRs are very welcome. + +Read the [Contributing guide](CONTRIBUTING.md) before you start—it covers the build, the test gauntlet, commit conventions, and how to open a PR that gets merged instead of bounced. + +--- + +## 8. Support + +Got a question, hit a wall, or think something's broken? The [Support guide](SUPPORT.md) points you to the right door—usage questions and bug reports go to GitHub Issues (discussions are off), and security holes go somewhere private. No more shouting into the void. + +--- + +## 9. Security + +Found a hole? Don't shout it from the rooftops. Report it privately through GitHub's "Report a vulnerability" button on the repository's Security tab. The [Security policy](SECURITY.md) has the full drill—what to include, what to expect, and why public issues are off-limits for vulnerabilities. + +--- + +## 10. License + +Codefang is released under the [Apache-2.0](LICENSE) license. Use it, fork it, ship it—just keep the notice intact. + +--- + ### Ready to lift? Check out the [full documentation](https://sumatoshi-tech.github.io/codefang/) for deep dives, or just start piping commands and see what breaks. diff --git a/SECURITY.md b/SECURITY.md new file mode 100644 index 00000000..92e73de0 --- /dev/null +++ b/SECURITY.md @@ -0,0 +1,45 @@ + + +# Security policy + +Found a way to make Codefang misbehave that goes beyond a normal bug? We want to +hear about it privately first, so we can fix it before it gets weaponised. + +## Supported versions + +Codefang is pre-1.0 and ships no tagged releases yet, so the supported surface is +simple: we patch the tip and we do not backport. + +| Version | Supported | +|---|---| +| Latest `main` (most recent release line) | Yes | +| Older pre-release commits | No | + +If you are running an older commit, the first fix is to update to the latest +`main` and confirm the issue still reproduces. + +## Reporting a vulnerability + +Please **do not open a public issue, discussion, or pull request** for a security +vulnerability. A public report tells attackers about the hole before we can patch +it. + +Instead, use GitHub's private vulnerability reporting: + +1. Go to the repository's **Security** tab. +2. Choose **Report a vulnerability** to open a private + [GitHub Security Advisory](https://docs.github.com/en/code-security/security-advisories/guidance-on-reporting-and-writing-information-about-vulnerabilities/privately-reporting-a-security-vulnerability). +3. Include: + - A description of the vulnerability and the impact you think it has. + - Reproduction steps or a proof-of-concept. + - The affected commit, branch, or build. + +This channel is private between you and the maintainers until a fix is ready. + +## What happens next + +- We aim to acknowledge your report within a few business days. +- Once confirmed, we work on a fix and keep you posted as it lands on `main`. +- We credit reporters in the advisory and release notes unless you ask us not to. + +Thanks for helping keep Codefang and the people who run it safe. diff --git a/SUPPORT.md b/SUPPORT.md new file mode 100644 index 00000000..da8df9a0 --- /dev/null +++ b/SUPPORT.md @@ -0,0 +1,41 @@ + + +# Support + +Stuck, confused, or pretty sure you found a bug? You're in the right place. This +page sends you to the right door so nothing gets lost and the maintainer's inbox +doesn't turn into one big undifferentiated pile. + +**GitHub Discussions is not enabled** for this project. Everything routes through +GitHub Issues or the security channel below—pick the one that matches what you have. + +## I have a question or need usage help + +Open a [GitHub issue](https://github.com/Sumatoshi-tech/codefang/issues/new) and add +the `question` label. Use this for "how do I...", "is this the right command for...", +or "why does Codefang do X" questions. + +Before you open it, a quick win: skim the +[documentation](https://sumatoshi-tech.github.io/codefang/), which covers the +analyzers, the `uast` / `codefang` split, and the common workflows. Your answer may +already be flexing there. + +## I found a bug + +Open a [GitHub issue](https://github.com/Sumatoshi-tech/codefang/issues/new) using +the **bug report** template. Tell us what you expected, what actually happened, the +steps to reproduce, and your environment (Codefang version, OS, Go version). The more +we can reproduce, the faster we can fix. + +## I found a security vulnerability + +**Do not open a public issue.** Security holes go through the private channel so we +can patch before they get weaponised. See [SECURITY.md](SECURITY.md) for the full +drill—it routes you to GitHub's private vulnerability reporting (Security Advisories), +not the public Issues tracker. + +## What to expect + +This project is maintained on a best-effort basis. We triage questions and bug reports +as time allows, and we prioritise reproducible reports. Be kind, be patient, and bring +details—that's how the dumbbells get racked. diff --git a/THIRD_PARTY_NOTICES.md b/THIRD_PARTY_NOTICES.md new file mode 100644 index 00000000..653c9537 --- /dev/null +++ b/THIRD_PARTY_NOTICES.md @@ -0,0 +1,30 @@ +# Third-party notices + +Codefang is licensed under [Apache-2.0](LICENSE). It reuses or bundles content +from the projects below. This file names each borrowed component and points to +its upstream licence; it does not reproduce the full licence texts. Consult the +linked upstream sources for the authoritative terms. + +The full dependency graph (the MIT / Apache-2.0 / BSD Go modules pulled in via +`go.mod`) is surfaced by the release tooling and is not enumerated here. The +entries below cover content that is borrowed verbatim, vendored, or otherwise +warrants an explicit notice. + +## Borrowed content + +| Component | What it is | Upstream licence | Link | +| --- | --- | --- | --- | +| Contributor Covenant 2.1 | Text adapted into our [code of conduct](site/contributing/code-of-conduct.md) | CC-BY-4.0 | | +| Tree-sitter grammars | Per-language parser grammars consumed via `go-sitter-forest` (one module per language in `go.mod`) | Per-grammar (mostly MIT / Apache-2.0) | | +| enry / Linguist dataset | Language-identification data and heuristics used for file classification | MIT | | +| libgit2 | Git plumbing library vendored as a submodule under `third_party/libgit2/` and used via `git2go` | GPL-2.0 with a linking exception (see `third_party/libgit2/COPYING`) | | + +## Notes + +- The Tree-sitter grammars are a category: each language grammar listed in + `go.mod` under `github.com/alexaandru/go-sitter-forest/*` carries its own + upstream licence. See the linked aggregator for the per-grammar provenance. +- libgit2 ships under GPL-2.0 **with a linking exception** that permits linking + the compiled library into other programs and distributing those combinations + without restriction. The exception and full text live in + `third_party/libgit2/COPYING`. diff --git a/cspell.json b/cspell.json new file mode 100644 index 00000000..6cfd717c --- /dev/null +++ b/cspell.json @@ -0,0 +1,88 @@ +{ + "$schema": "https://raw.githubusercontent.com/streetsidesoftware/cspell/main/cspell.schema.json", + "version": "0.2", + "language": "en", + "files": ["**/*.md"], + "ignorePaths": [ + "node_modules/**", + "build/**", + "vendor/**", + "third_party/**", + ".github/styles/**", + "CHANGELOG.md" + ], + "dictionaries": ["en_US", "softwareTerms", "go", "filetypes"], + "words": [ + "codefang", + "Sumatoshi", + "hercules", + "abandonware", + "burndown", + "couples", + "shotness", + "Halstead", + "LCOM", + "cohesion", + "anomaly", + "uast", + "UAST", + "libgit", + "libgit2", + "treesitter", + "enry", + "linguist", + "goreleaser", + "mkdocs", + "godoc", + "pkgsite", + "gosec", + "golangci", + "deadcode", + "filestats", + "hibernate", + "blob", + "blobs", + "monorepo", + "OTEL", + "otel", + "opentelemetry", + "MCP", + "mcp", + "stdio", + "stdout", + "stdin", + "JSONC", + "jsonc", + "lychee", + "lycheeignore", + "markdownlint", + "cspell", + "Vale", + "Diataxis", + "Diátaxis", + "MADR", + "Nygard", + "OpenSSF", + "CHAOSS", + "DCO", + "semver", + "changelog", + "vettool", + "cgo", + "CGO", + "pprof", + "goroutine", + "goroutines", + "tick", + "ticks", + "devs", + "LRU", + "hibernatable", + "timeseries" + ], + "ignoreRegExpList": [ + "/\\b[0-9a-f]{7,40}\\b/", + "/`[^`]+`/", + "/\\]\\([^)]*\\)/" + ] +} diff --git a/docs/adr/0001-record-architecture-decisions.md b/docs/adr/0001-record-architecture-decisions.md new file mode 100644 index 00000000..471985ad --- /dev/null +++ b/docs/adr/0001-record-architecture-decisions.md @@ -0,0 +1,68 @@ + + +# 0001 — Record architecture decisions + +- Status: accepted +- Date: record-architecture-decisions +- Deciders: @dmytrogajewski + +## Context and problem statement + +Codefang is a reincarnation of the abandoned `hercules` tool, rebuilt with a modern engine. The rewrite made several significant choices — a two-binary split, a Tree-sitter-backed Universal Abstract Syntax Tree, libgit2 via cgo, a streaming pipeline, MkDocs Material for docs, and the Apache-2.0 licence. These decisions currently live only as prose scattered across the README and `site/architecture/`, or as feature-scoped FRDs under `specs/frds/`. New contributors cannot see *why* a choice was made, what was considered, or whether a past decision still holds. How should the project capture architecturally significant decisions so the reasoning survives the people who made it? + +## Decision drivers + +- New contributors need the rationale behind a decision, not just its result. +- Decisions must be reviewable in the same pull-request flow as code. +- The record must outlive any individual maintainer. +- FRDs under `specs/frds/` are feature-scoped; they do not capture cross-cutting architectural rationale. +- The format must be lightweight enough that writing an ADR is not a deterrent. + +## Considered options + +- Architecture Decision Records (ADRs) in MADR 4.0 shape, stored in the repository. +- Keep recording decisions only in the README and `site/architecture/` prose. +- Reuse the existing `specs/frds/` FRD format for architectural decisions. +- A wiki or external document store outside version control. + +## Decision outcome + +Chosen option: "ADRs in MADR 4.0 shape, stored in the repository", because they version with the code, review through the normal pull-request flow, and carry the Context, Drivers, Options, and Consequences that prose and feature-scoped FRDs omit. + +ADRs live as numbered Markdown files at `docs/adr/NNNN-.md`. The canonical files are surfaced in the MkDocs site (under `docs_dir: site`) through a `site/adr` symlink so the documentation site can render them without duplicating content. Each ADR is immutable once accepted; a later decision that overturns it is a new ADR that marks the old one superseded and links to it. + +### Consequences + +- Good: The reasoning behind each significant choice is discoverable, reviewable, and durable. +- Good: A superseded decision and the decision that replaced it are linked, so the history of the architecture reads as a chain. +- Neutral: Every architecturally significant change now carries the small overhead of authoring an ADR. +- Bad: An ADR can drift from the implementation if a decision changes without a follow-up ADR; the governance process must require an ADR for significant changes to mitigate this. + +## Pros and cons of the options + +### ADRs in MADR 4.0 shape + +- Good: Captures context, drivers, considered options, and consequences in a consistent shape. +- Good: Versions with the code and reviews in the normal pull-request flow. +- Neutral: Requires a numbering and supersession convention, established by this ADR. + +### Prose in README and site/architecture/ + +- Good: No new convention to learn. +- Bad: Records the *what* but rarely the *why*, the alternatives, or the trade-offs. +- Bad: Prose is rewritten in place, so the history of a decision is lost. + +### Reuse specs/FRDs/ FRDs + +- Good: An existing, familiar format. +- Bad: FRDs are feature-scoped and describe what to build, not cross-cutting architectural rationale. + +### External wiki or document store + +- Good: Rich editing surface. +- Bad: Drifts out of sync with the code, is not reviewed in pull requests, and may not outlive the hosting service. + +## Links + +- Supersedes: none +- Superseded by: none diff --git a/docs/adr/0002-two-binary-split-uast-codefang.md b/docs/adr/0002-two-binary-split-uast-codefang.md new file mode 100644 index 00000000..7c39fa93 --- /dev/null +++ b/docs/adr/0002-two-binary-split-uast-codefang.md @@ -0,0 +1,63 @@ + + +# 0002 — Two-binary split: UAST and Codefang + +- Status: accepted +- Date: two-binary-split-uast-codefang +- Deciders: @dmytrogajewski + +## Context and problem statement + +The original `hercules` was a monolith. Codefang has two distinct jobs: turning source code into a standardized tree structure, and running static and history analyzers over code and Git repositories. Parsing is a self-contained transformation from source text to a Universal Abstract Syntax Tree (UAST); analysis consumes UASTs or Git repositories and produces metrics. Should both jobs ship as one binary, or be split so each does one thing? + +## Decision drivers + +- The project commits to the Unix philosophy: small tools joined by pipes. +- Parsing and analysis have different inputs, outputs, and likely consumers (an editor or agent may want only the parser). +- Composability: a UAST produced by one tool should pipe into the other. +- A clear seam keeps each tool's command surface small and its responsibility single. + +## Considered options + +- Two binaries: `uast` (parser) and `codefang` (analyzer), composed via pipes. +- A single monolithic binary with subcommands for both parsing and analysis. +- A library-only distribution with no standalone parser binary. + +## Decision outcome + +Chosen option: "Two binaries — `uast` and `codefang`", because parsing and analysis are separable responsibilities with different consumers, and a pipe-friendly split lets each tool stay small while still composing into a pipeline. + +`uast` (`cmd/uast/`) turns source code into UAST and exposes `parse`, `query`, `diff`, `explore`, and `server`. `codefang` (`cmd/codefang/`) consumes UASTs for static analysis or Git repositories for history analysis via `run`, `mcp`, and related commands. Both CLIs are built with Cobra and share the same `pkg/` libraries, so the split is at the command surface, not a duplication of core logic. The canonical composition is `uast parse main.go | codefang run -a static/* --format json`. + +### Consequences + +- Good: Each binary has one responsibility and a small command surface. +- Good: `uast` is usable on its own as a universal parser, independent of any analysis. +- Good: The tools compose through stdin/stdout, fitting CI pipelines and AI-agent tool wiring. +- Neutral: Two binaries must be installed (`go install .../cmd/codefang` and `.../cmd/uast`), and both are versioned together from one module. +- Bad: A workflow that always wants both pays an extra process boundary and a serialization step between them. + +## Pros and cons of the options + +### Two binaries (UAST + Codefang) + +- Good: Single-responsibility tools; either is useful alone. +- Good: Pipe composition matches the Unix philosophy the project commits to. +- Neutral: Shared `pkg/` code means the split costs no logic duplication. + +### Single monolithic binary + +- Good: One install, no inter-process serialization. +- Bad: One large command surface mixing parsing and analysis concerns. +- Bad: Cannot offer the parser alone to consumers that do not want the analyzer. + +### Library-only distribution + +- Good: Maximum flexibility for embedders. +- Bad: No paste-ready CLI for the common case; raises the barrier for first use and for pipe-based and agent workflows. + +## Links + +- Supersedes: none +- Superseded by: none +- Related: [0001 — Record architecture decisions](0001-record-architecture-decisions.md) diff --git a/docs/adr/0003-libgit2-via-cgo.md b/docs/adr/0003-libgit2-via-cgo.md new file mode 100644 index 00000000..ce395433 --- /dev/null +++ b/docs/adr/0003-libgit2-via-cgo.md @@ -0,0 +1,65 @@ + + +# 0003 — Git access through libgit2 via cgo + +- Status: accepted +- Date: libgit2-via-cgo +- Deciders: @dmytrogajewski + +## Context and problem statement + +History analysis is one of Codefang's two modes: it opens a Git repository, walks the commit history, computes per-commit tree and file diffs, and reads blob content for many commits across potentially planet-scale repositories. This is the hot path for burndown, couples, devs, and every other history analyzer. The project needs a Git implementation that is fast at bulk commit walking, tree diffing, and blob lookup, and that supports both normal and bare repositories. How should Codefang access Git object and history data? + +## Decision drivers + +- History analysis must walk large commit histories and diff trees at high throughput. +- The implementation must support both normal and bare repositories. +- The project targets planet-scale repositories, so memory and CPU efficiency of Git operations are first-order concerns. +- A stable, well-maintained Git core with broad object-model coverage reduces the surface the project must reimplement. + +## Considered options + +- libgit2 through the `git2go/v34` cgo bindings, with libgit2 vendored and built statically (`third_party/libgit2`). +- A pure-Go Git implementation (`go-git`). +- Shelling out to the `git` command-line binary. + +## Decision outcome + +Chosen option: "libgit2 via the `git2go/v34` cgo bindings", because libgit2 is a mature C implementation of the Git object model that delivers the throughput history analysis needs for tree diffing and blob lookup, and the cgo bindings give direct, in-process access without per-operation subprocess overhead. + +`pkg/gitlib/` wraps `git2go` for repository open, commit walking, tree diff, changes, blob reads, and its worker pool / batch processing. libgit2 is vendored under `third_party/libgit2` and built statically; `make` drives the build and sets `CGO_CFLAGS`, `CGO_LDFLAGS`, and `CGO_ENABLED=1` so the toolchain links the vendored library. The plumbing analyzers (`TreeDiffAnalyzer`, `BlobCacheAnalyzer`, `FileDiffAnalyzer`) sit directly on top of this layer. + +### Consequences + +- Good: High-throughput tree diffing and blob lookup in-process, matching the planet-scale target. +- Good: First-class support for both normal and bare repositories through libgit2's object model. +- Good: Vendoring and a static build pin a reproducible libgit2 version rather than depending on whatever the host provides. +- Neutral: Git access is isolated behind `pkg/gitlib/`, so the rest of the codebase does not depend on the binding directly. +- Bad: The build requires cgo and a C toolchain; `CGO_ENABLED=1` and the libgit2 include/lib paths must be set, which complicates cross-compilation and pure-Go static builds. +- Bad: cgo introduces a Go/C boundary with manual resource lifetimes; the `pkg/gitlib/` layer must free libgit2 objects carefully to avoid leaks. + +## Pros and cons of the options + +### libgit2 via git2go (cgo), vendored and static + +- Good: Mature C Git core; strong throughput on the diff and blob hot paths. +- Good: Vendored static build pins the version and avoids host-library drift. +- Bad: Requires cgo, a C toolchain, and explicit CGO build flags; harder to cross-compile. +- Bad: Manual memory management across the cgo boundary. + +### Pure-Go go-git + +- Good: No cgo; trivial cross-compilation and pure-Go static binaries. +- Bad: Lower throughput on bulk history walking and tree diffing for very large repositories, which is exactly the planet-scale hot path. + +### Shell out to the git CLI + +- Good: No build-time dependency beyond a `git` on PATH. +- Bad: Per-operation subprocess overhead is prohibitive when diffing and reading blobs across an entire history. +- Bad: Parsing CLI output is brittle compared to a typed object-model API. + +## Links + +- Supersedes: none +- Superseded by: none +- Related: [0001 — Record architecture decisions](0001-record-architecture-decisions.md) diff --git a/llms-full.txt b/llms-full.txt new file mode 100644 index 00000000..aea3e5c1 --- /dev/null +++ b/llms-full.txt @@ -0,0 +1,2974 @@ +# Codefang — llms-full.txt + +> Full-text concatenation of the documents indexed in llms.txt, in the same order. Each document is preceded by a "# File: " separator. Generated from the repository sources; do not edit by hand. + + +--- + +# File: README.md + +# Codefang + +

+ Codefang Logo +

+ +[![CI](https://github.com/Sumatoshi-tech/codefang/actions/workflows/ci.yml/badge.svg)](https://github.com/Sumatoshi-tech/codefang/actions/workflows/ci.yml) +[![Go Reference](https://pkg.go.dev/badge/github.com/Sumatoshi-tech/codefang.svg)](https://pkg.go.dev/github.com/Sumatoshi-tech/codefang) + +**The heavy lifter for your codebase.** + +Codefang is a comprehensive code analysis platform that understands your code deeply—not just as text, but as structure (AST) and history (Git). Whether you're tracking technical debt, analyzing developer churn, or feeding context to an AI, Codefang does the heavy lifting so you don't have to. + +--- + +## 1. Preface + +For detailed technical documentation, architectural decisions, and algorithm deep-dives, please visit our **[Documentation](https://sumatoshi-tech.github.io/codefang/)**. + +So, you've stumbled upon Codefang. You might be thinking, *"Wait, didn't `src-d` build something like this years ago?"* + +Yes, they did. And it was glorious. But like many great empires, `src-d` faded into the annals of history, and the original `hercules` was left gathering dust in the digital attic. + +**This is not that project.** + +This is a **reincarnation**. We took the core philosophy, stripped out the obsolete parts, and rebuilt it with a modern engine. It's not a drop-in replacement; it's a spiritual successor with a gym membership and a PhD in Abstract Syntax Trees. + +--- + +## 2. Historical context + +Once upon a time, there was a company called **source{d}**. They pioneered "Mining Software Repositories" on a massive scale. Their crown jewel was `hercules`, a tool that could chew through git logs faster than you can say `git blame`. + +It gave us insights like: +- **Burndown Charts**: How much code from 2015 is still surviving today? +- **Coupling**: If I change `A.go`, does `B.go` always change with it? + +When `src-d` ceased operations, the tool became abandonware. But the need to understand code didn't go away. + +We revived the project with a new mission: **Combine history with structure.** +The old `hercules` was great at Git history. Codefang adds **UAST (Universal Abstract Syntax Tree)** powers, allowing it to understand the *meaning* of the code across 60+ languages, not just the diffs. + +--- + +## 3. Quickstart + +Enough history class. Let's see some muscles. + +### Installation + +You'll need Go installed. Then, just grab the binaries: + +```sh +# The brain: Code analysis and history mining +go install github.com/Sumatoshi-tech/codefang/cmd/codefang@latest + +# The eyes: Universal parser (supports 60+ languages) +go install github.com/Sumatoshi-tech/codefang/cmd/uast@latest +``` + +### Let's flex + +Codefang follows the UNIX philosophy: small tools, joined by pipes. + +**1. Analyze complexity (Static Analysis):** +Parse your code and pipe it into the analyzer. + +```sh +# How messy is my main.go? +uast parse main.go | codefang analyze -a complexity + +# ...or my entire codebase? +uast parse **/*.go | codefang analyze -a complexity +``` + +**2. Analyze history (Git Forensics):** +See who knows what, and how code is aging. + +```sh +# Generate a burndown chart (lines surviving over time) +codefang history -a burndown . +``` + +**3. Find the experts:** +Who actually wrote the code that's running in production? + +```sh +codefang history -a devs --head . +``` + +--- + +## 4. Architecture & design decisions + +We made a few bold choices in the rewrite. + +### The split: `uast` vs `codefang` + +The original `hercules` was a monolith. We split it in two: + +1. **`uast` (The Parser)**: This tool focuses on one thing—turning source code into a standardized tree structure. It uses **Tree-sitter** under the hood to support practically every language you care about (Go, Python, JS, Rust, C++, Java, and even COBOL... probably). +2. **`codefang` (The Analyzer)**: This tool consumes the data. It takes UASTs for static analysis or Git repositories for history analysis. + +### Why UAST? + +Most linters are language-specific. `eslint` for JS, `golangci-lint` for Go. Codefang uses a **Universal** AST. This means we can write a single "Complexity Analyzer" and it immediately works for Python, Go, and TypeScript. + +### Pluggable analysis + +Analyzers are modular. You want to measure "Sentiment of Code Comments"? There's an analyzer for that (`sentiment`). You want to find "Typos in Variable Names"? There's one for that (`typos`). + +--- + +## 5. Codefang as an AI agent tool + +Don't just use AI to write code. Use Codefang to verify it. By giving `codefang` and `uast` to your AI agent as tools (via MCP or shell), you create a self-correcting quality loop. + +**1. The Self-Correcting Coder** +* **Scenario:** Agent generates a new function or refactors a file. +* **Action:** Agent runs `codefang analyze -a complexity` on the new code. +* **Outcome:** If complexity scores are high (e.g., > 15), the agent self-reflects: *"This is too complex. I need to simplify logic before showing it to the user."* +* **Result:** Clean, maintainable code *before* it even hits the PR. + +**2. Architectural Context Injection** +* **Scenario:** You ask the agent about a high-level architectural change. +* **Problem:** "Context Window Exceeded" or the agent hallucinates file relationships. +* **Solution:** Agent runs `uast parse` to get the AST structure or `codefang analyze -a imports` to map the dependency graph. It learns the system architecture without reading every single line of text. + +**3. Risk-Aware Refactoring** +* **Scenario:** Agent is asked to refactor a legacy module. +* **Action:** Agent runs `codefang history -a couples`. +* **Insight:** *"Warning: Changing `User.go` usually breaks `Billing.go` 80% of the time."* +* **Result:** Agent proactively checks related files for regressions, preventing bugs that a simple text-based analysis would miss. + +**4. Style & Consistency Enforcement** +* **Scenario:** "Make this look like the rest of the project." +* **Action:** Agent analyzes `cohesion` and `comments` metrics of existing high-quality files to set a baseline. +* **Result:** Agent generates code that matches the *structural* quality standards of your specific repo, not just generic language syntax. + +--- + +## 6. Maintainers + +The dumbbells don't rack themselves. This project is kept in fighting shape by: + +- **Dmytro Gajewski** + +--- + +## 7. Contributing + +Spotted a bug? Want to teach Codefang a new trick? PRs are very welcome. + +Read the [Contributing guide](CONTRIBUTING.md) before you start—it covers the build, the test gauntlet, commit conventions, and how to open a PR that gets merged instead of bounced. + +--- + +## 8. License + +Codefang is released under the [Apache-2.0](LICENSE) license. Use it, fork it, ship it—just keep the notice intact. + +--- + +### Ready to lift? + +Check out the [full documentation](https://sumatoshi-tech.github.io/codefang/) for deep dives, or just start piping commands and see what breaks. + +*Happy Mining!* + + +--- + +# File: site/getting-started/quickstart.md + +# Quick start + +This page shows the fastest path to useful results. In five minutes you will +parse code, run static analysis, and mine Git history. + +!!! tip "Prerequisites" + Make sure both `codefang` and `uast` are installed. + See [Installation](installation.md) if you have not set them up yet. + +--- + +## 1. Parse source code into a UAST + +Start by parsing a single file into a Universal Abstract Syntax Tree: + +```bash +uast parse main.go +``` + +??? example "Expected output (truncated)" + + ```json + { + "language": "go", + "file": "main.go", + "root": { + "type": "SourceFile", + "roles": ["File"], + "children": [ + { + "type": "PackageClause", + "roles": ["Package", "Declaration"], + "props": { "name": "main" } + }, + { + "type": "FunctionDeclaration", + "roles": ["Function", "Declaration"], + "props": { "name": "main" }, + "children": [ "..." ] + } + ] + } + } + ``` + +You can parse entire directories, force a language, or read from stdin: + +```bash +uast parse --all # every source file in the codebase +uast parse -l python script.sh # force Python language detection +cat main.go | uast parse - # read from stdin +``` + +--- + +## 2. Static analysis + +Run structural analysis on a folder. The `codefang run` command with +`static/*` analyzers walks the directory, parses every supported file, and +computes metrics: + +```bash +codefang run -a static/complexity --format text . +``` + +??? example "Expected output" + + ``` + Codefang -- Code Analysis Report + =============================== + + static/complexity: + internal/framework/runner.go + Function RunPipeline -- Cyclomatic: 12 Cognitive: 8 + Function initWorkers -- Cyclomatic: 4 Cognitive: 3 + cmd/codefang/commands/run.go + Function run -- Cyclomatic: 9 Cognitive: 7 + ``` + +!!! tip "Multiple static analyzers" + Combine several analyzers in one pass: + + ```bash + codefang run -a static/complexity,static/halstead,static/cohesion --format text . + ``` + +--- + +## 3. History analysis -- burndown + +Analyze how code ages and survives over your repository's lifetime: + +```bash +codefang run -a history/burndown --format yaml . +``` + +??? example "Expected output (truncated)" + + ```yaml + burndown: + granularity: 30 + sampling: 30 + project: + - [0, 1200, 1100, 980, 920, 870] + - [0, 0, 320, 290, 270, 250] + - [0, 0, 0, 180, 160, 140] + ticks: + - "2024-01-15" + - "2024-02-14" + - "2024-03-15" + - "2024-04-14" + - "2024-05-14" + - "2024-06-13" + ``` + +Each row in `project` represents lines added in a given time window; each +column shows how many of those lines survive into subsequent windows. + +--- + +## 4. Developer analysis + +See per-developer contribution statistics with a single commit snapshot: + +```bash +codefang run -a history/devs --head --format yaml . +``` + +??? example "Expected output (truncated)" + + ```yaml + devs: + developers: + - name: "Alice" + commits: 142 + added: 12400 + removed: 3200 + changed: 8900 + - name: "Bob" + commits: 87 + added: 6300 + removed: 1800 + changed: 4100 + ``` + +!!! info "The `--head` flag" + `--head` tells the history pipeline to analyze only the HEAD commit's + state rather than walking the full history. It is much faster and useful + for snapshot metrics like developer totals. + +--- + +## 5. Run everything at once + +Use glob patterns to run all history analyzers (or all analyzers) in a single +invocation: + +=== "All history analyzers" + + ```bash + codefang run -a "history/*" --format json . + ``` + +=== "All static analyzers" + + ```bash + codefang run -a "static/*" --format json . + ``` + +=== "Everything" + + ```bash + codefang run -a "*" --format json . + ``` + +!!! tip "JSON output for scripting" + JSON is the default format and the most machine-friendly. Pipe it into + `jq` for quick exploration: + + ```bash + codefang run -a history/burndown --format json . | jq '.burndown.ticks' + ``` + +--- + +## 6. Interactive HTML plots + +Generate a self-contained HTML report with interactive charts: + +```bash +codefang run -a history/burndown,history/devs --format plot . +``` + +This writes an HTML page to **stdout**. Redirect it to a file and open in your +browser: + +```bash +codefang run -a history/burndown --format plot . > burndown.html +open burndown.html # macOS +xdg-open burndown.html # Linux +``` + +--- + +## 7. Pipe into AI (MCP integration) + +Codefang ships a built-in **Model Context Protocol** server so AI agents can +query analysis results programmatically: + +```bash +codefang mcp serve +``` + +This starts a JSON-RPC server that tools like Claude Desktop, Cursor, and +other MCP-compatible clients can connect to. See the +[MCP Integration](../integrations/mcp.md) guide for configuration details. + +!!! note "AI-ready output" + Any `--format json` output can also be fed directly to an LLM for + summarization or code review: + + ```bash + codefang run -a static/complexity --format json . | llm "summarize the complexity hotspots" + ``` + +--- + +## What's next? + +| Goal | Page | +|------|------| +| Understand static vs. history analysis in depth | [First Analysis](first-analysis.md) | +| Learn every CLI flag | [CLI Reference](../guide/cli-reference.md) | +| Tune output formats | [Output Formats](../guide/output-formats.md) | +| Configure via `.codefang.yaml` | [Configuration](../guide/configuration.md) | +| Set up CI/CD pipelines | [Docker & GitHub Actions](../integrations/docker-and-actions.md) | + + +--- + +# File: site/how-to/analyze-a-monorepo.md + +# How to analyze a monorepo + + + +## Goal + +Run Codefang against a single large repository that holds many languages and +services, and get a clean per-service report without drowning in vendored or +generated noise. + +## Prerequisites + +- `codefang` and `uast` installed. See [Installation](../getting-started/installation.md). +- A monorepo checked out locally, with full Git history (history analyzers need + every commit, not a shallow clone). +- `jq` if you want to post-process the JSON output. + +## Steps + +1. Run static analysis across the whole tree first. By default Codefang already + excludes vendored dependencies (`vendor/`, `node_modules/`, `third_party/`, + `testdata/`) and generated files, so the report covers your own code only: + + ```bash + codefang run -a 'static/*' --format json --silent . + ``` + +2. Narrow a noisy polyglot repo to the languages you care about. The + `--languages` filter applies to both the static and history phases and skips + non-matching files before they are parsed: + + ```bash + codefang run -a 'static/*' --languages go,typescript,python --format json . + ``` + +3. Scope a run to a single service directory instead of the whole repo by + passing its path (or `--path`): + + ```bash + codefang run -a 'static/complexity,static/cohesion' --format json ./services/api + ``` + +4. Add extra exclusion prefixes for tooling directories that Linguist's + heuristics do not catch, such as a Python virtualenv or a Rust target dir: + + ```bash + codefang run -a '*' --extra-excluded-prefixes '.venv/,target/,build/' --format json . + ``` + +5. Constrain memory on a large monorepo so the streaming pipeline auto-tunes + worker count and spill thresholds rather than consuming all available RAM: + + ```bash + codefang run -a 'history/*' --memory-budget 4GB --workers 4 --format json --silent . + ``` + +6. If you need only a subset of the history, limit the commit range with + `--since` or `--limit` to keep the run bounded: + + ```bash + codefang run -a 'history/devs' --since 2025-01-01 --format json . + ``` + +## Result + +You have a JSON report scoped to your own source code (vendored and generated +files excluded) and, when you narrow by `--languages` or path, scoped to the +service you care about. Confirm the scope by inspecting the report — for +example, count the files Codefang actually analyzed: + +```bash +codefang run -a 'static/complexity' --languages go --format json . | jq '.["static/complexity"] | length' +``` + +## See also + +- [How to run Codefang in CI](run-in-ci.md) +- [Large-scale repository scanning](../operations/large-scale-scanning.md) +- [CLI reference](../guide/cli-reference.md) + + +--- + +# File: site/how-to/run-in-ci.md + +# How to run Codefang in CI + + + +## Goal + +Add Codefang to a continuous-integration pipeline so every push and pull request +runs code analysis automatically, and optionally fail the build when analysis +detects issues. + +## Prerequisites + +- A repository hosted on GitHub (this guide uses the published GitHub Action). +- Permission to add workflow files under `.github/workflows/`. +- For history analyzers, the checkout step must fetch full history + (`fetch-depth: 0`); a shallow clone only sees the most recent commits. + +## Steps + +1. Create `.github/workflows/codefang.yml` and add the action with the + analyzers you want. Static analysis works on a shallow clone: + + ```yaml + name: Code Quality + on: [push, pull_request] + + jobs: + analyze: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + + - name: Run Codefang + id: codefang + uses: Sumatoshi-tech/codefang@main + with: + analyzers: "static/*" + format: "json" + ``` + +2. Turn the run into a quality gate by setting `fail-on-error: "true"`. The step + fails the workflow when analysis detects issues: + + ```yaml + - name: Run Codefang + uses: Sumatoshi-tech/codefang@main + with: + analyzers: "static/complexity,static/comments" + fail-on-error: "true" + ``` + +3. For history analyzers, fetch the full Git history in the checkout step, + otherwise the analysis only sees the shallow clone: + + ```yaml + - uses: actions/checkout@v4 + with: + fetch-depth: 0 + + - name: Run Codefang History + uses: Sumatoshi-tech/codefang@main + with: + analyzers: "history/burndown,history/devs" + format: "json" + ``` + +4. Read the action outputs in a later step. `pass` is `true` when analysis + completed without errors, and `report` holds the full report content: + + ```yaml + - name: Check results + run: | + echo "Pass: ${{ steps.codefang.outputs.pass }}" + echo "${{ steps.codefang.outputs.report }}" + ``` + +5. If a large repository runs out of memory in CI, constrain it with a memory + budget by running the binary directly inside the job, or in Docker: + + ```bash + docker run --rm -v "$(pwd):/workspace:ro" \ + codefang run -a 'history/*' --memory-budget 2GiB --format json --silent /workspace + ``` + +## Result + +Open the Actions tab after a push. The Codefang job runs, prints the report, and +— when `fail-on-error` is set — marks the check red if analysis detected issues. +A green check with a populated `report` output confirms the pipeline works. + +## See also + +- [Docker & GitHub Actions reference](../integrations/docker-and-actions.md) +- [How to analyze a monorepo](analyze-a-monorepo.md) +- [Output formats](../guide/output-formats.md) + + +--- + +# File: site/how-to/read-a-burndown-chart.md + +# How to read a burndown chart + + + +## Goal + +Generate a code-survival burndown for a repository and interpret the result so +you can see how much code from each era still survives at the latest sample. + +## Prerequisites + +- `codefang` installed and a Git repository checked out with full history. +- A web browser to view the interactive HTML chart. +- `jq` if you want to inspect the raw matrix. + +## Steps + +1. Run the burndown analyzer and write the YAML output so you can read the raw + matrix: + + ```bash + codefang run -a history/burndown --format yaml . + ``` + +2. Read the `project` matrix. Each row is a sampling point in time; each value + in a row is the number of lines, created in a given age band, that still + survive at that sample. As you read down the rows, watch a band's values + shrink — that is code being rewritten or deleted over time: + + ```yaml + burndown: + granularity: 30 + sampling: 30 + project: + - [0, 1200, 1100, 980, 920, 870] + - [0, 0, 320, 290, 270, 250] + ticks: + - "2024-01-15" + - "2024-02-14" + ``` + +3. Use `ticks` to map matrix positions back to calendar dates. The `granularity` + key is the width of each age band (in ticks) and `sampling` is how often a + snapshot is taken; the default tick is a 24-hour window. + +4. Add a per-developer breakdown to see whose code survives, by enabling + developer tracking: + + ```bash + codefang run -a history/burndown --burndown-people --format yaml . + ``` + +5. Generate the interactive stacked-area chart and open it in a browser. This is + the visual form of the same matrix: + + ```bash + codefang run -a history/burndown --format plot . > burndown.html + xdg-open burndown.html # Linux + open burndown.html # macOS + ``` + +6. Extract just the tick axis for scripting or a custom plot: + + ```bash + codefang run -a history/burndown --format json . | jq '.burndown.ticks' + ``` + +## Result + +You can point at any value in the chart or matrix and state what it means: "this +many lines, written around this date, still survive at this later date." A +declining survival curve signals churn; a flat top band signals stable, +long-lived code. + +## See also + +- [Understanding code burndown](../explanation/burndown.md) — the mental model. +- [Burndown reference](../analyzers/burndown.md) — configuration keys and output schema. +- [How to run Codefang in CI](run-in-ci.md) + + +--- + +# File: site/how-to/analyze-a-bare-repo.md + +# How to analyze a bare repository + + + +## Goal + +Run history analysis directly against a bare Git repository — such as a GitLab +backup, a Gitolite mirror, or a `git clone --bare` mirror — without first +checking out a working tree. + +## Prerequisites + +- `codefang` installed. +- A bare repository on disk (a directory ending in `.git` with no working tree). +- For GitLab backups, enough disk space to extract the `repositories/` tree. + +## Steps + +1. Point `codefang run` directly at the bare repository path. Codefang uses + libgit2, which opens bare and non-bare repositories transparently, so no + clone or checkout is required: + + ```bash + codefang run /data/backups/repositories/@hashed/ab/cd/abcdef.git \ + -a 'history/*' --format json --silent + ``` + +2. To scan a GitLab backup, extract the repositories tree from the tarball: + + ```bash + tar xf backup.tar -C /data/extract repositories/ + ``` + +3. Build a list of real project repos, skipping wiki and design repos: + + ```bash + find /data/extract/repositories -name "*.git" \ + ! -name "*.wiki.git" \ + ! -name "*.design.git" \ + -type d > /tmp/repo_paths.txt + ``` + +4. Run a single bare repo with a memory budget so the streaming pipeline stays + bounded on large histories: + + ```bash + codefang run /data/extract/repositories/@hashed/ab/cd/abcdef.git \ + -a 'history/*' --memory-budget 4GiB --workers 4 --format json --silent + ``` + +5. For periodic scans, analyze only new commits with `--since` rather than + re-walking the whole history each run: + + ```bash + codefang run /data/extract/repositories/@hashed/ab/cd/abcdef.git \ + -a 'history/*' --since 168h --format json --silent + ``` + +## Result + +You get the same history report you would get from a normal working clone — +burndown, developers, couples, and more — produced straight from the bare repo +with no checkout. A non-empty JSON report on stdout confirms the bare repo was +opened and analyzed. + +## See also + +- [Large-scale repository scanning](../operations/large-scale-scanning.md) +- [How to analyze a monorepo](analyze-a-monorepo.md) +- [CLI reference](../guide/cli-reference.md) + + +--- + +# File: site/how-to/attach-an-mcp-client.md + +# How to attach an MCP client + + + +## Goal + +Connect an AI agent to Codefang over the Model Context Protocol (MCP) so the +agent can run code analysis as a tool. + +## Prerequisites + +- `codefang` installed and on your `PATH`. +- An MCP-compatible client (for example, Claude Code or another agent that + launches MCP servers over stdio). +- For the `codefang_history` tool, an absolute path to a local Git repository. + +## Steps + +1. Confirm the MCP server starts. It communicates over stdio using JSON-RPC, the + standard transport for local MCP tool use: + + ```bash + codefang mcp + ``` + + Press `Ctrl-C` to stop it — the agent will launch its own copy. + +2. Register Codefang as an MCP server in your client. For Claude Code, add it to + `~/.claude/settings.json` (user scope) or `.claude/settings.json` (project + scope): + + ```json + { + "mcpServers": { + "codefang": { + "command": "codefang", + "args": ["mcp"], + "env": {} + } + } + } + ``` + +3. Restart the client so it picks up the new server. The agent then discovers + three tools via the standard MCP `tools/list` method: `codefang_analyze` + (static analysis on inline code), `uast_parse` (parse source into a UAST), + and `codefang_history` (history analysis on a local repo). + +4. Ask the agent to run history analysis. The `codefang_history` tool requires + an absolute `repo_path`; a relative path is rejected: + + ```json + { + "name": "codefang_history", + "arguments": { + "repo_path": "/home/user/projects/myapp", + "analyzers": ["burndown", "devs"], + "limit": 500 + } + } + ``` + +5. If a call misbehaves, enable debug logging to stderr by adding the `--debug` + argument to the server registration: + + ```json + { + "mcpServers": { + "codefang": { + "command": "codefang", + "args": ["mcp", "--debug"], + "env": {} + } + } + } + ``` + +## Result + +The agent lists the three Codefang tools and returns structured analysis when it +calls them. A successful `codefang_analyze` or `codefang_history` call that comes +back without `isError: true` confirms the client is attached. + +## See also + +- [MCP server reference](../integrations/mcp.md) +- [AI agents](../integrations/ai-agents.md) +- [How to read a burndown chart](read-a-burndown-chart.md) + + +--- + +# File: site/analyzers/index.md + +# Analyzers + +Codefang ships two families of analyzers that extract quantitative insights from your codebase. + +**Static analyzers** operate on source code via UAST (Universal Abstract Syntax Tree) and evaluate structural properties of the code as it exists right now. **History analyzers** walk Git commit history and track how the codebase evolves over time. + +--- + +## Static analyzers + +Static analyzers parse source files into a UAST and compute metrics in a single pass. They require no repository history and can run on individual files or entire directory trees. + +| Analyzer | ID | Description | +|---|---|---| +| [Complexity](complexity.md) | `complexity` | Cyclomatic complexity, cognitive complexity, nesting depth | +| [Cohesion](cohesion.md) | `cohesion` | LCOM4 class cohesion, method-field usage graphs | +| [Halstead](halstead.md) | `halstead` | Program length, vocabulary, volume, difficulty, effort | +| [Comments](comments.md) | `comments` | Documentation coverage, comment placement quality | +| [Imports](imports.md) | `imports` | Import and dependency analysis | +| [Clones](clones.md) | `clones` | Cross-file duplicate function detection via MinHash + LSH | + +### Running static analyzers + +Pipe a UAST through `codefang analyze`: + +```bash +# Single file, single analyzer +uast parse main.go | codefang analyze -a complexity + +# Single file, all static analyzers +uast parse main.go | codefang analyze + +# Entire directory tree +codefang analyze -a complexity ./src/ +``` + +--- + +## History analyzers + +History analyzers iterate over Git commits (oldest to newest) and accumulate statistics about how files, developers, and code structure change over time. They require a Git repository. + +| Analyzer | ID | Description | +|---|---|---| +| [Burndown](burndown.md) | `history/burndown` | Code survival over time, line ownership tracking | +| [Developers](developers.md) | `history/devs` | Developer activity, language breakdown, bus factor | +| [Couples](couples.md) | `history/couples` | File coupling, co-change patterns | +| [File History](file-history.md) | `history/file-history` | Per-file lifecycle and modification tracking | +| [Quality](quality.md) | `history/quality` | Complexity, Halstead, comment, and cohesion metrics over time | +| [Sentiment](sentiment.md) | `history/sentiment` | Comment sentiment analysis over time | +| [Shotness](shotness.md) | `history/shotness` | Structural hotspots (function-level change tracking) | +| [Typos](typos.md) | `history/typos` | Typo detection dataset builder | +| [Anomaly](anomaly.md) | `history/anomaly` | Z-score temporal anomaly detection | + +### Running history analyzers + +Use `codefang run` with the `-a` flag: + +```bash +# Single analyzer +codefang run -a history/burndown . + +# Multiple analyzers +codefang run -a history/devs -a history/couples . + +# All history analyzers +codefang run -a 'history/*' . +``` + +--- + +## Glob patterns + +You can select analyzers using glob patterns with the `-a` flag: + +```bash +# All static analyzers +codefang analyze -a 'static/*' ./src/ + +# All history analyzers +codefang run -a 'history/*' . + +# All analyzers (both static and history) +codefang run -a '*' . +``` + +--- + +## Output formats + +All analyzers support multiple output formats via the `-f` flag: + +=== "JSON" + + ```bash + codefang run -a history/devs -f json . + ``` + +=== "YAML" + + ```bash + codefang run -a history/devs -f yaml . + ``` + +=== "Plot" + + ```bash + codefang run -a history/burndown -f plot . + ``` + +=== "Binary" + + ```bash + codefang run -a history/burndown -f binary . + ``` + +!!! tip "Combining analyzers" + When running multiple history analyzers together, they share the same commit walk, making combined runs significantly faster than running each analyzer separately. + + +--- + +# File: site/guide/cli-reference.md + +# CLI reference + +Codefang ships two binaries: **`codefang`**, the analyzer engine, and **`uast`**, the +Universal Abstract Syntax Tree parser. This page is the exhaustive reference for +every command, sub-command, and flag. + +--- + +## `codefang` + +The main analysis binary. It runs static and history analyzers against source +code and Git repositories. + +### `codefang run` + +The primary entry point. Runs selected static and/or history analyzers on a +codebase or repository. + +```bash +codefang run [path] [flags] +``` + +If `path` is omitted the current directory (or the value of `--path`) is used. + +#### Analyzer selection + +| Flag | Short | Type | Default | Description | +|------|-------|------|---------|-------------| +| `--analyzers` | `-a` | `[]string` | `nil` | Analyzer IDs or glob patterns. Comma-separated. | + +Analyzer IDs follow a `/` convention. Globs are supported: + +```bash +# Single analyzer +codefang run -a static/complexity . + +# All static analyzers +codefang run -a 'static/*' . + +# All history analyzers +codefang run -a 'history/*' . + +# Multiple specific analyzers +codefang run -a static/complexity,history/burndown,history/devs . + +# Everything +codefang run -a '*' . +``` + +??? info "Available Analyzer IDs" + + **Static analyzers:** + `static/complexity`, `static/comments`, `static/halstead`, + `static/cohesion`, `static/imports` + + **History analyzers:** + `history/anomaly`, `history/burndown`, `history/couples`, + `history/devs`, `history/file-history`, `history/imports`, + `history/quality`, `history/sentiment`, `history/shotness`, + `history/typos` + +#### Output flags + +| Flag | Short | Type | Default | Description | +|------|-------|------|---------|-------------| +| `--format` | | `string` | `json` | Output format: `json`, `text`, `compact`, `yaml`, `plot`, `bin`, `timeseries` | +| `--verbose` | `-v` | `bool` | `false` | Enable detailed output (RSS/memory stats in progress logs, full static report details) | +| `--silent` | | `bool` | `false` | Suppress progress output on stderr | +| `--no-color` | | `bool` | `false` | Disable colored static output | + +```bash +# Human-readable table +codefang run -a 'static/*' --format text . + +# Interactive HTML charts +codefang run -a 'history/*' --format plot . + +# Unified time-series JSON +codefang run -a 'history/devs,history/sentiment' --format timeseries . +``` + +#### Path & input flags + +| Flag | Short | Type | Default | Description | +|------|-------|------|---------|-------------| +| `--path` | `-p` | `string` | `.` | Folder or repository path to analyze | +| `--input` | | `string` | `""` | Input report path for cross-format conversion | +| `--input-format` | | `string` | `auto` | Input format: `auto`, `json`, `bin` | + +!!! tip "Format Conversion" + + Use `--input` to convert a previously generated report to a different + format without re-running analysis: + + ```bash + codefang run -a 'history/*' --format bin . > report.bin + codefang run -a 'history/*' --input report.bin --format plot + ``` + +#### Git history flags + +| Flag | Type | Default | Description | +|------|------|---------|-------------| +| `--limit` | `int` | `0` | Maximum commits to analyze (`0` = no limit) | +| `--since` | `string` | `""` | Only analyze commits after this time | +| `--first-parent` | `bool` | `false` | Follow only first parent of merge commits | +| `--head` | `bool` | `false` | Analyze only HEAD commit | + +The `--since` flag accepts multiple formats: + +- **Go duration**: `24h`, `168h`, `720h` +- **Date**: `2025-01-01` +- **RFC 3339**: `2025-01-01T00:00:00Z` + +```bash +# Last 7 days only +codefang run -a history/devs --since 168h . + +# Since a specific date +codefang run -a history/burndown --since 2025-01-01 . + +# Only the latest 500 commits +codefang run -a history/couples --limit 500 . +``` + +!!! note "Burndown and `--first-parent`" + + The burndown analyzer automatically enables `--first-parent` when selected. + This is required for correct line-tracking across merge commits. + +#### Language filtering + +| Flag | Type | Default | Description | +|------|------|---------|-------------| +| `--languages` | `[]string` | `[all]` | Restrict analysis to the given Linguist languages; comma-separated. `all` (default) disables the filter. Applies to **both** history and static phases. | + +**History phase** — the filter is pushed down into libgit2's `pathspec` +at the tree-diff stage, so non-matching files are skipped before the +diff crosses the cgo boundary. On a polyglot repo a narrow filter can +reduce wall time by 30–40 %. The Go-side language check still runs as +the authoritative pass for content-disambiguated extensions (`.h`, +`.pl`, `.m`, `.r`). + +**Static phase** — the filter is applied at the directory walker +(`matchesLanguageGlobs`) before the UAST parser or raw-file analyzers +see the file. It's path-based only: the parser's own language router +remains the final authority for how a matched file is parsed (e.g. a +`.h` under `--languages c++` is still parsed as C). Both phases read +from the same `langpath.Globs` helper, so the flag value has one +meaning across `-a 'static/*'`, `-a 'history/*'`, and `-a '*'` runs. + +Language names are [Linguist keys](https://github.com/github/linguist/blob/master/lib/linguist/languages.yml) +and common aliases resolve automatically: + +```bash +# Canonical names (any case, whitespace is trimmed) +codefang run -a 'history/devs' --languages go,python,typescript . + +# Aliases resolve via enry +codefang run -a 'history/devs' --languages golang,js,ts . + +# Unknown language fails fast at configure time instead of silently +# returning an empty report: +codefang run -a 'history/devs' --languages notalang . +# → Error: failed to configure TreeDiff: tree-diff pathspec: unknown language: "notalang" +``` + +Filename-only languages (e.g. `Dockerfile`, `Makefile`) are also supported: + +```bash +codefang run -a 'history/devs' --languages dockerfile . +``` + +See `specs/optimize-lang/PROPOSAL.md` for the architecture and acceptance-gate +numbers. + +#### Vendor & generated exclusion + +By default, Codefang excludes **vendored dependencies** and **auto-generated +files** from analysis. This matches the convention of every major +single-language analyser (`go vet`, `eslint`, `ruff`, `rubocop`, `scalafix`, +`phpcs`, …) — vendor/generated code is noise for a code-quality report. + +| Flag | Type | Default | Description | +|------|------|---------|-------------| +| `--include-vendored` | `bool` | `false` | Re-include vendored dependencies (detected by enry / Linguist) in analysis. Cross-language: covers `vendor/`, `node_modules/`, `third_party/`, `testdata/`, minified bundles, and more. | +| `--include-generated` | `bool` | `false` | Re-include auto-generated files in analysis. Covers `*.pb.go`, `zz_generated_*.go`, `*_pb2.py`, `*.min.js`, and any file whose first 512 bytes contain a generated-file marker (`DO NOT EDIT`, `Code generated`, etc.). | +| `--extra-excluded-prefixes` | `[]string` | `[]` | Additional UNIX path prefixes to exclude on top of enry heuristics (e.g. `".venv/,target/,build/"`). | + +All three flags apply **identically** to both static and history phases. + +```bash +# default: your own code only +codefang run -a '*' . + +# include vendored deps (node_modules/, vendor/, …) +codefang run -a '*' --include-vendored . + +# restore pre-codefang-2026-04 behaviour (include everything) +codefang run -a '*' --include-vendored --include-generated . + +# skip extras that enry doesn't know about (e.g. Python venv, Rust target/) +codefang run -a '*' --extra-excluded-prefixes '.venv/,target/' . +``` + +!!! warning "Breaking change in 2026-04" + + Earlier versions of Codefang analysed vendored and generated files by + default (they needed the confusingly-named `--skip-blacklist=true` to be + excluded). Starting from 2026-04, defaults flip: vendor / generated + are **excluded by default**. To restore the old behaviour: + + ```bash + codefang run ... --include-vendored --include-generated + ``` + + The deprecated `--skip-blacklist` and `--blacklisted-prefixes` flags + still work with a cobra deprecation warning and will be removed in + the next minor release. Map them to: + + - `--skip-blacklist` → no-op (the new default already excludes) + - `--blacklisted-prefixes X,Y` → `--extra-excluded-prefixes X,Y` + +See `specs/exclude-vendored/PROPOSAL.md` for the full cross-phase design +and `specs/frds/FRD-20260419-exclude-vendored.md` for implementation +details. + +#### Pipeline tuning flags + +| Flag | Type | Default | Description | +|------|------|---------|-------------| +| `--workers` | `int` | `0` | Parallel workers (`0` = CPU count via `GOMAXPROCS`) | +| `--buffer-size` | `int` | `0` | Internal pipeline channel size (`0` = `workers * 2`) | +| `--commit-batch-size` | `int` | `0` | Commits per processing batch (`0` = default 100) | +| `--blob-cache-size` | `string` | `""` | Max blob cache size (e.g. `256MB`, `1GB`; empty = 1 GB) | +| `--diff-cache-size` | `int` | `0` | Max diff cache entries (`0` = default 10000) | +| `--blob-arena-size` | `string` | `""` | Memory arena for blob loading (e.g. `4MB`; empty = 4 MB) | +| `--memory-budget` | `string` | `""` | Memory budget for auto-tuning (e.g. `512MB`, `2GB`). Applies to both history and static phases — auto-derives worker count, spill thresholds, and Go memory limit | + +```bash +# Large repository with constrained memory +codefang run -a 'history/*' --workers 4 --memory-budget 2GB . + +# Static analysis with memory budget (auto-tunes workers and spill thresholds) +codefang run -a 'static/*' --memory-budget 512MB . + +# High-throughput with large caches +codefang run -a 'history/*' --blob-cache-size 2GB --diff-cache-size 50000 . +``` + +#### GC tuning flags + +| Flag | Type | Default | Description | +|------|------|---------|-------------| +| `--gogc` | `int` | `0` | GC target percentage (`0` = Go default 100) | +| `--ballast-size` | `string` | `"0"` | GC ballast allocation (`0` = disabled) | + +#### Checkpoint & resume flags + +| Flag | Type | Default | Description | +|------|------|---------|-------------| +| `--checkpoint` | `bool` | `true` | Enable checkpointing for crash recovery | +| `--checkpoint-dir` | `string` | `""` | Checkpoint directory (default: `~/.codefang/checkpoints`) | +| `--resume` | `bool` | `true` | Resume from checkpoint if available | +| `--clear-checkpoint` | `bool` | `false` | Clear existing checkpoint before run | + +```bash +# Disable checkpointing entirely +codefang run -a history/burndown --checkpoint=false . + +# Resume a previously interrupted run +codefang run -a 'history/*' --resume . + +# Start fresh, clearing old checkpoint data +codefang run -a 'history/*' --clear-checkpoint . +``` + +#### Profiling & debug flags + +| Flag | Type | Default | Description | +|------|------|---------|-------------| +| `--profile` | `bool` | `false` | Start pprof HTTP server on `localhost:6060` and memory watchdog (heap dumps on RSS spikes) | +| `--cpuprofile` | `string` | `""` | Write CPU profile to file | +| `--heapprofile` | `string` | `""` | Write heap profile to file | +| `--debug-trace` | `bool` | `false` | Enable 100% OpenTelemetry trace sampling | + +`--profile` is a **global flag** (available on all subcommands). It enables: + +- A pprof HTTP server on `localhost:6060` for live heap/goroutine/CPU inspection +- A memory watchdog that logs RSS every 2s and dumps heap profiles to `/tmp` when RSS exceeds 4 GiB + +```bash +# CPU profile a large run +codefang run -a 'history/*' --cpuprofile cpu.prof . + +# Live pprof + memory watchdog +codefang run -a 'static/*' --profile --path ~/sources/kubernetes + +# Full debug tracing +codefang run -a 'history/*' --debug-trace . +``` + +--- + +### `codefang mcp` + +Start a Model Context Protocol (MCP) server on stdio transport. This exposes +Codefang analysis capabilities as tools that AI agents can discover and invoke. + +```bash +codefang mcp [flags] +``` + +| Flag | Type | Default | Description | +|------|------|---------|-------------| +| `--debug` | `bool` | `false` | Enable debug logging to stderr | + +The MCP server exposes the following tools: + +| Tool | Description | +|------|-------------| +| `codefang_analyze` | Static code analysis (complexity, cohesion, halstead, comments, imports) | +| `codefang_history` | Git history analysis (burndown, couples, devs, sentiment, etc.) | +| `uast_parse` | Parse source code into Universal AST | + +```bash +# Start MCP server for agent integration +codefang mcp + +# With debug logging +codefang mcp --debug +``` + +--- + +## `uast` + +The UAST binary parses source files into a Universal Abstract Syntax Tree and +provides querying, diffing, and exploration utilities. + +### `uast parse` + +Parse one or more source files into UAST format. + +```bash +uast parse [files...] [flags] +``` + +| Flag | Short | Type | Default | Description | +|------|-------|------|---------|-------------| +| `--language` | `-l` | `string` | `""` | Force language detection override | +| `--output` | `-o` | `string` | `""` | Output file (default: stdout) | +| `--format` | `-f` | `string` | `json` | Output format: `json`, `compact`, `tree` | +| `--progress` | `-p` | `bool` | `false` | Show progress for multiple files | +| `--all` | | `bool` | `false` | Parse all source files in the codebase recursively | + +```bash +# Parse a single file +uast parse main.go + +# Parse all Go files with progress +uast parse -p *.go + +# Force language detection +uast parse -l go main.c + +# Read from stdin +cat main.go | uast parse - + +# Save to file +uast parse -o output.json main.go + +# Parse entire codebase +uast parse --all +``` + +--- + +### `uast query` + +Query UAST nodes using the functional DSL. + +```bash +uast query [files...] [flags] +``` + +| Flag | Short | Type | Default | Description | +|------|-------|------|---------|-------------| +| `--input` | `-i` | `string` | `""` | Input file (UAST JSON or source code) | +| `--output` | `-o` | `string` | `""` | Output file (default: stdout) | +| `--format` | `-f` | `string` | `json` | Output format: `json`, `compact`, `count` | +| `--interactive` | `-t` | `bool` | `false` | Interactive query mode | + +```bash +# Find all functions +uast query "filter(.type == 'Function')" main.go + +# Find exported items +uast query "filter(.roles has 'Exported')" *.go + +# Count all nodes +uast query "reduce(count)" main.go + +# Query from piped UAST JSON +uast parse main.go | uast query "filter(.type == 'Call')" - + +# Count format +uast query -f count "filter(.type == 'Function')" main.go + +# Interactive mode +uast query -t main.go +``` + +??? info "DSL Syntax Reference" + + | Expression | Description | + |-----------|-------------| + | `filter(.type == "Function")` | Filter by node type | + | `filter(.type == "Call")` | Find function calls | + | `filter(.type == "Identifier")` | Find identifiers | + | `filter(.type == "Literal")` | Find literal values | + | `filter(.roles has "Exported")` | Filter by role annotation | + | `reduce(count)` | Count matching nodes | + +--- + +### `uast diff` + +Compare two files at the UAST structural level and report changes. + +```bash +uast diff [flags] +``` + +| Flag | Short | Type | Default | Description | +|------|-------|------|---------|-------------| +| `--output` | `-o` | `string` | `""` | Output file (default: stdout) | +| `--format` | `-f` | `string` | `unified` | Output format: `unified`, `summary`, `json` | + +```bash +# Unified diff output (default) +uast diff old.go new.go + +# Summary of change types +uast diff -f summary old.go new.go + +# Machine-readable JSON +uast diff -f json old.go new.go + +# Save to file +uast diff -o changes.json -f json old.go new.go +``` + +--- + +### `uast explore` + +Start an interactive session for exploring the UAST structure of a file. + +```bash +uast explore [file] [flags] +``` + +| Flag | Short | Type | Default | Description | +|------|-------|------|---------|-------------| +| `--language` | `-l` | `string` | `""` | Force language detection override | + +```bash +uast explore main.go +``` + +Available interactive commands: + +| Command | Description | +|---------|-------------| +| `tree` | Show AST tree structure | +| `stats` | Show node type statistics | +| `find ` | Find nodes by type | +| `query ` | Execute a DSL query | +| `help` | Show available commands | +| `quit` | Exit exploration | + +--- + +### `uast server` + +Start an HTTP development server that provides UAST parsing and querying +via a REST API. + +```bash +uast server [flags] +``` + +| Flag | Short | Type | Default | Description | +|------|-------|------|---------|-------------| +| `--port` | `-p` | `string` | `8080` | Port to listen on | +| `--static` | `-s` | `string` | `""` | Directory to serve static files from | + +```bash +# Start on default port +uast server + +# Custom port with static file serving +uast server -p 3000 -s ./web +``` + +**API Endpoints:** + +| Method | Path | Description | +|--------|------|-------------| +| `POST` | `/api/parse` | Parse source code to UAST | +| `POST` | `/api/query` | Query a UAST with DSL expression | +| `GET` | `/api/mappings` | List available language mappings | +| `GET` | `/api/mappings/` | Get a specific language mapping | + +--- + +### `uast analyze` + +Quick analysis shortcut that parses and analyzes source files in one step. + +```bash +uast analyze [files...] [flags] +``` + +| Flag | Short | Type | Default | Description | +|------|-------|------|---------|-------------| +| `--output` | `-o` | `string` | `""` | Output file (default: stdout) | +| `--format` | `-f` | `string` | `text` | Output format: `text`, `json`, `html` | + +```bash +# Analyze a single file +uast analyze main.go + +# Analyze all Go files as JSON +uast analyze -f json *.go + +# Generate an HTML report +uast analyze -o report.html -f html *.go +``` + + +--- + +# File: site/explanation/complexity.md + +# Understanding complexity metrics + +This page explains the mental model behind the complexity analyzer: what +each metric means, why the analyzer measures three dimensions, and how +cyclomatic and cognitive complexity differ. For configuration keys and the +output schema, see the [Complexity reference](../analyzers/complexity.md). + +--- + +## What it measures + +The complexity analyzer measures three dimensions of code complexity from +source code: **cyclomatic complexity**, **cognitive complexity**, and +**nesting depth**. It operates on the UAST representation of your source +files. + +### Cyclomatic complexity (McCabe, 1976) + +Counts the number of linearly independent paths through a function's control flow graph. + +For Go code, the analyzer follows the same practical counting model used by `gocyclo`: + +- Base score is `1` +- `if`, `for` / `range` add `+1` +- Non-default `case` adds `+1` +- Each `&&` and `||` adds `+1` +- `default` case does **not** add complexity + +!!! info "Interpretation" + - **1-10**: Simple, low risk + - **11-20**: Moderate complexity + - **21-50**: High complexity, consider refactoring + - **51+**: Very high risk, untestable + +### Cognitive complexity (SonarSource, 2017) + +Measures how difficult code is for a human to read and understand. Unlike cyclomatic complexity, it penalizes nested structures more heavily and rewards linear sequences of logic. + +Key differences from cyclomatic complexity: + +- Nesting increments add to the score (deeper nesting = higher penalty) +- `else if` adds structural complexity, but avoids extra nesting penalty vs deeply nested `if` +- Logical operator sequences account for readability, not just path count +- Direct recursion adds a penalty + +### Nesting depth + +Tracks the maximum depth of nested control structures within each function. Deep nesting is a strong signal for refactoring. + +--- + +## Validation against golden implementations + +The complexity analyzer is continuously validated against battle-tested Go references: + +- **Cyclomatic parity target:** `gocyclo` (v0.6.0) +- **Cognitive parity target:** `gocognit` (v1.2.1) + +For stabilization, we run a controlled methodology sample through both references and `codefang`, then align discrepancies until parity is reached. + +--- + +## Use cases + +- **Code review gates**: Reject pull requests where any function exceeds a cyclomatic complexity threshold. +- **Refactoring prioritization**: Sort functions by cognitive complexity to find the hardest-to-understand code. +- **Technical debt tracking**: Monitor complexity trends across releases. +- **Test planning**: Functions with high cyclomatic complexity need more test cases for full path coverage. + +--- + +## Limitations + +- **Language coverage**: Only languages supported by the UAST parser are analyzed. Unsupported files are silently skipped. +- **Generated code**: The analyzer does not distinguish hand-written code from generated code. Consider excluding generated directories. +- **Macros and metaprogramming**: Complexity within macros or template metaprogramming may not be fully captured, since the UAST represents the source as written, not as expanded. +- **Cognitive complexity model**: The cognitive complexity scoring follows the SonarSource specification. Other tools may use slightly different weightings. + +--- + +## See also + +- [Complexity reference](../analyzers/complexity.md) — configuration keys and output schema. +- [Quick start](../getting-started/quickstart.md) — run your first analysis. + + +--- + +# File: site/explanation/cohesion.md + +# Understanding cohesion metrics + +This page explains the mental model behind the cohesion analyzer: what +LCOM-HS measures, how the variable-sharing ratio captures communicational +cohesion, and how to read the scores. For configuration keys and the output +schema, see the [Cohesion reference](../analyzers/cohesion.md). + +--- + +## What it measures + +The cohesion analyzer computes **LCOM-HS (Henderson-Sellers)** and **variable +sharing ratio** metrics to identify files and modules with low internal +cohesion. Low cohesion indicates functions that are poorly related to each +other — a strong signal for refactoring. + +### LCOM-HS (lack of cohesion of methods — Henderson-Sellers) + +LCOM-HS is the industry-standard cohesion metric used by NDepend, JArchitect, and CppDepend. + +**Formula:** + +``` +LCOM = 1 - sum(mA) / (m × a) +``` + +Where: + +- `m` = number of functions in the file +- `a` = number of distinct variables across all functions +- `mA` = for each variable, the count of functions that reference it +- `sum(mA)` = sum of access counts across all variables + +**Range:** 0.0 (perfect cohesion) to 1.0 (no cohesion). + +!!! info "Interpretation" + - **LCOM ≤ 0.3** (green): Excellent cohesion — functions share most variables. + - **LCOM ≤ 0.6** (yellow): Moderate cohesion — some functions may be loosely related. + - **LCOM > 0.6** (red): Poor cohesion — functions share few variables and may belong in separate files. + +### Cohesion score + +A convenience inversion of LCOM-HS: `cohesion_score = 1.0 - LCOM`. Higher is better. + +| Cohesion Score | Assessment | +|---|---| +| ≥ 0.7 | Excellent | +| ≥ 0.4 | Good | +| ≥ 0.3 | Fair | +| < 0.3 | Poor | + +### Function-level cohesion (variable sharing ratio) + +For each function, the analyzer computes what fraction of its variables are shared with at least one other function in the same file. This measures **communicational cohesion** (Yourdon-Constantine level 5). + +**Formula:** + +``` +function_cohesion = shared_vars / total_unique_vars +``` + +Where: + +- `shared_vars` = variables in this function that also appear in at least one other function +- `total_unique_vars` = total distinct variables in this function +- Functions with no variables receive a score of 1.0 (trivial, no penalty) + +**Range:** 0.0 (completely isolated function) to 1.0 (all variables shared). + +| Function Cohesion | Assessment | +|---|---| +| ≥ 0.6 | Excellent — function shares most variables with the module | +| ≥ 0.4 | Good — reasonable sharing with room for improvement | +| ≥ 0.3 | Fair — consider refactoring | +| < 0.3 | Poor — function is isolated from the module | + +--- + +## Use cases + +- **Refactoring targets**: Find functions with low sharing ratio that do unrelated work and should be split. +- **Architecture reviews**: Validate that files follow the Single Responsibility Principle by checking LCOM-HS scores. +- **Code quality tracking**: Monitor cohesion trends over time to catch degradation. +- **Code review**: Use per-function cohesion scores to identify newly added functions that lack cohesion with existing code. + +--- + +## Methodology references + +- **LCOM-HS**: Henderson-Sellers, B. (1996). *Object-Oriented Metrics: Measures of Complexity*. Prentice Hall. The Henderson-Sellers variant normalizes LCOM to [0, 1] and is the variant used by NDepend, JArchitect, and CppDepend. +- **Variable Sharing Ratio**: Measures communicational cohesion per the Yourdon-Constantine classification. A function that shares all its variables with other functions in the module exhibits high communicational cohesion. + +--- + +## Limitations + +- **Language scope**: Works with any language supported by the UAST. Best results with languages that have explicit variable declarations and function definitions. +- **Accessor methods**: Simple getters/setters may inflate cohesion since each touches only one shared field. +- **Variable naming**: The analyzer uses lexical variable names. Different variables with the same name across functions will be counted as shared. +- **Single-function files**: Files with only one function always receive perfect cohesion (LCOM = 0.0, cohesion = 1.0) since there are no other functions to compare against. +- **Trivial functions**: Functions with no variables receive a cohesion score of 1.0 to avoid penalizing simple utility functions. + +--- + +## See also + +- [Cohesion reference](../analyzers/cohesion.md) — configuration keys and output schema. +- [Quick start](../getting-started/quickstart.md) — run your first analysis. + + +--- + +# File: site/explanation/halstead.md + +# Understanding Halstead metrics + +This page explains the mental model behind the Halstead analyzer: what +operators and operands are, which derived metrics matter most, and how the +counting policy keeps results stable across languages. For configuration keys +and the output schema, see the [Halstead reference](../analyzers/halstead.md). + +--- + +## What it measures + +The Halstead analyzer computes **Halstead complexity metrics** (1977) based on +operator and operand counts extracted from the UAST. These metrics provide an +objective, quantitative assessment of program size and complexity. + +The Halstead model treats a program as a sequence of **operators** (keywords, symbols, function calls) and **operands** (variables, literals, constants). From these counts, it derives: + +| Metric | Formula | Description | +|---|---|---| +| **n1** | -- | Number of distinct operators | +| **n2** | -- | Number of distinct operands | +| **N1** | -- | Total number of operators | +| **N2** | -- | Total number of operands | +| **Vocabulary** (n) | n1 + n2 | Distinct tokens used | +| **Length** (N) | N1 + N2 | Total tokens in the program | +| **Volume** (V) | N * log2(n) | Information content in bits | +| **Difficulty** (D) | (n1/2) * (N2/n2) | Error proneness | +| **Effort** (E) | D * V | Mental effort to implement | +| **Time** (T) | E / 18 | Estimated implementation time (seconds) | +| **Bugs** (B) | V / 3000 | Estimated delivered bugs | + +!!! info "Key insight" + **Volume** measures the size of the implementation. **Difficulty** captures how error-prone it is. Their product, **Effort**, is the best single number for comparing overall complexity. + +### Counting policy + +Codefang computes Halstead metrics from UAST with a lexical-first policy: + +- Counts lexical operands (`Identifier`, `Literal`, `Field`) and operator nodes/roles. +- Excludes structural wrappers and declaration-only artifacts from operand counts. +- Uses explicit operator properties when available, then token extraction fallback. + +This improves stability across languages and avoids pseudo-operands like structural `Parameter` nodes. + +--- + +## Use cases + +- **Effort estimation**: Use the Effort metric to compare the relative complexity of different modules or features. +- **Bug prediction**: The Bugs metric provides a rough upper bound on expected defects. Modules with high estimated bugs warrant more thorough testing. +- **Code review guidance**: Functions with high Difficulty scores are more error-prone and deserve extra scrutiny. +- **Language comparison**: Halstead metrics allow cross-language comparisons since they are based on abstract operator/operand counts. + +--- + +## Limitations + +- **Cross-tool variance**: Language-specific tools (for example, Python- or ESTree-only analyzers) can differ in tokenization and counting policy. Compare trends within one toolchain rather than mixing absolute numbers across tools. +- **Operator classification**: UAST-based classification targets cross-language consistency, not byte-for-byte parity with each language parser. +- **Estimation accuracy**: The Bugs and Time formulas are empirical approximations from the 1970s. Treat them as relative indicators, not precise predictions. +- **Macro expansion**: Halstead metrics count tokens as written, not as expanded. Heavy use of macros or code generation can skew results. +- **Comments excluded**: Comments and whitespace are excluded from Halstead counts (by design). + +--- + +## See also + +- [Halstead reference](../analyzers/halstead.md) — configuration keys and output schema. +- [Quick start](../getting-started/quickstart.md) — run your first analysis. + + +--- + +# File: site/explanation/comments.md + +# Understanding documentation coverage + +This page explains the mental model behind the comments analyzer: what +documentation coverage means, how the analyzer distinguishes comment types, +and what a healthy comment density looks like. For configuration keys and the +output schema, see the [Comments reference](../analyzers/comments.md). + +--- + +## What it measures + +The comments analyzer evaluates **documentation coverage** by examining comment +presence, placement, and density across your source code. It identifies +functions and classes that lack documentation and measures overall +documentation quality. + +### Documentation coverage + +The percentage of public functions, methods, and classes that have an associated documentation comment. A comment is "associated" when it immediately precedes the declaration. + +### Comment placement quality + +Not all comments are equally useful. The analyzer distinguishes: + +- **Doc comments**: Comments immediately before a function/class declaration +- **Inline comments**: Comments within function bodies +- **Orphan comments**: Comments not associated with any code structure + +### Comment density + +The ratio of comment lines to total lines of code. Provides a quick overall measure of how well-documented a file is. + +!!! info "Healthy ratios" + A comment density between **15-30%** is generally considered healthy. Below 10% suggests under-documentation; above 40% may indicate stale or redundant comments. + +--- + +## Use cases + +- **Documentation enforcement**: Require minimum documentation coverage in CI/CD pipelines. +- **Onboarding assessment**: Measure how well-documented a codebase is before new team members join. +- **API surface quality**: Ensure all exported/public functions have doc comments. +- **Stale comment detection**: Identify files with unusually high comment density that may contain outdated comments. + +--- + +## Limitations + +- **Comment quality**: The analyzer measures comment *presence* and *placement*, not comment *quality*. A comment saying `// do the thing` counts the same as a thorough description. +- **Language conventions**: Documentation comment conventions vary by language (e.g., `///` in Rust, `"""` in Python, `//` in Go). The UAST normalizes these, but edge cases in less common languages may be missed. +- **Auto-generated docs**: Comments generated by tools (e.g., protobuf stubs) are counted the same as hand-written documentation. +- **Multilingual comments**: Comment text is not analyzed for language quality or correctness. + +--- + +## See also + +- [Comments reference](../analyzers/comments.md) — configuration keys and output schema. +- [Quick start](../getting-started/quickstart.md) — run your first analysis. + + +--- + +# File: site/explanation/imports.md + +# Understanding import and dependency analysis + +This page explains the mental model behind the imports analyzer: what it +measures in static and history modes, and how the history-mode aggregation +pipeline works. For configuration keys and the output schema, see the +[Imports reference](../analyzers/imports.md). + +--- + +## What it measures + +The imports analyzer extracts **import and dependency information** from source +code using UAST parsing. It operates in both static mode (single-file analysis) +and history mode (tracking import usage per developer over time). + +### Static mode + +- **Import list**: All imports/dependencies declared in each file +- **Language detection**: Automatically detects the language and normalizes import paths +- **Dependency graph**: Maps which files depend on which packages + +### History mode + +Tracks import usage across Git history, producing a per-developer, per-language, per-tick breakdown of which dependencies each developer introduces or modifies. This reveals: + +- **Dependency adoption timeline**: When new libraries were introduced +- **Developer expertise signals**: Which developers work with which dependencies +- **Technology spread**: How quickly new dependencies propagate across the team + +--- + +## Architecture (history mode) + +The imports history analyzer follows the **TC/Aggregator pattern**: + +1. **Consume phase**: For each commit, `Consume()` extracts imports from changed files via parallel UAST parsing and returns them as `TC{Data: []ImportEntry}`. Each `ImportEntry` carries a language and import path. The analyzer retains no per-commit state; only the UAST parser is kept as working state. +2. **Aggregation phase**: An `imports.Aggregator` collects TCs into a 4-level `Map` (author -> language -> import -> tick -> count) using `SpillStore[Map]`. The `AuthorID` and `Tick` from each TC index the entries correctly. +3. **Serialization phase**: `SerializeTICKs()` merges all tick data back into the full `Map` with metadata (`author_index`, `tick_size`), then delegates to `Serialize()` for JSON, YAML, binary, or HTML plot output. + +This separation enables streaming output, budget-aware memory spilling, and decoupled aggregation. + +--- + +## Use cases + +- **Dependency auditing**: List all third-party dependencies used in a project. +- **Developer profiling**: Understand which developers work with which frameworks and libraries. +- **Technology adoption tracking**: Monitor when and how quickly new dependencies spread across the team. +- **License compliance**: Extract the full dependency list for license scanning pipelines. +- **Architecture enforcement**: Detect unauthorized imports from forbidden packages. + +--- + +## Limitations + +- **Language support**: Only languages with UAST parser support are analyzed. Unsupported file types are silently skipped. +- **Dynamic imports**: Runtime or dynamic imports (e.g., Python's `importlib.import_module()`, JavaScript's `import()`) are not detected since they are not present in the static UAST. +- **Transitive dependencies**: Only direct imports are reported. The analyzer does not resolve transitive dependency trees. +- **File size threshold**: Files exceeding `MaxFileSize` (default 1 MB) are skipped to avoid excessive memory usage during parallel extraction. +- **History mode overhead**: History mode creates a UAST parser per fork, which increases memory usage. Tune `Goroutines` based on available memory. + +--- + +## See also + +- [Imports reference](../analyzers/imports.md) — configuration keys and output schema. +- [Quick start](../getting-started/quickstart.md) — run your first analysis. + + +--- + +# File: site/explanation/clones.md + +# Understanding clone detection + +This page explains the mental model behind the clone detection analyzer: what +a clone pair is, the clone-type taxonomy, and why the two-stage MinHash + LSH +design keeps cross-file detection fast. For configuration keys and the output +schema, see the [Clones reference](../analyzers/clones.md). + +--- + +## What it measures + +The clone detection analyzer finds **duplicate and near-duplicate functions** +across your entire codebase using MinHash signatures and Locality-Sensitive +Hashing (LSH). It operates on the UAST representation and detects clones +**cross-file** — not just within a single file. + +### Clone pairs + +A clone pair is two functions whose AST structure is similar enough to be considered duplicated code. Each pair includes a **similarity score** (0.0–1.0) and a **clone type** classification. + +### Clone types + +| Type | Similarity | Description | +|---|---|---| +| **Type-1 (Exact)** | = 1.0 | Identical AST structure and tokens | +| **Type-2 (Renamed)** | ≥ 0.8 | Identical structure, different variable/function names | +| **Type-3 (Near-miss)** | ≥ 0.5 | Similar but modified structure | + +### Clone ratio + +The ratio of detected clone pairs to total functions, computed globally across all files: + +``` +clone_ratio = total_clone_pairs / total_functions +``` + +!!! info "Interpretation" + - **0.0** (green): No duplication detected + - **≤ 0.1** (yellow): Low duplication — few clone pairs + - **≤ 0.3** (red): Moderate duplication — consider refactoring + - **> 0.3**: High duplication — significant refactoring recommended + +--- + +## Methodology + +### Stage 1: per-file signature extraction + +For each source file, the analyzer: + +1. **Parses** the file into a UAST and finds all function/method nodes. +2. **Shingles** each function by performing a pre-order traversal of the AST subtree and extracting sliding windows of `k` consecutive node types (k=5 by default). For example, a function with nodes `[Function, Block, Assignment, Identifier, Call, Return]` produces shingles like `Function|Block|Assignment|Identifier|Call` and `Block|Assignment|Identifier|Call|Return`. +3. **MinHash** compresses each function's shingle set into a fixed-size signature of 128 hash values. Two functions with identical structure produce identical signatures; similar functions produce similar signatures. + +Signatures are exported per-file along with function names. At this stage, no clone detection occurs — comparison is deferred to the aggregation stage. + +### Stage 2: cross-file clone detection + +After all files have been processed, the aggregator: + +1. **Qualifies** function names with their source file path (`path/to/file.go::FuncName`) so that same-named functions across files are distinguishable. +2. **Builds a global LSH index** (16 bands, 8 rows per band) from all collected signatures. LSH hashes each signature's bands into buckets — functions whose signatures share any bucket are candidate clones. +3. **Queries** the LSH index for each function, retrieves candidates above the similarity threshold (0.5), computes exact Jaccard similarity from the full MinHash signatures, and classifies each pair by clone type. +4. **Computes metrics** globally: `clone_ratio = total_pairs / total_functions`. + +This two-stage design means clone detection works across file boundaries. A controller duplicated in `pkg/a/handler.go` and `pkg/b/handler.go` is detected even though each file is parsed independently. + +### Why MinHash + LSH? + +Pairwise comparison of N functions is O(N²). For a project with 100K functions this is infeasible. MinHash compresses each function into a 1 KB signature (128 hashes × 8 bytes), and LSH reduces candidate pair generation to near-linear time by only comparing functions that hash into the same bucket. The combination provides probabilistic guarantees on recall while keeping the analysis fast. + +--- + +## Use cases + +- **Refactoring targets**: Identify exact and near-duplicate functions that should be consolidated into shared utilities. +- **Code review gates**: Flag pull requests that introduce new clone pairs, especially Type-1 (exact) duplicates. +- **Copy-paste detection**: Find controllers, handlers, or boilerplate that were copy-pasted across packages instead of being extracted. +- **Technical debt tracking**: Monitor clone ratio over time to ensure duplication does not grow unbounded. +- **Cross-team coordination**: Detect when multiple teams independently implement the same logic in different packages. + +--- + +## Methodology references + +- **MinHash**: Broder, A. Z. (1997). *On the resemblance and containment of documents*. Proceedings of the Compression and Complexity of Sequences. MinHash estimates Jaccard similarity between sets using compact fixed-size signatures. +- **LSH (Locality-Sensitive Hashing)**: Indyk, P. & Motwani, R. (1998). *Approximate nearest neighbors: towards removing the curse of dimensionality*. STOC. Banded LSH reduces candidate pair generation from O(N²) to near-linear time. +- **AST-based clone detection**: Jiang, L. et al. (2007). *DECKARD: Scalable and accurate tree-based detection of code clones*. ICSE. Tree-based shingling captures structural similarity independent of identifier names. +- **Clone taxonomy**: Roy, C. K. & Cordy, J. R. (2007). *A survey on software clone detection research*. The Type-1/Type-2/Type-3 classification follows the standard clone taxonomy. + +--- + +## Limitations + +- **Language scope**: Works with any language supported by the UAST parser. Unsupported files are silently skipped. +- **Structural only**: Detection is based on AST node types, not token values. Two functions with identical structure but completely different semantics (e.g., add vs multiply with the same nesting shape) may appear as clones. +- **Minimum function size**: Functions smaller than the shingle window (5 AST nodes) produce no shingles and are excluded from clone detection. They are still counted in `total_functions`. +- **Probabilistic recall**: LSH is a probabilistic algorithm. Clone pairs with similarity just above the threshold (0.5) may occasionally be missed. High-similarity pairs (Type-1 and Type-2) have near-perfect recall. +- **Generated code**: The analyzer does not distinguish hand-written code from generated code. Consider excluding generated directories. +- **No Type-4 clones**: Semantic clones (different structure, same behavior) are not detected. The analyzer operates purely on syntactic structure. + +--- + +## See also + +- [Clones reference](../analyzers/clones.md) — configuration keys and output schema. +- [Quick start](../getting-started/quickstart.md) — run your first analysis. + + +--- + +# File: site/explanation/burndown.md + +# Understanding code burndown + +This page explains the mental model behind the burndown analyzer: what code +survival means, the per-file and per-developer breakdowns, and how the +line-tracking algorithm works. For configuration keys and the output schema, +see the [Burndown reference](../analyzers/burndown.md). + +--- + +## What it measures + +The burndown analyzer tracks **code survival over time** by following every +line of code through Git history. It produces burndown charts showing how code +written at different points in time persists, is modified, or is deleted. +Optionally, it tracks per-file and per-developer breakdowns. + +### Global code survival + +A time-series matrix where each row is a sampling point and each column is an age band. The value at `[sample][band]` is the number of lines that were last edited during that band and still survive at that sample point. + +This matrix produces the classic **burndown chart**: stacked area plots showing how much code from each era remains. + +### Per-file burndown + +When `--burndown-files` is enabled, the analyzer produces a separate survival matrix for each file, enabling file-level burndown charts. + +### Per-developer burndown + +When `--burndown-people` is enabled, the analyzer tracks which developer last edited each line. This reveals: + +- **Developer survival rates**: How much of each developer's code persists +- **Interaction matrix**: Which developers modify each other's code + +### File ownership + +When both `--burndown-files` and `--burndown-people` are enabled, the analyzer computes per-file ownership by iterating the live line segments in each file's internal tree. Each segment stores a packed `[author|tick]` value, from which the author ID is extracted to produce a `file -> author -> line_count` mapping. + +!!! note "Ownership requires developer tracking" + File ownership data is only available when `--burndown-people` is enabled. Without developer tracking, no author information is stored in the line segments, so file ownership will be empty. + +!!! warning "Memory usage" + Per-file and per-developer tracking significantly increases memory usage. For repositories with more than 100k commits, consider enabling hibernation (on by default). + +--- + +## How it works + +### Algorithm overview + +1. **Commit traversal**: Commits are processed sequentially in topological order. For each commit, the analyzer diffs the parent tree against the current tree to find inserted, deleted, and modified lines. + +2. **Line tracking**: Each line in every file is tracked using an augmented balanced binary tree (treap). Each tree node stores a contiguous range of lines with a packed value encoding `[author_id | creation_tick]`. When lines are inserted or deleted, the tree is split and merged to maintain the correct line-to-value mapping. + +3. **Sparse history accumulation**: At each commit, the analyzer records deltas in a sparse history structure: `map[currentTick]map[creationTick]lineCountDelta`. This captures how many lines created at `creationTick` are alive at `currentTick`. + +4. **Tick aggregation**: Sparse deltas from individual commits are accumulated into tick-level snapshots by a streaming aggregator. At each sampling boundary, the aggregator emits a `TickResult` containing the full sparse history state. + +5. **Dense matrix conversion**: The sparse history is converted to a dense matrix (`DenseHistory`) via `groupSparseHistory()`. This involves: + - **Tick normalization**: Map arbitrary tick values to sorted indices + - **Granularity grouping**: Collapse adjacent creation ticks into age bands of width `granularity` + - **Forward-fill**: For each age band, carry forward the last known value across sampling gaps to produce a complete matrix + +6. **Metrics computation**: The dense matrix is used to compute aggregate statistics (survival rate, peak lines, current lines), per-developer survival, and the interaction matrix. + +### Key parameters + +- **Granularity** controls the width of each age band (in ticks). Higher values produce fewer, wider bands. +- **Sampling** controls how frequently snapshots are taken (in ticks). Higher values reduce the number of data points. +- **Tick size** defaults to a 24-hour window. All commits within the same day share one tick. + +--- + +## Use cases + +- **Project health monitoring**: Track the overall code survival rate. A declining rate may indicate churn or instability. +- **Developer contribution analysis**: Understand whose code persists and who rewrites existing code. +- **Code age visualization**: Generate burndown charts showing how much ancient code remains in the codebase. +- **Refactoring impact**: Measure how much code a refactoring effort actually replaced. +- **Team dynamics**: The interaction matrix reveals collaboration patterns -- who reviews and modifies whose code. + +--- + +## Limitations + +- **Sequential processing**: Burndown tracks cumulative per-line state across all commits and must process commits sequentially. It cannot be parallelized across commits (though per-file processing within a commit is parallelized via goroutines). +- **Memory intensive**: Every line in every file is tracked throughout history. Large repositories (100k+ commits, 10k+ files) can require several GB of RAM. Use hibernation options to manage memory. +- **Binary files excluded**: Binary files are automatically skipped since they cannot be meaningfully diff'd line-by-line. +- **Rename tracking**: File renames are tracked using Git's rename detection. If Git does not detect a rename (e.g., content changed significantly), the file appears as a deletion + insertion. +- **Tick resolution**: The default 24-hour tick means that all commits within the same day share one tick. Sub-day granularity is not supported. +- **File ownership without developer tracking**: File ownership data requires `--burndown-people`. Without it, the `file_survival` array will have empty ownership maps. + +--- + +## See also + +- [Burndown reference](../analyzers/burndown.md) — configuration keys and output schema. +- [Quick start](../getting-started/quickstart.md) — run history analysis. + + +--- + +# File: site/explanation/developers.md + +# Understanding developer analysis + +This page explains the mental model behind the developers analyzer: what +per-developer statistics it computes, how the bus factor is derived, and how +the aggregation pipeline is structured. For configuration keys and the output +schema, see the [Developers reference](../analyzers/developers.md). + +--- + +## What it measures + +The developers analyzer computes **per-developer contribution statistics** +across Git history, including commit counts, line changes, language breakdown, +bus factor risk, and activity time series. + +### Developer statistics + +For each developer (identified by the identity detector): + +- **Commits**: Total number of commits +- **Lines added / removed / changed**: Aggregate line statistics +- **Net lines**: `added - removed`, showing net contribution +- **Languages**: Breakdown of line changes per programming language +- **Active period**: First and last ticks of activity, number of active ticks + +### Language statistics + +Aggregated across all developers: + +- **Total lines per language**: Lines added (for backward compatibility) +- **Total contribution per language**: Lines added + removed, used for bus factor and ownership calculations +- **Contributors per language**: Which developers contribute to each language, measured by total contribution (added + removed) + +### Bus factor + +Knowledge concentration risk per language, following the [CHAOSS Contributor Absence Factor](https://chaoss.community/kb/metric-bus-factor/) methodology. + +For each language, the analyzer computes: + +- **Bus factor number**: The smallest number of contributors responsible for 50% of total contributions (added + removed). This is the CHAOSS standard metric. +- **Total contributors**: Number of unique contributors to that language. +- **Primary/secondary owner**: The top two contributors and their ownership percentages. +- **Risk level**: Based on primary owner concentration. + +!!! danger "Risk levels" + - **CRITICAL** (>= 90%): A single developer owns nearly all the code + - **HIGH** (>= 80%): Very concentrated ownership + - **MEDIUM** (>= 60%): Moderate concentration + - **LOW** (< 60%): Healthy distribution + +A **project-level bus factor** is also computed across all developers and all languages, reported in the aggregate section. + +### Activity time series + +Per-tick commit counts broken down by developer. Shows contribution velocity over time. + +### Code churn (line velocity) + +Per-tick lines added and removed. This measures raw line velocity — the volume of code changes over time. High values may indicate refactoring, feature development, or instability. + +!!! note "Terminology" + In academic literature, "code churn" specifically refers to recently-written code that is quickly rewritten. This analyzer measures the broader concept of line velocity (total additions and removals per time period). + +--- + +## Architecture + +The developers analyzer is built on the **BaseHistoryAnalyzer** and **GenericAggregator** foundations: + +1. **Consume phase**: `Consume()` extracts author ID, line stats, and language breakdown, delegating state storage to the generic aggregator framework. The analyzer retains minimal internal state. +2. **Aggregation phase**: The `GenericAggregator` automatically handles per-commit memory spilling and per-tick grouping using pure function hooks (`extractTC`, `mergeState`, `buildTick`), eliminating custom state management boilerplate. +3. **Metrics & UI phase**: A pure function pipeline (`ComputeAllMetrics`) generates the output structures for JSON, YAML, and text, while declarative chart builders (`plotpage.BuildBarChart`, `plotpage.BuildLineChart`) render the HTML visualizations. + +This unified approach significantly reduces boilerplate while maintaining full support for streaming output, decoupled aggregation, and budget-aware memory spilling. + +--- + +## Use cases + +- **Team assessment**: Understand who contributes what, in which languages, and when. +- **Bus factor analysis**: Identify languages or components where a single developer departure would create critical knowledge gaps. The CHAOSS bus factor number tells you how many people need to leave before 50% of knowledge is lost. +- **Activity monitoring**: Track developer engagement over time. Declining activity may signal burnout or attrition risk. +- **Language migration tracking**: Monitor the adoption of a new language by watching language statistics over time. +- **Onboarding evaluation**: Measure how quickly new team members ramp up by comparing their activity curves. +- **Code churn analysis**: Detect periods of high line velocity that may correlate with instability or deadline pressure. + +--- + +## Limitations + +- **Identity resolution**: Developer identity is determined by the identity detector (email-based by default). Multiple email addresses for the same person will appear as separate developers unless a mailmap is configured. +- **Merge commits**: By default, merge commits are processed only once (first encounter). Trivial merges are skipped unless `ConsiderEmptyCommits` is enabled. +- **Line attribution**: Lines are attributed to the commit author, not the committer. In workflows with heavy rebasing, this may differ from expectations. +- **Contribution measurement**: Contributions are measured as lines added + lines removed. This gives fair credit to refactoring work but does not distinguish between code complexity or quality. +- **Active developer threshold**: "Active developers" are those with commits in the most recent 90-day window (when tick size is known). Falls back to the recent 30% of the analysis period when tick size is unavailable. This threshold is not configurable. + +--- + +## See also + +- [Developers reference](../analyzers/developers.md) — configuration keys and output schema. +- [Quick start](../getting-started/quickstart.md) — run developer analysis. + + +--- + +# File: site/explanation/couples.md + +# Understanding file and developer coupling + +This page explains the mental model behind the couples analyzer: what file and +developer coupling mean, how coupling strength is computed, and how the +aggregation pipeline is structured. For configuration keys and the output +schema, see the [Couples reference](../analyzers/couples.md). + +--- + +## What it measures + +The couples analyzer identifies **file coupling** and **developer coupling** by +analyzing co-change patterns across Git history. Files that frequently change +together are likely coupled; developers who frequently touch the same files are +collaborating (or competing). + +### File coupling + +A square matrix where each cell `[i][j]` counts the number of commits in which files `i` and `j` both appeared. High co-change counts indicate tight coupling between files. + +**Coupling strength** is computed using the code-maat formula: + +``` +strength = co_changes / average(revisions_file1, revisions_file2) +``` + +Where `revisions_fileN` is the diagonal element of the file matrix (the file's self-change count). The result is capped at 1.0. This normalizes coupling by the average activity of both files, so a pair of files that change together 10 times out of 20 average revisions scores 50%. + +!!! info "Changeset size filter" + Commits touching more than **1000 files** are excluded from coupling analysis. Mass changes (e.g., formatting, license updates, dependency bumps) produce noise rather than meaningful coupling signal. + +### Developer coupling + +A matrix where each cell `[i][j]` counts the number of times developers `i` and `j` committed to the same file. Developer coupling strength uses the same code-maat formula normalized by each developer's individual commit activity. + +### File ownership + +For each tracked file, the analyzer reports its line count and the number of distinct contributors. Single-owner files are flagged as bus-factor risks. + +### Rename tracking + +The analyzer tracks file renames to avoid counting a renamed file as both a deletion and an insertion, which would break coupling chains. + +--- + +## Use cases + +- **Architecture analysis**: Highly coupled file pairs that span package boundaries may indicate architectural violations. +- **Dependency discovery**: Find implicit dependencies that are not captured by import statements. If two files always change together, they are coupled even without explicit imports. +- **Module boundary validation**: Files within the same module should be more coupled than files across modules. Cross-module coupling is a design smell. +- **Team topology**: Developer coupling reveals who collaborates with whom. This can inform team structure decisions. +- **Change impact prediction**: When modifying a file, the coupling matrix predicts which other files are likely to need changes. +- **Bus factor assessment**: File ownership data identifies single-owner files that represent knowledge concentration risks. + +--- + +## Architecture + +The couples analyzer follows the **TC/Aggregator pattern**: + +1. **Consume phase**: Each commit produces a `TC{Data: *CommitData}` containing the coupling context (list of co-changed files), per-file author touch counts (always 1 per file per commit), rename pairs, and whether the author's commit count was incremented. Commits exceeding `CouplesMaximumMeaningfulContextSize` (1000 files) are skipped, producing an empty coupling context. + +2. **Aggregation phase**: The `Aggregator` accumulates the file co-occurrence matrix using `SpillStore[map[string]int]`, along with per-person file touch counts (`people`), commit counts (`peopleCommits`), and rename tracking. When memory pressure exceeds `SpillBudget`, the file coupling matrix is spilled to disk via gob encoding. `Collect()` merges spilled data using additive merge semantics. + +3. **Serialization phase**: `ticksToReport()` reconstructs the full report (PeopleMatrix, PeopleFiles, Files, FilesLines, FilesMatrix, ReversedPeopleDict) from aggregated TICKs, then `ComputeAllMetrics()` produces the final typed output for any format (text, plot, JSON, YAML, binary). + +**Working state** (`merges`, `seenFiles`) stays in the analyzer for merge-mode dedup across commits. **Accumulated output** (file couplings, people maps, renames) is owned entirely by the aggregator. + +--- + +## Methodology + +The coupling strength formula follows the **code-maat** academic standard (Adam Tornhill, "Your Code as a Crime Scene"): + +``` +degree = shared_revisions / average(revisions_A, revisions_B) +``` + +This measures what fraction of a file pair's average activity is shared. A coupling strength of 80% means the pair changes together in 80% of their average revision count. + +The aggregate `avg_coupling_strength` is the mean of all per-pair coupling strengths (not the mean of raw co-change counts). + +**Highly coupled pairs** are those with 10 or more co-changes (raw count threshold). + +--- + +## Limitations + +- **Large commits excluded**: Commits touching more than 1000 files are skipped to avoid noise from mass changes (formatting, license headers, dependency updates). +- **No temporal decay**: All commits are weighted equally. A coupling from long ago counts the same as a recent one. Consider filtering by date range for recent coupling analysis. +- **Merge commits**: Merge commits are processed only once (first encounter) to avoid double-counting changes that appear in multiple branches. +- **File deletions**: Deleted files are included in the coupling matrix during the analysis window but may not appear in the final output if they no longer exist at HEAD. + +--- + +## See also + +- [Couples reference](../analyzers/couples.md) — configuration keys and output schema. +- [Quick start](../getting-started/quickstart.md) — run history analysis. + + +--- + +# File: site/explanation/file-history.md + +# Understanding file history analysis + +This page explains the mental model behind the file history analyzer: what +per-file lifecycle data it records, how hotspots and churn are derived, and how +rename detection maintains continuity. For configuration keys and the output +schema, see the [File history reference](../analyzers/file-history.md). + +--- + +## What it measures + +The file history analyzer tracks the **lifecycle of every file** through Git +history, recording which commits touched each file, which developers modified +it, and aggregating line statistics per contributor. It supports rename +detection to maintain continuity across file moves. + +### Per-file commit history + +For each file present in the repository at HEAD, the analyzer records the ordered list of commits that created, modified, or deleted it. + +### Per-file contributor breakdown + +For each file, a map of developer IDs to line statistics (added, removed, changed). This shows who contributed what to each file. + +### Code hotspots + +Files with high commit counts are flagged as hotspots with risk levels: + +!!! warning "Hotspot risk levels" + - **CRITICAL**: >= 50 commits + - **HIGH**: >= 30 commits + - **MEDIUM**: >= 15 commits + - Files below 15 commits are not flagged + +### File churn + +A composite score combining commit frequency and line change volume. High-churn files may indicate instability or areas of active development. + +### Rename support + +When Git detects a file rename (e.g., `old/path.go` to `new/path.go`), the analyzer transfers the full history from the old path to the new path, maintaining a continuous record. + +--- + +## Use cases + +- **Hotspot identification**: Find the files that change most often. These are the highest-risk files for bugs and the best candidates for extra test coverage and code review scrutiny. +- **Code ownership mapping**: Determine who is the primary contributor for each file to establish code ownership. +- **Onboarding guides**: New team members can see which files each developer owns to know who to ask about specific code. +- **Refactoring ROI**: Identify files with both high churn and many contributors -- refactoring these produces the largest payoff. +- **Risk assessment**: Files with a single contributor and high commit counts are both hot and concentrated -- a bus factor risk. + +--- + +## Limitations + +- **HEAD-only output**: Only files present at HEAD are included in the final output. Files that were deleted before the last commit are tracked during analysis but excluded from results. +- **Merge handling**: Merge commits are processed only once to avoid double-counting. Non-merge context changes in merge commits are skipped. +- **Rename detection**: Depends on Git's rename detection heuristics. If a file is both renamed and heavily modified in the same commit, Git may not detect the rename. +- **Churn score formula**: The churn score is `commit_count + (added + removed + changed) / 100`. This is a simple heuristic, not a rigorous statistical measure. +- **No time-series**: Unlike the burndown analyzer, file history does not produce time-series data. It provides aggregate per-file statistics only. + +--- + +## See also + +- [File history reference](../analyzers/file-history.md) — configuration keys and output schema. +- [Quick start](../getting-started/quickstart.md) — run history analysis. + + +--- + +# File: site/explanation/quality.md + +# Understanding code quality over time + +This page explains the mental model behind the code quality analyzer: which +static metrics it tracks per commit and how it aggregates them per tick. For +configuration keys and the output schema, see the +[Quality reference](../analyzers/quality.md). + +--- + +## What it measures + +The code quality analyzer tracks **complexity, Halstead metrics, comment +quality, and cohesion** across Git history. For each commit, it runs four +static analyzers on UAST-parsed changed files and produces per-file quality +metrics. + +### Per-file metrics + +For each changed file in a commit, the analyzer computes: + +| Category | Metrics | +|---|---| +| **Complexity** | Cyclomatic complexity, cognitive complexity, max single-function complexity, function count | +| **Halstead** | Volume, effort, delivered bugs | +| **Comments** | Overall comment score, documentation coverage | +| **Cohesion** | Cohesion score | + +### Statistical aggregation + +Per-tick statistics are computed from per-file arrays: + +- **Mean, median, P95, max** for complexity and Halstead volume +- **Sum** for delivered bugs and Halstead volume +- **Min** for comment score and cohesion (worst-case tracking) +- **Total** files analyzed and functions counted + +--- + +## Architecture + +The quality analyzer follows the **TC/Aggregator pattern**: + +1. **Consume phase**: For each commit, `Consume()` runs four static analyzers (complexity, Halstead, comments, cohesion) on each changed file's UAST, returning per-file metrics as a `TC{Data: *TickQuality}`. The analyzer retains no per-commit state. +2. **Aggregation phase**: A `quality.Aggregator` collects TCs, merges `TickQuality` data by time bucket (tick), and produces `TICK` results. +3. **Serialization phase**: `SerializeTICKs()` converts aggregated TICKs into JSON, YAML, binary, or HTML plot output via `ComputeAllMetrics()`. + +This separation enables streaming output, budget-aware memory spilling, and decoupled aggregation. + +--- + +## See also + +- [Quality reference](../analyzers/quality.md) — configuration keys and output schema. +- [Quick start](../getting-started/quickstart.md) — run history analysis. + + +--- + +# File: site/explanation/sentiment.md + +# Understanding comment sentiment analysis + +This page explains the mental model behind the sentiment analyzer: how it +extracts and filters comments, how VADER and the multilingual lexicon score +them, and the software-engineering domain adjustments that keep technical +language from skewing the result. For configuration keys and the output schema, +see the [Sentiment reference](../analyzers/sentiment.md). + +--- + +## What it measures + +The sentiment analyzer classifies **comment sentiment** across Git history. For +each commit, it extracts new or changed comments via UAST parsing, filters out +noise, and classifies the remaining comments as positive, negative, or neutral. +This reveals how developer sentiment evolves over time. + +### Per-commit comment extraction + +For each commit, the analyzer: + +1. Computes the UAST diff between the old and new file versions +2. Extracts comment nodes from the new UAST +3. Merges adjacent comment lines into single blocks +4. Filters out noise (short comments, license headers, function signatures) +5. Returns the filtered comments as a `TC` payload (`sentiment.CommitResult`) + +### Comment filtering + +Comments are filtered using several heuristics: + +- **Minimum length**: Comments shorter than the threshold are skipped +- **Letter ratio**: At least 60% of characters must be Unicode letters (filters out commented-out code) +- **First character**: Must start with a Unicode letter or digit (supports all scripts) +- **License detection**: Comments matching license/copyright patterns (including UK "licence" spelling) are excluded +- **Function name removal**: Inline function references like `doThing()` are stripped before analysis + +### Multilingual support + +The analyzer supports comments in any language that uses Unicode letters: + +- **CJK**: Chinese, Japanese, Korean +- **Cyrillic**: Russian, Ukrainian, etc. +- **Arabic**: Right-to-left scripts +- **Latin**: English, French, German, Spanish, etc. +- All other Unicode letter scripts + +Comment filtering uses Unicode-aware regex patterns (`\p{L}` for letters, `\p{N}` for digits) rather than ASCII-only ranges, ensuring comments in non-English languages are not silently dropped. + +!!! info "Multilingual sentiment scoring" + Sentiment scoring uses VADER's English lexicon as the base, extended with **93,000+ multilingual word entries** from the Chen-Skiena lexicon dataset (ACL 2014) covering **32 languages**. Non-ASCII words from the dataset are injected into VADER's lexicon at startup, enabling basic sentiment scoring for comments in Russian, Chinese, Japanese, Korean, Arabic, and 27 other languages. VADER's grammatical rules (negation, intensifiers) still operate on English syntax, so scoring accuracy for non-English comments is lower than for English — but significantly better than no coverage. + +### Sentiment classification + +Filtered comments are classified as positive, negative, or neutral using **VADER** (Valence Aware Dictionary and sEntiment Reasoner) via the [GoVader](https://github.com/jonreiter/govader) library, enhanced with **software engineering domain adjustments**. + +#### VADER base scoring + +VADER is a lexicon and rule-based sentiment analyzer designed for social media and short text. It handles negations, intensifiers, and punctuation. The compound score (-1 to 1) is mapped to our [0, 1] range. + +#### Multilingual lexicon extension + +At startup, the analyzer injects ~93,000 multilingual word entries from the [Chen-Skiena lexicon dataset](https://aclanthology.org/P14-2063/) (ACL 2014) into VADER's lexicon. This covers 32 languages: + +| Language Family | Languages | +|---|---| +| **Slavic** | Russian, Ukrainian, Polish, Czech, Slovak, Croatian, Bulgarian | +| **CJK** | Chinese, Japanese, Korean | +| **Romance** | Spanish, French, Portuguese, Italian, Romanian | +| **Germanic** | German, Dutch, Swedish, Danish, Norwegian, Finnish | +| **Other** | Arabic, Hebrew, Hindi, Thai, Turkish, Greek, Hungarian, Indonesian, Malay, Vietnamese, Persian | + +Only non-ASCII words are injected to avoid overriding VADER's curated English entries. Words receive binary valence: +1.5 (positive) or -1.5 (negative), which is the mid-range of VADER's scale. + +To regenerate lexicons from updated source data: + +```bash +go run tools/lexgen/lexgen.go -pos pos_words.txt -neg neg_words.txt \ + -o internal/analyzers/sentiment/lexicons/lexicon_data.gen.go +``` + +#### SE-domain adjustments + +VADER frequently misclassifies technical terms that sound negative in natural language but are neutral in software engineering. The analyzer applies domain-specific adjustments: + +**Neutralized terms** (pushed toward neutral): +`kill`, `abort`, `fatal`, `terminate`, `dead`, `destroy`, `panic`, `deprecated`, `obsolete`, `execute`, `exploit`, `conflict`, `revert`, `reject`, `critical` + +**Genuinely negative in SE** (pushed toward negative): +`hack`, `hacky`, `kludge`, `workaround`, `spaghetti`, `nightmare`, `technical debt` + +#### Comment length weighting + +Longer comments carry more weight in the sentiment score, as they tend to contain more meaningful sentiment signal. Weight is capped at 3x the average to prevent a single long comment from dominating. + +#### Trend analysis + +Sentiment trend is computed using **linear regression** (least squares) rather than a simple first-to-last comparison. This makes the trend robust to outliers and intermediate noise. + +--- + +## SE-domain lexicon + +The following terms have special handling in the sentiment scorer: + +### Technical terms neutralized (not negative) + +These terms are common in SE but trigger VADER's negative scoring: + +| Term | Context | +|---|---| +| `kill` | Process management | +| `abort` | Transaction/operation cancellation | +| `fatal` | Error severity levels | +| `terminate` | Process/thread lifecycle | +| `panic` | Error handling (Go, Rust) | +| `deprecated` | API lifecycle | +| `execute` | Command/query execution | +| `conflict` | Merge conflicts | +| `critical` | Severity levels | + +### Terms with genuine negative sentiment + +These terms indicate real frustration or code quality issues: + +| Term | Indication | +|---|---| +| `hack` / `hacky` | Quick-and-dirty solutions | +| `kludge` | Inelegant fixes | +| `workaround` | Avoiding root cause | +| `spaghetti` | Poor code structure | +| `nightmare` | Maintenance difficulty | +| `technical debt` | Accumulated shortcuts | + +--- + +## Use cases + +- **Team morale tracking**: Monitor sentiment trends over time. A sustained drop in sentiment may correlate with deadline pressure, technical debt accumulation, or team issues. +- **Code quality signals**: Negative sentiment spikes often correspond to periods of rushed development, workarounds, or "hack" implementations. +- **Post-mortem analysis**: After incidents, examine whether comment sentiment degraded in the lead-up to the problem. +- **Documentation quality**: Projects with predominantly neutral or positive comments tend to have better documentation culture. +- **Technical debt detection**: Comments containing terms like "hack", "workaround", "kludge" are flagged as genuinely negative in SE context. + +--- + +## Limitations + +- **Non-English scoring accuracy**: While 32 languages have lexicon coverage via the Chen-Skiena dataset, VADER's grammatical rules (negation handling, intensifiers, punctuation effects) are English-specific. Non-English comments get word-level sentiment from the lexicon but miss syntactic nuances. +- **UAST dependency**: Requires UAST parsing support for the target language. Files in unsupported languages are skipped. +- **Sarcasm**: Sarcasm, irony, and context-dependent meaning can mislead the classifier. Comments like "great, another production outage" may be scored as positive. +- **Comment extraction**: Only comments that appear in the UAST are analyzed. Preprocessor directives, build file comments, and non-code files are excluded. +- **CPU intensive**: The sentiment analyzer performs UAST parsing for every modified file in every commit. For large repositories, this is significantly slower than non-UAST analyzers. It benefits from parallel execution via the framework's worker pool. +- **Minimum length filter**: The default minimum of 20 characters filters out many short but potentially meaningful comments (e.g., `// FIXME`, `// HACK`). Lower `MinLength` to capture these, at the cost of more noise. + +--- + +## See also + +- [Sentiment reference](../analyzers/sentiment.md) — configuration keys and output schema. +- [Quick start](../getting-started/quickstart.md) — run history analysis. + + +--- + +# File: site/explanation/shotness.md + +# Understanding structural hotness + +This page explains the mental model behind the shotness analyzer: what +node-level change frequency and coupling mean, how coupling strength is +computed, and how the aggregation pipeline is structured. For configuration +keys and the output schema, see the +[Shotness reference](../analyzers/shotness.md). + +--- + +## What it measures + +The shotness analyzer measures **structural hotness** -- the change frequency of +individual code entities (functions, methods, classes) across Git history. +Unlike the couples analyzer which operates at file granularity, shotness +operates at the UAST node level, providing fine-grained co-change analysis. + +### Node change frequency + +For each code entity matched by the DSL query (functions by default), the analyzer counts how many commits modified lines within that entity's span. Entities that change frequently are "hot" -- they are likely volatile, complex, or central to the system. + +### Node co-change coupling + +When two code entities are modified in the same commit, their coupling counter is incremented. This produces a fine-grained coupling matrix at the function level, which is more precise than file-level coupling from the couples analyzer. + +### Coupling strength + +Coupling strength is normalized to a 0-1 scale using the formula: + +``` +strength(A, B) = co_changes(A, B) / max(co_changes(A, B), changes(A), changes(B)) +``` + +This ensures the result is always in [0, 1] and provides a meaningful confidence metric. A strength of 1.0 means functions always change together; 0.5 means they co-change half the time relative to the most active function. + +### Risk classification + +Nodes are classified into risk levels based on absolute change counts: + +| Risk Level | Threshold | Meaning | +|---|---|---| +| **HIGH** | ≥ 20 changes | Requires immediate attention and robust test coverage | +| **MEDIUM** | ≥ 10 changes | Should be monitored and potentially refactored | +| **LOW** | < 10 changes | Normal change frequency | + +### How it works + +For each commit: + +1. Parse the before and after versions of each changed file into UAST +2. Apply the `dsl_struct` query to select target nodes (e.g., functions) +3. Apply the `dsl_name` query to extract the name of each node +4. Map diff hunks to nodes using line-range overlaps +5. Emit a per-commit TC (Transient Commit result) with touched node deltas and coupling pairs + +After all commits are processed, the Aggregator accumulates TCs into a final report with sorted nodes and a sparse co-change matrix. + +### Architecture + +The shotness analyzer follows the **TC/Aggregator** pattern: + +- **Consume phase**: Per-commit processing builds working state (`nodes`, `files` maps for deletion/rename tracking) and emits a `TC{Data: *CommitData}` with node touch deltas and coupling pairs. +- **Aggregation phase**: The `Aggregator` accumulates node counts and coupling matrices from the TC stream. It supports disk-backed spilling via `SpillStore` for memory-bounded operation. +- **Serialization phase**: `SerializeTICKs()` converts aggregated tick data into the `Nodes`/`Counters` report consumed by `ComputeAllMetrics()` and plot generation. + +The `nodes` map remains in the analyzer as working state because `handleDeletion`, `handleInsertion`, `handleModification`, and `applyRename` read and mutate it during `Consume()`. The aggregator maintains its own separate accumulation of counts and couplings. + +--- + +## Use cases + +- **Function-level hotspot detection**: Find the most frequently changed functions in the codebase. These are the highest-risk points for bugs. +- **Fine-grained coupling analysis**: Discover which functions always change together. This reveals implicit dependencies that file-level coupling misses. +- **Refactoring prioritization**: Functions that are both hot (high change count) and coupled (always change with others) are the best refactoring candidates. +- **Architecture validation**: Functions from different packages that are highly coupled may indicate a leaking abstraction. +- **Test prioritization**: Focus testing resources on the hottest functions. + +--- + +## Interpreting results + +### Reading the coupling strength + +| Strength | Interpretation | +|---|---| +| 0.8 - 1.0 | Very tight coupling. Functions almost always change together. Consider merging or extracting shared logic. | +| 0.5 - 0.8 | Moderate coupling. There is a significant shared dependency. Review if coupling is intentional. | +| 0.2 - 0.5 | Loose coupling. Occasional co-changes, likely due to shared APIs or data structures. | +| < 0.2 | Minimal coupling. Co-changes are incidental. | + +### Actionable insights + +1. **High hotness + High coupling**: Core function that drives many changes. Candidate for splitting or stabilizing the interface. +2. **High hotness + Low coupling**: Frequently bugfixed isolated function. Needs better tests and potentially a redesign. +3. **Low hotness + High coupling**: Stable function that always changes with others. Check if coupling is necessary or indicates a design smell. + +--- + +## Limitations + +- **UAST required**: Only languages with UAST parser support are analyzed. Files in unsupported languages are skipped entirely. +- **CPU intensive**: The analyzer performs UAST parsing on both the before and after versions of every changed file in every commit. This makes it one of the most expensive analyzers. It benefits from parallel execution. +- **Name collisions**: If two functions in different files have the same name, they are tracked as distinct nodes (the file path is part of the key). However, if a file is renamed, the analyzer updates all associated nodes. +- **Shallow extraction within a file**: When multiple structural nodes in the same file share the same extracted name (e.g., nested functions with identical names), only one is tracked. The last one encountered wins. Qualified paths (e.g., `OuterClass.innerMethod`) are not built. +- **DSL limitations**: The DSL query must match nodes that have position information (`Pos` field) in the UAST. Nodes without position data cannot be mapped to diff hunks. +- **Large functions**: A change anywhere within a function's line range counts as a change to that function. Very large functions (hundreds of lines) will have inflated change counts. + +--- + +## See also + +- [Shotness reference](../analyzers/shotness.md) — configuration keys and output schema. +- [Quick start](../getting-started/quickstart.md) — run history analysis. + + +--- + +# File: site/explanation/typos.md + +# Understanding typo detection + +This page explains the mental model behind the typos analyzer: how it detects +typo-fix identifier pairs, what Levenshtein distance measures, and how the +aggregation pipeline is structured. For configuration keys and the output +schema, see the [Typos reference](../analyzers/typos.md). + +--- + +## What it measures + +The typos analyzer detects **typo-fix identifier pairs** from source code in +commit diffs using Levenshtein distance. It builds a dataset of probable typos +and their corrections by analyzing UAST identifier changes across Git history. + +### Typo-fix pair detection + +For each commit, the analyzer: + +1. Computes the diff between the old and new versions of each changed file +2. Identifies delete/insert hunk pairs of equal size (same number of lines) +3. Compares corresponding lines using **Levenshtein distance** +4. For line pairs within the distance threshold, extracts UAST identifiers from the old and new versions +5. If exactly one identifier changed between the two versions, it records a typo-fix pair + +### Levenshtein distance + +The Levenshtein distance is the minimum number of single-character edits (insertions, deletions, substitutions) needed to transform one string into another. A small distance between the old and new line suggests a typo fix rather than a semantic change. + +!!! info "Example" + - `recieve` to `receive` -- distance 2 (transposed characters) + - `lenght` to `length` -- distance 2 (transposed characters) + - `calcualte` to `calculate` -- distance 2 (transposed characters) + +--- + +## Architecture + +The typos analyzer follows the **TC/Aggregator pattern**: + +1. **Consume phase**: For each commit, `Consume()` computes diffs, identifies line pairs within Levenshtein distance, and extracts UAST identifier changes. Per-commit typos are returned as `TC{Data: []Typo}`. The analyzer retains no per-commit state; only the `lcontext` (Levenshtein context) is kept as working state. +2. **Aggregation phase**: A `typos.Aggregator` collects TCs into a `SliceSpillStore[Typo]`. `FlushTick()` deduplicates typos by `wrong|correct` key (keeping the first occurrence), returning a `TickData` with the unique set. +3. **Serialization phase**: `SerializeTICKs()` assembles all tick data into an `analyze.Report{"typos": allTypos}`, then delegates to `ComputeAllMetrics()` for JSON, YAML, binary, or HTML plot output. + +This separation enables streaming output, budget-aware memory spilling, and decoupled aggregation. + +--- + +## Use cases + +- **Typo dataset building**: Build a corpus of real-world typos and their corrections from your project's history. This can train spell-checking tools or IDE plugins. +- **Code quality auditing**: Identify patterns of common misspellings in your codebase to add to a linting dictionary. +- **API consistency**: Detect identifier typos that may cause confusion (e.g., `getUserByNmae` vs `getUserByName`). +- **Automated fix suggestions**: Use the typo dataset to build automated correction rules for CI pipelines. +- **Research**: Academic research on developer typo patterns and their prevalence across different languages. + +--- + +## Limitations + +- **UAST required**: Only languages with UAST parser support are analyzed. Identifiers in unsupported languages are not extracted. +- **Single-identifier changes only**: The analyzer only records a typo when exactly one identifier changes between the old and new lines. Multi-identifier changes are skipped to avoid false positives. +- **Equal-length hunks only**: Only delete/insert hunk pairs with the same number of lines are considered. A typo fix that also adds or removes lines will be missed. +- **False positives**: Intentional identifier renames with small Levenshtein distance (e.g., `idx` to `jdx`) will be reported as typos. +- **Deduplication**: Typo pairs are deduplicated by the `wrong|correct` key. The same typo fixed in multiple commits is reported only once. +- **CPU intensive**: Like all UAST-based analyzers, the typos analyzer parses both file versions for every changed file in every commit. + +--- + +## See also + +- [Typos reference](../analyzers/typos.md) — configuration keys and output schema. +- [Quick start](../getting-started/quickstart.md) — run history analysis. + + +--- + +# File: site/explanation/anomaly.md + +# Understanding temporal anomaly detection + +This page explains the mental model behind the anomaly analyzer: how Z-score +analysis over a sliding window flags unusual activity, which metrics it tracks, +and how the aggregation pipeline is structured. For configuration keys and the +output schema, see the [Anomaly reference](../analyzers/anomaly.md). + +--- + +## What it measures + +The anomaly analyzer performs **temporal anomaly detection** using Z-score +analysis over a sliding window of per-tick commit metrics. It detects sudden +quality degradation -- unusual spikes in churn, file changes, or other metrics +that deviate significantly from the historical baseline. + +### Z-score analysis + +For each time tick, the analyzer computes Z-scores for six metrics using a trailing sliding window: + +| Metric | Description | +|---|---| +| **Net churn** | `lines_added - lines_removed` | +| **Files changed** | Number of files modified in the tick | +| **Lines added** | Total lines added across all commits in the tick | +| **Lines removed** | Total lines removed across all commits in the tick | +| **Language diversity** | Number of distinct programming languages modified | +| **Author count** | Number of distinct developers active in the tick | + +A Z-score measures how many standard deviations a value is from the rolling mean. When any metric's absolute Z-score exceeds the threshold, the tick is flagged as anomalous. + +!!! info "Z-score interpretation" + - **|Z| < 2.0**: Within normal variation + - **|Z| 2.0 - 3.0**: Unusual, worth investigating + - **|Z| > 3.0**: Highly anomalous, likely a significant event + +### Sliding window + +The Z-score is computed against a trailing window of the previous N ticks (default 20). This means the "normal" baseline adapts to gradual trends, and only sudden deviations are flagged. + +### Anomaly records + +Each detected anomaly includes: + +- **Tick index**: When it occurred +- **Z-scores**: Per-metric Z-scores showing which dimension is anomalous +- **Max absolute Z-score**: The highest Z-score across all metrics (used for severity ranking) +- **Raw metrics**: The actual values for that tick +- **Files**: List of files changed in the anomalous tick + +Anomalies are sorted by severity (highest absolute Z-score first). + +--- + +## Architecture + +The anomaly analyzer follows the **TC/Aggregator pattern**: + +1. **Consume phase**: For each commit, `Consume()` extracts file changes, line stats, language diversity, and author ID, returning per-commit metrics as a `TC{Data: *CommitAnomalyData}`. The analyzer retains no per-commit state. +2. **Aggregation phase**: An `anomaly.Aggregator` collects TCs, merges `CommitAnomalyData` into per-tick `TickMetrics` (aggregating files, lines, languages, and unique authors), and produces `TICK` results. +3. **Z-score detection**: `ticksToReport()` runs sliding-window Z-score analysis on six dimensions over the aggregated tick metrics to detect anomalies. +4. **Serialization phase**: `SerializeTICKs()` converts aggregated TICKs into JSON, YAML, binary, or HTML plot output via `ComputeAllMetrics()`. + +This separation enables streaming output, budget-aware memory spilling, and decoupled aggregation. + +--- + +## Use cases + +- **Quality gates**: Flag releases or sprints that contain anomalous commit patterns as higher risk. +- **Incident correlation**: Overlay anomaly timelines with production incident timelines to find patterns. +- **Process monitoring**: Detect process breakdowns -- e.g., a sudden spike in single-author commits to many files may indicate bypassed code review. +- **Onboarding safety**: Monitor anomaly rates during onboarding periods. New developers may produce unusual patterns. +- **Seasonal pattern detection**: Over long time periods, the analyzer reveals cyclical patterns (release crunches, holiday slowdowns). +- **Multi-metric alerts**: Unlike simple threshold alerts on individual metrics, the Z-score approach adapts to the repository's natural variance. + +--- + +## Limitations + +- **Cold start**: The first `window_size` ticks have no or limited history for Z-score computation. Anomalies during this period are less reliable. +- **Gradual drift**: Z-scores detect sudden deviations, not gradual trends. A slow but steady increase in churn will update the rolling mean and never trigger an anomaly. +- **Zero-variance sentinel**: When the window has zero standard deviation (all values identical) and the current value differs, a sentinel Z-score of 100 is assigned. This ensures detection but may inflate severity rankings. +- **Tick granularity**: Metrics are aggregated per tick (default 24-hour window). Sub-tick anomalies are not detected. +- **Not causal**: The analyzer detects statistical anomalies, not root causes. Investigation is required to determine whether an anomaly represents a real problem. +- **Multi-dimensional**: An anomaly is flagged when *any* single metric exceeds the threshold. This means false positive rates increase with the number of metrics (currently six). + +--- + +## See also + +- [Anomaly reference](../analyzers/anomaly.md) — configuration keys and output schema. +- [Quick start](../getting-started/quickstart.md) — run history analysis. diff --git a/llms.txt b/llms.txt new file mode 100644 index 00000000..b1c34527 --- /dev/null +++ b/llms.txt @@ -0,0 +1,44 @@ +# Codefang + +> Codefang is a comprehensive code analysis platform that understands your code deeply — not just as text, but as structure (AST) and history (Git) — for tracking technical debt, analyzing developer churn, and feeding context to AI agents. + +Codefang is a spiritual successor to source{d}'s `hercules`, rebuilt on a modern engine. It splits into two binaries: `uast` (the parser, turning source into a Universal Abstract Syntax Tree across 60+ languages via Tree-sitter) and `codefang` (the analyzer, consuming UASTs for static analysis or Git repositories for history mining). Analyzers are pluggable, and the tool is designed to be AI-agent ready through an MCP server and a UNIX-pipe-friendly CLI. + +## README + +- [README](https://raw.githubusercontent.com/Sumatoshi-tech/codefang/main/README.md): project entry point — what Codefang is, why it exists, install, and a quickstart. + +## Tutorial + +- [Quick start](https://sumatoshi-tech.github.io/codefang/getting-started/quickstart/): install the binaries and run your first static and history analyses end to end. + +## How-to + +- [How to analyze a monorepo](https://sumatoshi-tech.github.io/codefang/how-to/analyze-a-monorepo/): scope analysis to subprojects within a single large repository. +- [How to run Codefang in CI](https://sumatoshi-tech.github.io/codefang/how-to/run-in-ci/): wire Codefang into a continuous-integration pipeline. +- [How to read a burndown chart](https://sumatoshi-tech.github.io/codefang/how-to/read-a-burndown-chart/): interpret the survival-over-time output of the burndown analyzer. +- [How to analyze a bare repository](https://sumatoshi-tech.github.io/codefang/how-to/analyze-a-bare-repo/): run history analysis against a bare Git clone with no working tree. +- [How to attach an MCP client](https://sumatoshi-tech.github.io/codefang/how-to/attach-an-mcp-client/): connect an AI agent to Codefang through the MCP server. + +## Reference + +- [Analyzers](https://sumatoshi-tech.github.io/codefang/analyzers/): reference index of every static and history analyzer with configuration keys and output schemas. +- [CLI reference](https://sumatoshi-tech.github.io/codefang/guide/cli-reference/): complete `codefang` and `uast` command, flag, and subcommand reference. + +## Explanation + +- [Understanding complexity metrics](https://sumatoshi-tech.github.io/codefang/explanation/complexity/): the mental model behind cyclomatic and cognitive complexity. +- [Understanding cohesion metrics](https://sumatoshi-tech.github.io/codefang/explanation/cohesion/): how Codefang measures structural cohesion. +- [Understanding Halstead metrics](https://sumatoshi-tech.github.io/codefang/explanation/halstead/): the operator/operand model behind Halstead volume and difficulty. +- [Understanding documentation coverage](https://sumatoshi-tech.github.io/codefang/explanation/comments/): how comment-coverage scoring works and what it does not measure. +- [Understanding import and dependency analysis](https://sumatoshi-tech.github.io/codefang/explanation/imports/): how the import graph is built and read. +- [Understanding clone detection](https://sumatoshi-tech.github.io/codefang/explanation/clones/): the approach to finding duplicated code structure. +- [Understanding code burndown](https://sumatoshi-tech.github.io/codefang/explanation/burndown/): the model of code survival and decay over history. +- [Understanding developer analysis](https://sumatoshi-tech.github.io/codefang/explanation/developers/): how authorship and developer churn are attributed. +- [Understanding file and developer coupling](https://sumatoshi-tech.github.io/codefang/explanation/couples/): how co-change coupling between files and developers is derived. +- [Understanding file history analysis](https://sumatoshi-tech.github.io/codefang/explanation/file-history/): the model for per-file lifecycle over time. +- [Understanding code quality over time](https://sumatoshi-tech.github.io/codefang/explanation/quality/): how quality trends are tracked across history. +- [Understanding comment sentiment analysis](https://sumatoshi-tech.github.io/codefang/explanation/sentiment/): how sentiment in code comments is scored and interpreted. +- [Understanding structural hotness](https://sumatoshi-tech.github.io/codefang/explanation/shotness/): how structurally hot (frequently changed) code is identified. +- [Understanding typo detection](https://sumatoshi-tech.github.io/codefang/explanation/typos/): how typos in identifiers and comments are flagged. +- [Understanding temporal anomaly detection](https://sumatoshi-tech.github.io/codefang/explanation/anomaly/): how anomalous spikes in repository activity are detected. diff --git a/mkdocs.yml b/mkdocs.yml index 93578c1c..03da4a99 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -102,46 +102,80 @@ extra_css: nav: - Home: index.md - - Getting Started: + - Getting started: - Installation: getting-started/installation.md - - Quick Start: getting-started/quickstart.md - - First Analysis: getting-started/first-analysis.md - - User Guide: - - CLI Reference: guide/cli-reference.md + - Quick start: getting-started/quickstart.md + - First analysis: getting-started/first-analysis.md + - User guide: + - CLI reference: guide/cli-reference.md - Configuration: guide/configuration.md - - Output Formats: guide/output-formats.md + - Output formats: guide/output-formats.md + - How to: + - Analyze a monorepo: how-to/analyze-a-monorepo.md + - Run in CI: how-to/run-in-ci.md + - Read a burndown chart: how-to/read-a-burndown-chart.md + - Analyze a bare repository: how-to/analyze-a-bare-repo.md + - Attach an MCP client: how-to/attach-an-mcp-client.md - Analyzers: - Overview: analyzers/index.md - - Static Analyzers: + - Static analyzers: - Complexity: analyzers/complexity.md - Cohesion: analyzers/cohesion.md - Halstead: analyzers/halstead.md - Comments: analyzers/comments.md - - Imports (Static): analyzers/imports.md - - History Analyzers: + - Imports (static): analyzers/imports.md + - Clones: analyzers/clones.md + - History analyzers: - Burndown: analyzers/burndown.md - Developers: analyzers/developers.md - Couples: analyzers/couples.md - - File History: analyzers/file-history.md + - File history: analyzers/file-history.md - Quality: analyzers/quality.md - Sentiment: analyzers/sentiment.md - Shotness: analyzers/shotness.md - Typos: analyzers/typos.md - - Anomaly Detection: analyzers/anomaly.md + - Anomaly detection: analyzers/anomaly.md + - Reference: + - Glossary: glossary.md - Examples: - - Kubernetes Analysis: examples/index.md + - Kubernetes analysis: examples/index.md - Architecture: - Overview: architecture/overview.md - - UAST System: architecture/uast.md - - Streaming Pipeline: architecture/streaming-pipeline.md + - UAST system: architecture/uast.md + - Streaming pipeline: architecture/streaming-pipeline.md + - Explanation: + - Static analyzers: + - Complexity metrics: explanation/complexity.md + - Cohesion metrics: explanation/cohesion.md + - Halstead metrics: explanation/halstead.md + - Documentation coverage: explanation/comments.md + - Import analysis: explanation/imports.md + - Clone detection: explanation/clones.md + - History analyzers: + - Code burndown: explanation/burndown.md + - Developer analysis: explanation/developers.md + - File and developer coupling: explanation/couples.md + - File history analysis: explanation/file-history.md + - Code quality over time: explanation/quality.md + - Comment sentiment: explanation/sentiment.md + - Structural hotness: explanation/shotness.md + - Typo detection: explanation/typos.md + - Temporal anomaly detection: explanation/anomaly.md - Integrations: - - MCP Server: integrations/mcp.md - - Docker & GitHub Actions: integrations/docker-and-actions.md - - AI Agents: integrations/ai-agents.md + - MCP server: integrations/mcp.md + - Docker & GitHub actions: integrations/docker-and-actions.md + - AI agents: integrations/ai-agents.md - Operations: - Observability: operations/observability.md - - Large-Scale Scanning: operations/large-scale-scanning.md + - Large-scale scanning: operations/large-scale-scanning.md + - ADRs: + - Record architecture decisions: adr/0001-record-architecture-decisions.md + - Two-binary split (UAST + Codefang): adr/0002-two-binary-split-uast-codefang.md + - libgit2 via cgo: adr/0003-libgit2-via-cgo.md + - Release notes: + - Overview: release-notes/index.md + - Upcoming release: release-notes/unreleased.md - Contributing: - - How to Contribute: contributing/index.md - - Code of Conduct: contributing/code-of-conduct.md + - How to contribute: contributing/index.md + - Code of conduct: contributing/code-of-conduct.md - Changelog: contributing/changelog.md diff --git a/site/adr b/site/adr new file mode 120000 index 00000000..57d03653 --- /dev/null +++ b/site/adr @@ -0,0 +1 @@ +../docs/adr \ No newline at end of file diff --git a/site/analyzers/anomaly.md b/site/analyzers/anomaly.md index 79630f54..f6d14d9c 100644 --- a/site/analyzers/anomaly.md +++ b/site/analyzers/anomaly.md @@ -1,80 +1,16 @@ -# Anomaly Analyzer +# Anomaly analyzer reference -The anomaly analyzer performs **temporal anomaly detection** using Z-score analysis over a sliding window of per-tick commit metrics. It detects sudden quality degradation -- unusual spikes in churn, file changes, or other metrics that deviate significantly from the historical baseline. +The anomaly analyzer performs **temporal anomaly detection** using Z-score +analysis over a sliding window of per-tick commit metrics. ---- - -## Quick Start - -```bash -codefang run -a history/anomaly . -``` - -With custom threshold and window: - -```bash -codefang run -a history/anomaly \ - --anomaly-threshold 2.5 \ - --anomaly-window 30 \ - . -``` - ---- - -## Architecture - -The anomaly analyzer follows the **TC/Aggregator pattern**: - -1. **Consume phase**: For each commit, `Consume()` extracts file changes, line stats, language diversity, and author ID, returning per-commit metrics as a `TC{Data: *CommitAnomalyData}`. The analyzer retains no per-commit state. -2. **Aggregation phase**: An `anomaly.Aggregator` collects TCs, merges `CommitAnomalyData` into per-tick `TickMetrics` (aggregating files, lines, languages, and unique authors), and produces `TICK` results. -3. **Z-score detection**: `ticksToReport()` runs sliding-window Z-score analysis on six dimensions over the aggregated tick metrics to detect anomalies. -4. **Serialization phase**: `SerializeTICKs()` converts aggregated TICKs into JSON, YAML, binary, or HTML plot output via `ComputeAllMetrics()`. - -This separation enables streaming output, budget-aware memory spilling, and decoupled aggregation. - ---- - -## What It Measures - -### Z-Score Analysis - -For each time tick, the analyzer computes Z-scores for six metrics using a trailing sliding window: - -| Metric | Description | -|---|---| -| **Net churn** | `lines_added - lines_removed` | -| **Files changed** | Number of files modified in the tick | -| **Lines added** | Total lines added across all commits in the tick | -| **Lines removed** | Total lines removed across all commits in the tick | -| **Language diversity** | Number of distinct programming languages modified | -| **Author count** | Number of distinct developers active in the tick | - -A Z-score measures how many standard deviations a value is from the rolling mean. When any metric's absolute Z-score exceeds the threshold, the tick is flagged as anomalous. - -!!! info "Z-score interpretation" - - **|Z| < 2.0**: Within normal variation - - **|Z| 2.0 - 3.0**: Unusual, worth investigating - - **|Z| > 3.0**: Highly anomalous, likely a significant event - -### Sliding Window - -The Z-score is computed against a trailing window of the previous N ticks (default 20). This means the "normal" baseline adapts to gradual trends, and only sudden deviations are flagged. - -### Anomaly Records - -Each detected anomaly includes: - -- **Tick index**: When it occurred -- **Z-scores**: Per-metric Z-scores showing which dimension is anomalous -- **Max absolute Z-score**: The highest Z-score across all metrics (used for severity ranking) -- **Raw metrics**: The actual values for that tick -- **Files**: List of files changed in the anomalous tick - -Anomalies are sorted by severity (highest absolute Z-score first). +For the conceptual model — how Z-score detection works and which metrics it +tracks — see +[Understanding temporal anomaly detection](../explanation/anomaly.md). To run +it, see the [Quick start](../getting-started/quickstart.md). --- -## Configuration Options +## Configuration options | Option | Type | Default | Description | |---|---|---|---| @@ -89,6 +25,8 @@ history: window_size: 20 ``` +The `--anomaly-threshold` and `--anomaly-window` CLI flags set these options for a single run. + !!! tip "Tuning parameters" - **Lower threshold** (e.g., 1.5): More sensitive, flags more events. Good for quiet repositories. - **Higher threshold** (e.g., 3.0): Only flags extreme events. Good for noisy repositories with high variance. @@ -97,7 +35,7 @@ history: --- -## Example Output +## Example output === "JSON" @@ -202,7 +140,7 @@ history: --- -## Output Structure +## Output structure The anomaly analyzer produces three main output sections: @@ -214,22 +152,7 @@ The anomaly analyzer produces three main output sections: --- -## Use Cases - -- **Quality gates**: Flag releases or sprints that contain anomalous commit patterns as higher risk. -- **Incident correlation**: Overlay anomaly timelines with production incident timelines to find patterns. -- **Process monitoring**: Detect process breakdowns -- e.g., a sudden spike in single-author commits to many files may indicate bypassed code review. -- **Onboarding safety**: Monitor anomaly rates during onboarding periods. New developers may produce unusual patterns. -- **Seasonal pattern detection**: Over long time periods, the analyzer reveals cyclical patterns (release crunches, holiday slowdowns). -- **Multi-metric alerts**: Unlike simple threshold alerts on individual metrics, the Z-score approach adapts to the repository's natural variance. - ---- - -## Limitations +## See also -- **Cold start**: The first `window_size` ticks have no or limited history for Z-score computation. Anomalies during this period are less reliable. -- **Gradual drift**: Z-scores detect sudden deviations, not gradual trends. A slow but steady increase in churn will update the rolling mean and never trigger an anomaly. -- **Zero-variance sentinel**: When the window has zero standard deviation (all values identical) and the current value differs, a sentinel Z-score of 100 is assigned. This ensures detection but may inflate severity rankings. -- **Tick granularity**: Metrics are aggregated per tick (default 24 hours). Sub-tick anomalies are not detected. -- **Not causal**: The analyzer detects statistical anomalies, not root causes. Investigation is required to determine whether an anomaly represents a real problem. -- **Multi-dimensional**: An anomaly is flagged when *any* single metric exceeds the threshold. This means false positive rates increase with the number of metrics (currently six). +- [Understanding temporal anomaly detection](../explanation/anomaly.md) — the mental model, Z-score analysis, architecture, use cases, and limitations. +- [Quick start](../getting-started/quickstart.md) — run history analysis. diff --git a/site/analyzers/burndown.md b/site/analyzers/burndown.md index 5a40b3ed..7457f00b 100644 --- a/site/analyzers/burndown.md +++ b/site/analyzers/burndown.md @@ -1,85 +1,16 @@ -# Burndown Analyzer +# Burndown analyzer reference -The burndown analyzer tracks **code survival over time** by following every line of code through Git history. It produces burndown charts showing how code written at different points in time persists, is modified, or is deleted. Optionally, it tracks per-file and per-developer breakdowns. +The burndown analyzer tracks **code survival over time** by following every +line of code through Git history. Optionally, it tracks per-file and +per-developer breakdowns. ---- - -## Quick Start - -```bash -codefang run -a history/burndown . -``` - -With per-file and per-developer tracking: - -```bash -codefang run -a history/burndown \ - --burndown-files \ - --burndown-people \ - . -``` - ---- - -## What It Measures - -### Global Code Survival - -A time-series matrix where each row is a sampling point and each column is an age band. The value at `[sample][band]` is the number of lines that were last edited during that band and still survive at that sample point. - -This matrix produces the classic **burndown chart**: stacked area plots showing how much code from each era remains. - -### Per-File Burndown - -When `--burndown-files` is enabled, the analyzer produces a separate survival matrix for each file, enabling file-level burndown charts. - -### Per-Developer Burndown - -When `--burndown-people` is enabled, the analyzer tracks which developer last edited each line. This reveals: - -- **Developer survival rates**: How much of each developer's code persists -- **Interaction matrix**: Which developers modify each other's code - -### File Ownership - -When both `--burndown-files` and `--burndown-people` are enabled, the analyzer computes per-file ownership by iterating the live line segments in each file's internal tree. Each segment stores a packed `[author|tick]` value, from which the author ID is extracted to produce a `file -> author -> line_count` mapping. - -!!! note "Ownership requires developer tracking" - File ownership data is only available when `--burndown-people` is enabled. Without developer tracking, no author information is stored in the line segments, so file ownership will be empty. - -!!! warning "Memory usage" - Per-file and per-developer tracking significantly increases memory usage. For repositories with more than 100k commits, consider enabling hibernation (on by default). +For the conceptual model — what code survival means and how the line-tracking +algorithm works — see [Understanding code burndown](../explanation/burndown.md). +To run it, see the [Quick start](../getting-started/quickstart.md). --- -## How It Works - -### Algorithm Overview - -1. **Commit traversal**: Commits are processed sequentially in topological order. For each commit, the analyzer diffs the parent tree against the current tree to find inserted, deleted, and modified lines. - -2. **Line tracking**: Each line in every file is tracked using an augmented balanced binary tree (treap). Each tree node stores a contiguous range of lines with a packed value encoding `[author_id | creation_tick]`. When lines are inserted or deleted, the tree is split and merged to maintain the correct line-to-value mapping. - -3. **Sparse history accumulation**: At each commit, the analyzer records deltas in a sparse history structure: `map[currentTick]map[creationTick]lineCountDelta`. This captures how many lines created at `creationTick` are alive at `currentTick`. - -4. **Tick aggregation**: Sparse deltas from individual commits are accumulated into tick-level snapshots by a streaming aggregator. At each sampling boundary, the aggregator emits a `TickResult` containing the full sparse history state. - -5. **Dense matrix conversion**: The sparse history is converted to a dense matrix (`DenseHistory`) via `groupSparseHistory()`. This involves: - - **Tick normalization**: Map arbitrary tick values to sorted indices - - **Granularity grouping**: Collapse adjacent creation ticks into age bands of width `granularity` - - **Forward-fill**: For each age band, carry forward the last known value across sampling gaps to produce a complete matrix - -6. **Metrics computation**: The dense matrix is used to compute aggregate statistics (survival rate, peak lines, current lines), per-developer survival, and the interaction matrix. - -### Key Parameters - -- **Granularity** controls the width of each age band (in ticks). Higher values produce fewer, wider bands. -- **Sampling** controls how frequently snapshots are taken (in ticks). Higher values reduce the number of data points. -- **Tick size** defaults to 24 hours. All commits within the same day share one tick. - ---- - -## Configuration Options +## Configuration options | Option | Type | Default | Description | |---|---|---|---| @@ -110,15 +41,11 @@ history: --- -## Output Formats +## Output formats ### Text (terminal) -```bash -codefang run -a history/burndown --format text . -``` - -Produces a concise terminal summary: +The `--format text` output produces a concise terminal summary: ``` ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ @@ -133,11 +60,11 @@ Produces a concise terminal summary: Code Age Distribution ─────────────────────────────────────────────── - < 1 month ████████████████████ 34% (17,796) - 1-3 months ████████████░░░░░░░░ 24% (12,562) - 3-6 months ████████░░░░░░░░░░░░ 18% ( 9,421) - 6-12 months ██████░░░░░░░░░░░░░░ 14% ( 7,328) - > 12 months ████░░░░░░░░░░░░░░░░ 10% ( 5,233) + < 1 mo ████████████████████ 34% (17,796) + 1–3 mo ████████████░░░░░░░░ 24% (12,562) + 3–6 mo ████████░░░░░░░░░░░░ 18% ( 9,421) + 6–12 mo ██████░░░░░░░░░░░░░░ 14% ( 7,328) + > 12 mo ████░░░░░░░░░░░░░░░░ 10% ( 5,233) Top Developers (by surviving lines) ─────────────────────────────────────────────── @@ -146,15 +73,11 @@ Produces a concise terminal summary: charlie 5,100 [████████░░░░░░░░░░░░] 48.2% ``` -The text output shows at most 5 age bands and 5 developers. Survival rates are color-coded: green (>70%), yellow (50-70%), red (<50%). +The text output shows at most 5 age bands and 5 developers (`mo` = months). Survival rates are color-coded: green (>70%), yellow (50-70%), red (<50%). ### Plot (HTML) -```bash -codefang run -a history/burndown --format plot . -``` - -Generates an interactive HTML page with: +The `--format plot` output generates an interactive HTML page with: - **Summary section**: Key statistics (current lines, peak lines, survival rate, analysis period, developer/file counts) - **Burndown chart**: Stacked area chart showing code survival by age band or by year (for projects spanning 2+ years) @@ -162,11 +85,6 @@ Generates an interactive HTML page with: ### JSON / YAML -```bash -codefang run -a history/burndown --format json . -codefang run -a history/burndown --format yaml . -``` - === "JSON" ```json @@ -242,21 +160,7 @@ codefang run -a history/burndown --format yaml . --- -## Use Cases - -- **Project health monitoring**: Track the overall code survival rate. A declining rate may indicate churn or instability. -- **Developer contribution analysis**: Understand whose code persists and who rewrites existing code. -- **Code age visualization**: Generate burndown charts showing how much ancient code remains in the codebase. -- **Refactoring impact**: Measure how much code a refactoring effort actually replaced. -- **Team dynamics**: The interaction matrix reveals collaboration patterns -- who reviews and modifies whose code. - ---- - -## Limitations +## See also -- **Sequential processing**: Burndown tracks cumulative per-line state across all commits and must process commits sequentially. It cannot be parallelized across commits (though per-file processing within a commit is parallelized via goroutines). -- **Memory intensive**: Every line in every file is tracked throughout history. Large repositories (100k+ commits, 10k+ files) can require several GB of RAM. Use hibernation options to manage memory. -- **Binary files excluded**: Binary files are automatically skipped since they cannot be meaningfully diff'd line-by-line. -- **Rename tracking**: File renames are tracked using Git's rename detection. If Git does not detect a rename (e.g., content changed significantly), the file appears as a deletion + insertion. -- **Tick resolution**: The default 24-hour tick means that all commits within the same day share one tick. Sub-day granularity is not supported. -- **File ownership without developer tracking**: File ownership data requires `--burndown-people`. Without it, the `file_survival` array will have empty ownership maps. +- [Understanding code burndown](../explanation/burndown.md) — the mental model, algorithm, use cases, and limitations. +- [Quick start](../getting-started/quickstart.md) — run history analysis. diff --git a/site/analyzers/clones.md b/site/analyzers/clones.md index 4a767b4a..80c0dfc5 100644 --- a/site/analyzers/clones.md +++ b/site/analyzers/clones.md @@ -1,83 +1,18 @@ -# Clone Detection Analyzer +# Clone detection analyzer reference -The clone detection analyzer finds **duplicate and near-duplicate functions** across your entire codebase using MinHash signatures and Locality-Sensitive Hashing (LSH). It operates on the UAST representation and detects clones **cross-file** — not just within a single file. +The clone detection analyzer finds **duplicate and near-duplicate functions** +across your entire codebase using MinHash signatures and Locality-Sensitive +Hashing (LSH). It operates on the UAST representation and detects clones +**cross-file** — not just within a single file. ---- - -## Quick Start - -```bash -uast parse main.go | codefang analyze -a clones -``` - -Or analyze an entire directory: - -```bash -codefang analyze -a clones ./src/ -``` +For the conceptual model — the clone-type taxonomy, the methodology, and why +MinHash + LSH — see +[Understanding clone detection](../explanation/clones.md). To run it, see the +[Quick start](../getting-started/quickstart.md). --- -## What It Measures - -### Clone Pairs - -A clone pair is two functions whose AST structure is similar enough to be considered duplicated code. Each pair includes a **similarity score** (0.0–1.0) and a **clone type** classification. - -### Clone Types - -| Type | Similarity | Description | -|---|---|---| -| **Type-1 (Exact)** | = 1.0 | Identical AST structure and tokens | -| **Type-2 (Renamed)** | ≥ 0.8 | Identical structure, different variable/function names | -| **Type-3 (Near-miss)** | ≥ 0.5 | Similar but modified structure | - -### Clone Ratio - -The ratio of detected clone pairs to total functions, computed globally across all files: - -``` -clone_ratio = total_clone_pairs / total_functions -``` - -!!! info "Interpretation" - - **0.0** (green): No duplication detected - - **≤ 0.1** (yellow): Low duplication — few clone pairs - - **≤ 0.3** (red): Moderate duplication — consider refactoring - - **> 0.3**: High duplication — significant refactoring recommended - ---- - -## Methodology - -### Phase 1: Per-File Signature Extraction - -For each source file, the analyzer: - -1. **Parses** the file into a UAST and finds all function/method nodes. -2. **Shingles** each function by performing a pre-order traversal of the AST subtree and extracting sliding windows of `k` consecutive node types (k=5 by default). For example, a function with nodes `[Function, Block, Assignment, Identifier, Call, Return]` produces shingles like `Function|Block|Assignment|Identifier|Call` and `Block|Assignment|Identifier|Call|Return`. -3. **MinHash** compresses each function's shingle set into a fixed-size signature of 128 hash values. Two functions with identical structure produce identical signatures; similar functions produce similar signatures. - -Signatures are exported per-file along with function names. At this stage, no clone detection occurs — comparison is deferred to the aggregation phase. - -### Phase 2: Cross-File Clone Detection - -After all files have been processed, the aggregator: - -1. **Qualifies** function names with their source file path (`path/to/file.go::FuncName`) so that same-named functions across files are distinguishable. -2. **Builds a global LSH index** (16 bands, 8 rows per band) from all collected signatures. LSH hashes each signature's bands into buckets — functions whose signatures share any bucket are candidate clones. -3. **Queries** the LSH index for each function, retrieves candidates above the similarity threshold (0.5), computes exact Jaccard similarity from the full MinHash signatures, and classifies each pair by clone type. -4. **Computes metrics** globally: `clone_ratio = total_pairs / total_functions`. - -This two-phase design means clone detection works across file boundaries. A controller duplicated in `pkg/a/handler.go` and `pkg/b/handler.go` is detected even though each file is parsed independently. - -### Why MinHash + LSH? - -Pairwise comparison of N functions is O(N²). For a project with 100K functions this is infeasible. MinHash compresses each function into a 1 KB signature (128 hashes × 8 bytes), and LSH reduces candidate pair generation to near-linear time by only comparing functions that hash into the same bucket. The combination provides probabilistic guarantees on recall while keeping the analysis fast. - ---- - -## Configuration Options +## Configuration options The clone detection analyzer uses the UAST directly and has no analyzer-specific configuration options. @@ -87,7 +22,7 @@ The clone detection analyzer uses the UAST directly and has no analyzer-specific --- -## Example Output +## Example output === "Text" @@ -144,30 +79,7 @@ The clone detection analyzer uses the UAST directly and has no analyzer-specific --- -## Use Cases - -- **Refactoring targets**: Identify exact and near-duplicate functions that should be consolidated into shared utilities. -- **Code review gates**: Flag pull requests that introduce new clone pairs, especially Type-1 (exact) duplicates. -- **Copy-paste detection**: Find controllers, handlers, or boilerplate that were copy-pasted across packages instead of being extracted. -- **Technical debt tracking**: Monitor clone ratio over time to ensure duplication does not grow unbounded. -- **Cross-team coordination**: Detect when multiple teams independently implement the same logic in different packages. - ---- - -## Methodology References - -- **MinHash**: Broder, A. Z. (1997). *On the resemblance and containment of documents*. Proceedings of the Compression and Complexity of Sequences. MinHash estimates Jaccard similarity between sets using compact fixed-size signatures. -- **LSH (Locality-Sensitive Hashing)**: Indyk, P. & Motwani, R. (1998). *Approximate nearest neighbors: towards removing the curse of dimensionality*. STOC. Banded LSH reduces candidate pair generation from O(N²) to near-linear time. -- **AST-based clone detection**: Jiang, L. et al. (2007). *DECKARD: Scalable and accurate tree-based detection of code clones*. ICSE. Tree-based shingling captures structural similarity independent of identifier names. -- **Clone taxonomy**: Roy, C. K. & Cordy, J. R. (2007). *A survey on software clone detection research*. The Type-1/Type-2/Type-3 classification follows the standard clone taxonomy. - ---- - -## Limitations +## See also -- **Language scope**: Works with any language supported by the UAST parser. Unsupported files are silently skipped. -- **Structural only**: Detection is based on AST node types, not token values. Two functions with identical structure but completely different semantics (e.g., add vs multiply with the same nesting shape) may appear as clones. -- **Minimum function size**: Functions smaller than the shingle window (5 AST nodes) produce no shingles and are excluded from clone detection. They are still counted in `total_functions`. -- **Probabilistic recall**: LSH is a probabilistic algorithm. Clone pairs with similarity just above the threshold (0.5) may occasionally be missed. High-similarity pairs (Type-1 and Type-2) have near-perfect recall. -- **Generated code**: The analyzer does not distinguish hand-written code from generated code. Consider excluding generated directories. -- **No Type-4 clones**: Semantic clones (different structure, same behavior) are not detected. The analyzer operates purely on syntactic structure. +- [Understanding clone detection](../explanation/clones.md) — the mental model, methodology, references, and limitations. +- [Quick start](../getting-started/quickstart.md) — run static analysis. diff --git a/site/analyzers/cohesion.md b/site/analyzers/cohesion.md index 9ba81f56..76c00cb5 100644 --- a/site/analyzers/cohesion.md +++ b/site/analyzers/cohesion.md @@ -1,88 +1,16 @@ -# Cohesion Analyzer +# Cohesion analyzer reference -The cohesion analyzer computes **LCOM-HS (Henderson-Sellers)** and **variable sharing ratio** metrics to identify files and modules with low internal cohesion. Low cohesion indicates functions that are poorly related to each other — a strong signal for refactoring. +The cohesion analyzer computes **LCOM-HS (Henderson-Sellers)** and **variable +sharing ratio** metrics to identify files and modules with low internal +cohesion. ---- - -## Quick Start - -```bash -uast parse main.go | codefang analyze -a cohesion -``` - -Or analyze an entire directory: - -```bash -codefang analyze -a cohesion ./src/ -``` +For the conceptual model — what LCOM-HS measures and how to read the scores — +see [Understanding cohesion metrics](../explanation/cohesion.md). To run it, +see the [Quick start](../getting-started/quickstart.md). --- -## What It Measures - -### LCOM-HS (Lack of Cohesion of Methods — Henderson-Sellers) - -LCOM-HS is the industry-standard cohesion metric used by NDepend, JArchitect, and CppDepend. - -**Formula:** - -``` -LCOM = 1 - sum(mA) / (m × a) -``` - -Where: - -- `m` = number of functions in the file -- `a` = number of distinct variables across all functions -- `mA` = for each variable, the count of functions that reference it -- `sum(mA)` = sum of access counts across all variables - -**Range:** 0.0 (perfect cohesion) to 1.0 (no cohesion). - -!!! info "Interpretation" - - **LCOM ≤ 0.3** (green): Excellent cohesion — functions share most variables. - - **LCOM ≤ 0.6** (yellow): Moderate cohesion — some functions may be loosely related. - - **LCOM > 0.6** (red): Poor cohesion — functions share few variables and may belong in separate files. - -### Cohesion Score - -A convenience inversion of LCOM-HS: `cohesion_score = 1.0 - LCOM`. Higher is better. - -| Cohesion Score | Assessment | -|---|---| -| ≥ 0.7 | Excellent | -| ≥ 0.4 | Good | -| ≥ 0.3 | Fair | -| < 0.3 | Poor | - -### Function-Level Cohesion (Variable Sharing Ratio) - -For each function, the analyzer computes what fraction of its variables are shared with at least one other function in the same file. This measures **communicational cohesion** (Yourdon-Constantine level 5). - -**Formula:** - -``` -function_cohesion = shared_vars / total_unique_vars -``` - -Where: - -- `shared_vars` = variables in this function that also appear in at least one other function -- `total_unique_vars` = total distinct variables in this function -- Functions with no variables receive a score of 1.0 (trivial, no penalty) - -**Range:** 0.0 (completely isolated function) to 1.0 (all variables shared). - -| Function Cohesion | Assessment | -|---|---| -| ≥ 0.6 | Excellent — function shares most variables with the module | -| ≥ 0.4 | Good — reasonable sharing with room for improvement | -| ≥ 0.3 | Fair — consider refactoring | -| < 0.3 | Poor — function is isolated from the module | - ---- - -## Configuration Options +## Configuration options The cohesion analyzer uses the UAST directly and has no analyzer-specific configuration options. @@ -92,7 +20,7 @@ The cohesion analyzer uses the UAST directly and has no analyzer-specific config --- -## Example Output +## Example output === "Text" @@ -195,26 +123,7 @@ The cohesion analyzer uses the UAST directly and has no analyzer-specific config --- -## Use Cases - -- **Refactoring targets**: Find functions with low sharing ratio that do unrelated work and should be split. -- **Architecture reviews**: Validate that files follow the Single Responsibility Principle by checking LCOM-HS scores. -- **Code quality tracking**: Monitor cohesion trends over time to catch degradation. -- **Code review**: Use per-function cohesion scores to identify newly added functions that lack cohesion with existing code. - ---- - -## Methodology References - -- **LCOM-HS**: Henderson-Sellers, B. (1996). *Object-Oriented Metrics: Measures of Complexity*. Prentice Hall. The Henderson-Sellers variant normalizes LCOM to [0, 1] and is the variant used by NDepend, JArchitect, and CppDepend. -- **Variable Sharing Ratio**: Measures communicational cohesion per the Yourdon-Constantine classification. A function that shares all its variables with other functions in the module exhibits high communicational cohesion. - ---- - -## Limitations +## See also -- **Language scope**: Works with any language supported by the UAST. Best results with languages that have explicit variable declarations and function definitions. -- **Accessor methods**: Simple getters/setters may inflate cohesion since each touches only one shared field. -- **Variable naming**: The analyzer uses lexical variable names. Different variables with the same name across functions will be counted as shared. -- **Single-function files**: Files with only one function always receive perfect cohesion (LCOM = 0.0, cohesion = 1.0) since there are no other functions to compare against. -- **Trivial functions**: Functions with no variables receive a cohesion score of 1.0 to avoid penalizing simple utility functions. +- [Understanding cohesion metrics](../explanation/cohesion.md) — the mental model, formulas, references, and limitations. +- [Quick start](../getting-started/quickstart.md) — run static analysis. diff --git a/site/analyzers/comments.md b/site/analyzers/comments.md index 353f1da3..1ab238fb 100644 --- a/site/analyzers/comments.md +++ b/site/analyzers/comments.md @@ -1,47 +1,16 @@ -# Comments Analyzer +# Comments analyzer reference -The comments analyzer evaluates **documentation coverage** by examining comment presence, placement, and density across your source code. It identifies functions and classes that lack documentation and measures overall documentation quality. +The comments analyzer evaluates **documentation coverage** by examining comment +presence, placement, and density across your source code. ---- - -## Quick Start - -```bash -uast parse main.go | codefang analyze -a comments -``` - -Or analyze an entire directory: - -```bash -codefang analyze -a comments ./src/ -``` - ---- - -## What It Measures - -### Documentation Coverage - -The percentage of public functions, methods, and classes that have an associated documentation comment. A comment is "associated" when it immediately precedes the declaration. - -### Comment Placement Quality - -Not all comments are equally useful. The analyzer distinguishes: - -- **Doc comments**: Comments immediately before a function/class declaration -- **Inline comments**: Comments within function bodies -- **Orphan comments**: Comments not associated with any code structure - -### Comment Density - -The ratio of comment lines to total lines of code. Provides a quick overall measure of how well-documented a file is. - -!!! info "Healthy ratios" - A comment density between **15-30%** is generally considered healthy. Below 10% suggests under-documentation; above 40% may indicate stale or redundant comments. +For the conceptual model — what documentation coverage means and what a healthy +comment density looks like — see +[Understanding documentation coverage](../explanation/comments.md). To run it, +see the [Quick start](../getting-started/quickstart.md). --- -## Configuration Options +## Configuration options The comments analyzer uses the UAST directly and has no analyzer-specific configuration options. @@ -51,7 +20,7 @@ The comments analyzer uses the UAST directly and has no analyzer-specific config --- -## Example Output +## Example output === "JSON" @@ -98,18 +67,7 @@ The comments analyzer uses the UAST directly and has no analyzer-specific config --- -## Use Cases - -- **Documentation enforcement**: Require minimum documentation coverage in CI/CD pipelines. -- **Onboarding assessment**: Measure how well-documented a codebase is before new team members join. -- **API surface quality**: Ensure all exported/public functions have doc comments. -- **Stale comment detection**: Identify files with unusually high comment density that may contain outdated comments. - ---- - -## Limitations +## See also -- **Comment quality**: The analyzer measures comment *presence* and *placement*, not comment *quality*. A comment saying `// do the thing` counts the same as a thorough description. -- **Language conventions**: Documentation comment conventions vary by language (e.g., `///` in Rust, `"""` in Python, `//` in Go). The UAST normalizes these, but edge cases in less common languages may be missed. -- **Auto-generated docs**: Comments generated by tools (e.g., protobuf stubs) are counted the same as hand-written documentation. -- **Multilingual comments**: Comment text is not analyzed for language quality or correctness. +- [Understanding documentation coverage](../explanation/comments.md) — the mental model, comment types, and limitations. +- [Quick start](../getting-started/quickstart.md) — run static analysis. diff --git a/site/analyzers/complexity.md b/site/analyzers/complexity.md index c6e145b5..0f47b206 100644 --- a/site/analyzers/complexity.md +++ b/site/analyzers/complexity.md @@ -1,61 +1,18 @@ -# Complexity Analyzer +# Complexity analyzer reference -The complexity analyzer measures three dimensions of code complexity from source code: **cyclomatic complexity**, **cognitive complexity**, and **nesting depth**. It operates on the UAST representation of your source files. +The complexity analyzer measures three dimensions of code complexity from +source code: **cyclomatic complexity**, **cognitive complexity**, and +**nesting depth**. It operates on the UAST representation of your source +files. ---- - -## Quick Start - -```bash -uast parse main.go | codefang analyze -a complexity -``` - -Or analyze an entire directory: - -```bash -codefang analyze -a complexity ./src/ -``` +For the conceptual model — what each metric means and how cyclomatic and +cognitive complexity differ — see +[Understanding complexity metrics](../explanation/complexity.md). To run it, +see the [Quick start](../getting-started/quickstart.md). --- -## What It Measures - -### Cyclomatic Complexity (McCabe, 1976) - -Counts the number of linearly independent paths through a function's control flow graph. - -For Go code, the analyzer follows the same practical counting model used by `gocyclo`: - -- Base score is `1` -- `if`, `for` / `range` add `+1` -- Non-default `case` adds `+1` -- Each `&&` and `||` adds `+1` -- `default` case does **not** add complexity - -!!! info "Interpretation" - - **1-10**: Simple, low risk - - **11-20**: Moderate complexity - - **21-50**: High complexity, consider refactoring - - **51+**: Very high risk, untestable - -### Cognitive Complexity (SonarSource, 2017) - -Measures how difficult code is for a human to read and understand. Unlike cyclomatic complexity, it penalizes nested structures more heavily and rewards linear sequences of logic. - -Key differences from cyclomatic complexity: - -- Nesting increments add to the score (deeper nesting = higher penalty) -- `else if` adds structural complexity, but avoids extra nesting penalty vs deeply nested `if` -- Logical operator sequences account for readability, not just path count -- Direct recursion adds a penalty - -### Nesting Depth - -Tracks the maximum depth of nested control structures within each function. Deep nesting is a strong signal for refactoring. - ---- - -## Configuration Options +## Configuration options The complexity analyzer uses the UAST directly and has no analyzer-specific configuration options. Language support is determined by the UAST parser. @@ -65,7 +22,7 @@ The complexity analyzer uses the UAST directly and has no analyzer-specific conf --- -## Example Output +## Example output === "JSON" @@ -139,18 +96,7 @@ The complexity analyzer uses the UAST directly and has no analyzer-specific conf --- -## Validation Against Golden Implementations - -The complexity analyzer is continuously validated against battle-tested Go references: - -- **Cyclomatic parity target:** `gocyclo` (v0.6.0) -- **Cognitive parity target:** `gocognit` (v1.2.1) - -For stabilization, we run a controlled methodology sample through both references and `codefang`, then align discrepancies until parity is reached. - ---- - -## Terminal UX (Readable Risk Triage) +## Terminal output (readable risk triage) Top issue rows include all core complexity dimensions in one compact value: @@ -167,7 +113,7 @@ Issues are sorted numerically by cyclomatic complexity (then cognitive, then nes --- -## Plot UX (Fast Visual Scanning) +## Plot output (fast visual scanning) The plot output emphasizes fast interpretation for code-review and planning: @@ -177,18 +123,7 @@ The plot output emphasizes fast interpretation for code-review and planning: --- -## Use Cases - -- **Code review gates**: Reject pull requests where any function exceeds a cyclomatic complexity threshold. -- **Refactoring prioritization**: Sort functions by cognitive complexity to find the hardest-to-understand code. -- **Technical debt tracking**: Monitor complexity trends across releases. -- **Test planning**: Functions with high cyclomatic complexity need more test cases for full path coverage. - ---- - -## Limitations +## See also -- **Language coverage**: Only languages supported by the UAST parser are analyzed. Unsupported files are silently skipped. -- **Generated code**: The analyzer does not distinguish hand-written code from generated code. Consider excluding generated directories. -- **Macros and metaprogramming**: Complexity within macros or template metaprogramming may not be fully captured, since the UAST represents the source as written, not as expanded. -- **Cognitive complexity model**: The cognitive complexity scoring follows the SonarSource specification. Other tools may use slightly different weightings. +- [Understanding complexity metrics](../explanation/complexity.md) — the mental model, McCabe vs cognitive, and limitations. +- [Quick start](../getting-started/quickstart.md) — run static analysis. diff --git a/site/analyzers/couples.md b/site/analyzers/couples.md index dfb5234e..8c8a0e03 100644 --- a/site/analyzers/couples.md +++ b/site/analyzers/couples.md @@ -1,49 +1,16 @@ -# Couples Analyzer +# Couples analyzer reference -The couples analyzer identifies **file coupling** and **developer coupling** by analyzing co-change patterns across Git history. Files that frequently change together are likely coupled; developers who frequently touch the same files are collaborating (or competing). +The couples analyzer identifies **file coupling** and **developer coupling** by +analyzing co-change patterns across Git history. ---- - -## Quick Start - -```bash -codefang run -a history/couples . -``` +For the conceptual model — what coupling means, how coupling strength is +computed, and the methodology — see +[Understanding file and developer coupling](../explanation/couples.md). To run +it, see the [Quick start](../getting-started/quickstart.md). --- -## What It Measures - -### File Coupling - -A square matrix where each cell `[i][j]` counts the number of commits in which files `i` and `j` both appeared. High co-change counts indicate tight coupling between files. - -**Coupling strength** is computed using the code-maat formula: - -``` -strength = co_changes / average(revisions_file1, revisions_file2) -``` - -Where `revisions_fileN` is the diagonal element of the file matrix (the file's self-change count). The result is capped at 1.0. This normalizes coupling by the average activity of both files, so a pair of files that change together 10 times out of 20 average revisions scores 50%. - -!!! info "Changeset size filter" - Commits touching more than **1000 files** are excluded from coupling analysis. Mass changes (e.g., formatting, license updates, dependency bumps) produce noise rather than meaningful coupling signal. - -### Developer Coupling - -A matrix where each cell `[i][j]` counts the number of times developers `i` and `j` committed to the same file. Developer coupling strength uses the same code-maat formula normalized by each developer's individual commit activity. - -### File Ownership - -For each tracked file, the analyzer reports its line count and the number of distinct contributors. Single-owner files are flagged as bus-factor risks. - -### Rename Tracking - -The analyzer tracks file renames to avoid counting a renamed file as both a deletion and an insertion, which would break coupling chains. - ---- - -## Configuration Options +## Configuration options The couples analyzer has no additional configuration options. It uses the identity detector for developer mapping and the tree diff analyzer for change detection. @@ -53,7 +20,7 @@ The couples analyzer has no additional configuration options. It uses the identi --- -## Output Formats +## Output formats ### Text @@ -76,7 +43,7 @@ The HTML plot includes three interactive chart sections: Structured output with four top-level sections: `file_coupling`, `developer_coupling`, `file_ownership`, and `aggregate`. -### Executive Summary (ReportSection) +### Executive summary (ReportSection) The couples analyzer provides a `ReportSection` for use in combined reports: @@ -87,7 +54,7 @@ The couples analyzer provides a `ReportSection` for use in combined reports: --- -## Example Output +## Example output === "JSON" @@ -161,50 +128,7 @@ The couples analyzer provides a `ReportSection` for use in combined reports: --- -## Use Cases - -- **Architecture analysis**: Highly coupled file pairs that span package boundaries may indicate architectural violations. -- **Dependency discovery**: Find implicit dependencies that are not captured by import statements. If two files always change together, they are coupled even without explicit imports. -- **Module boundary validation**: Files within the same module should be more coupled than files across modules. Cross-module coupling is a design smell. -- **Team topology**: Developer coupling reveals who collaborates with whom. This can inform team structure decisions. -- **Change impact prediction**: When modifying a file, the coupling matrix predicts which other files are likely to need changes. -- **Bus factor assessment**: File ownership data identifies single-owner files that represent knowledge concentration risks. - ---- - -## Architecture - -The couples analyzer follows the **TC/Aggregator pattern**: - -1. **Consume phase**: Each commit produces a `TC{Data: *CommitData}` containing the coupling context (list of co-changed files), per-file author touch counts (always 1 per file per commit), rename pairs, and whether the author's commit count was incremented. Commits exceeding `CouplesMaximumMeaningfulContextSize` (1000 files) are skipped, producing an empty coupling context. - -2. **Aggregation phase**: The `Aggregator` accumulates the file co-occurrence matrix using `SpillStore[map[string]int]`, along with per-person file touch counts (`people`), commit counts (`peopleCommits`), and rename tracking. When memory pressure exceeds `SpillBudget`, the file coupling matrix is spilled to disk via gob encoding. `Collect()` merges spilled data using additive merge semantics. - -3. **Serialization phase**: `ticksToReport()` reconstructs the full report (PeopleMatrix, PeopleFiles, Files, FilesLines, FilesMatrix, ReversedPeopleDict) from aggregated TICKs, then `ComputeAllMetrics()` produces the final typed output for any format (text, plot, JSON, YAML, binary). - -**Working state** (`merges`, `seenFiles`) stays in the analyzer for merge-mode dedup across commits. **Accumulated output** (file couplings, people maps, renames) is owned entirely by the aggregator. - ---- - -## Methodology - -The coupling strength formula follows the **code-maat** academic standard (Adam Tornhill, "Your Code as a Crime Scene"): - -``` -degree = shared_revisions / average(revisions_A, revisions_B) -``` - -This measures what fraction of a file pair's average activity is shared. A coupling strength of 80% means the pair changes together in 80% of their average revision count. - -The aggregate `avg_coupling_strength` is the mean of all per-pair coupling strengths (not the mean of raw co-change counts). - -**Highly coupled pairs** are those with 10 or more co-changes (raw count threshold). - ---- - -## Limitations +## See also -- **Large commits excluded**: Commits touching more than 1000 files are skipped to avoid noise from mass changes (formatting, license headers, dependency updates). -- **No temporal decay**: All commits are weighted equally. A coupling from three years ago counts the same as one from yesterday. Consider filtering by date range for recent coupling analysis. -- **Merge commits**: Merge commits are processed only once (first encounter) to avoid double-counting changes that appear in multiple branches. -- **File deletions**: Deleted files are included in the coupling matrix during the analysis window but may not appear in the final output if they no longer exist at HEAD. +- [Understanding file and developer coupling](../explanation/couples.md) — the mental model, architecture, methodology, and limitations. +- [Quick start](../getting-started/quickstart.md) — run history analysis. diff --git a/site/analyzers/developers.md b/site/analyzers/developers.md index c527510e..6c3577ae 100644 --- a/site/analyzers/developers.md +++ b/site/analyzers/developers.md @@ -1,94 +1,16 @@ -# Developers Analyzer +# Developers analyzer reference -The developers analyzer computes **per-developer contribution statistics** across Git history, including commit counts, line changes, language breakdown, bus factor risk, and activity time series. +The developers analyzer computes **per-developer contribution statistics** +across Git history, including commit counts, line changes, language breakdown, +bus factor risk, and activity time series. ---- - -## Quick Start - -```bash -codefang run -a history/devs . -``` - -With anonymization: - -```bash -codefang run -a history/devs --anonymize . -``` - -Text output (terminal-friendly): - -```bash -codefang run -a history/devs -f text . -``` - ---- - -## Architecture - -The developers analyzer is built on the **BaseHistoryAnalyzer** and **GenericAggregator** foundations: - -1. **Consume phase**: `Consume()` extracts author ID, line stats, and language breakdown, delegating state storage to the generic aggregator framework. The analyzer retains minimal internal state. -2. **Aggregation phase**: The `GenericAggregator` automatically handles per-commit memory spilling and per-tick grouping using pure function hooks (`extractTC`, `mergeState`, `buildTick`), eliminating custom state management boilerplate. -3. **Metrics & UI phase**: A pure function pipeline (`ComputeAllMetrics`) generates the output structures for JSON, YAML, and text, while declarative chart builders (`plotpage.BuildBarChart`, `plotpage.BuildLineChart`) render the HTML visualizations. - -This unified approach significantly reduces boilerplate while maintaining full support for streaming output, decoupled aggregation, and budget-aware memory spilling. - ---- - -## What It Measures - -### Developer Statistics - -For each developer (identified by the identity detector): - -- **Commits**: Total number of commits -- **Lines added / removed / changed**: Aggregate line statistics -- **Net lines**: `added - removed`, showing net contribution -- **Languages**: Breakdown of line changes per programming language -- **Active period**: First and last ticks of activity, number of active ticks - -### Language Statistics - -Aggregated across all developers: - -- **Total lines per language**: Lines added (for backward compatibility) -- **Total contribution per language**: Lines added + removed, used for bus factor and ownership calculations -- **Contributors per language**: Which developers contribute to each language, measured by total contribution (added + removed) - -### Bus Factor - -Knowledge concentration risk per language, following the [CHAOSS Contributor Absence Factor](https://chaoss.community/kb/metric-bus-factor/) methodology. - -For each language, the analyzer computes: - -- **Bus factor number**: The smallest number of contributors responsible for 50% of total contributions (added + removed). This is the CHAOSS standard metric. -- **Total contributors**: Number of unique contributors to that language. -- **Primary/secondary owner**: The top two contributors and their ownership percentages. -- **Risk level**: Based on primary owner concentration. - -!!! danger "Risk levels" - - **CRITICAL** (>= 90%): A single developer owns nearly all the code - - **HIGH** (>= 80%): Very concentrated ownership - - **MEDIUM** (>= 60%): Moderate concentration - - **LOW** (< 60%): Healthy distribution - -A **project-level bus factor** is also computed across all developers and all languages, reported in the aggregate section. - -### Activity Time Series - -Per-tick commit counts broken down by developer. Shows contribution velocity over time. - -### Code Churn (Line Velocity) - -Per-tick lines added and removed. This measures raw line velocity — the volume of code changes over time. High values may indicate refactoring, feature development, or instability. - -!!! note "Terminology" - In academic literature, "code churn" specifically refers to recently-written code that is quickly rewritten. This analyzer measures the broader concept of line velocity (total additions and removals per time period). +For the conceptual model — what each statistic means and how the bus factor is +derived — see [Understanding developer analysis](../explanation/developers.md). +To run it, see the [Quick start](../getting-started/quickstart.md). --- -## Configuration Options +## Configuration options | Option | Type | Default | Description | |---|---|---|---| @@ -105,7 +27,7 @@ history: --- -## Example Output +## Example output === "JSON" @@ -228,21 +150,7 @@ history: --- -## Use Cases - -- **Team assessment**: Understand who contributes what, in which languages, and when. -- **Bus factor analysis**: Identify languages or components where a single developer departure would create critical knowledge gaps. The CHAOSS bus factor number tells you how many people need to leave before 50% of knowledge is lost. -- **Activity monitoring**: Track developer engagement over time. Declining activity may signal burnout or attrition risk. -- **Language migration tracking**: Monitor the adoption of a new language by watching language statistics over time. -- **Onboarding evaluation**: Measure how quickly new team members ramp up by comparing their activity curves. -- **Code churn analysis**: Detect periods of high line velocity that may correlate with instability or deadline pressure. - ---- - -## Limitations +## See also -- **Identity resolution**: Developer identity is determined by the identity detector (email-based by default). Multiple email addresses for the same person will appear as separate developers unless a mailmap is configured. -- **Merge commits**: By default, merge commits are processed only once (first encounter). Trivial merges are skipped unless `ConsiderEmptyCommits` is enabled. -- **Line attribution**: Lines are attributed to the commit author, not the committer. In workflows with heavy rebasing, this may differ from expectations. -- **Contribution measurement**: Contributions are measured as lines added + lines removed. This gives fair credit to refactoring work but does not distinguish between code complexity or quality. -- **Active developer threshold**: "Active developers" are those with commits in the last 90 days (when tick size is known). Falls back to the recent 30% of the analysis period when tick size is unavailable. This threshold is not configurable. +- [Understanding developer analysis](../explanation/developers.md) — the mental model, bus factor methodology, architecture, and limitations. +- [Quick start](../getting-started/quickstart.md) — run history analysis. diff --git a/site/analyzers/file-history.md b/site/analyzers/file-history.md index d5c3a1bb..b0f54297 100644 --- a/site/analyzers/file-history.md +++ b/site/analyzers/file-history.md @@ -1,48 +1,17 @@ -# File History Analyzer +# File history analyzer reference -The file history analyzer tracks the **lifecycle of every file** through Git history, recording which commits touched each file, which developers modified it, and aggregating line statistics per contributor. It supports rename detection to maintain continuity across file moves. +The file history analyzer tracks the **lifecycle of every file** through Git +history, recording which commits touched each file, which developers modified +it, and aggregating line statistics per contributor. ---- - -## Quick Start - -```bash -codefang run -a history/file-history . -``` +For the conceptual model — what the lifecycle data means and how hotspots and +churn are derived — see +[Understanding file history analysis](../explanation/file-history.md). To run +it, see the [Quick start](../getting-started/quickstart.md). --- -## What It Measures - -### Per-File Commit History - -For each file present in the repository at HEAD, the analyzer records the ordered list of commits that created, modified, or deleted it. - -### Per-File Contributor Breakdown - -For each file, a map of developer IDs to line statistics (added, removed, changed). This shows who contributed what to each file. - -### Code Hotspots - -Files with high commit counts are flagged as hotspots with risk levels: - -!!! warning "Hotspot risk levels" - - **CRITICAL**: >= 50 commits - - **HIGH**: >= 30 commits - - **MEDIUM**: >= 15 commits - - Files below 15 commits are not flagged - -### File Churn - -A composite score combining commit frequency and line change volume. High-churn files may indicate instability or areas of active development. - -### Rename Support - -When Git detects a file rename (e.g., `old/path.go` to `new/path.go`), the analyzer transfers the full history from the old path to the new path, maintaining a continuous record. - ---- - -## Configuration Options +## Configuration options The file history analyzer has no additional configuration options. @@ -52,7 +21,7 @@ The file history analyzer has no additional configuration options. --- -## Example Output +## Example output === "JSON" @@ -131,20 +100,7 @@ The file history analyzer has no additional configuration options. --- -## Use Cases - -- **Hotspot identification**: Find the files that change most often. These are the highest-risk files for bugs and the best candidates for extra test coverage and code review scrutiny. -- **Code ownership mapping**: Determine who is the primary contributor for each file to establish code ownership. -- **Onboarding guides**: New team members can see which files each developer owns to know who to ask about specific code. -- **Refactoring ROI**: Identify files with both high churn and many contributors -- refactoring these produces the largest payoff. -- **Risk assessment**: Files with a single contributor and high commit counts are both hot and concentrated -- a bus factor risk. - ---- - -## Limitations +## See also -- **HEAD-only output**: Only files present at HEAD are included in the final output. Files that were deleted before the last commit are tracked during analysis but excluded from results. -- **Merge handling**: Merge commits are processed only once to avoid double-counting. Non-merge context changes in merge commits are skipped. -- **Rename detection**: Depends on Git's rename detection heuristics. If a file is both renamed and heavily modified in the same commit, Git may not detect the rename. -- **Churn score formula**: The churn score is `commit_count + (added + removed + changed) / 100`. This is a simple heuristic, not a rigorous statistical measure. -- **No time-series**: Unlike the burndown analyzer, file history does not produce time-series data. It provides aggregate per-file statistics only. +- [Understanding file history analysis](../explanation/file-history.md) — the mental model, hotspots, churn, and limitations. +- [Quick start](../getting-started/quickstart.md) — run history analysis. diff --git a/site/analyzers/halstead.md b/site/analyzers/halstead.md index 15f0001d..897e20dd 100644 --- a/site/analyzers/halstead.md +++ b/site/analyzers/halstead.md @@ -1,58 +1,16 @@ -# Halstead Analyzer +# Halstead analyzer reference -The Halstead analyzer computes **Halstead complexity metrics** (1977) based on operator and operand counts extracted from the UAST. These metrics provide an objective, quantitative assessment of program size and complexity. +The Halstead analyzer computes **Halstead complexity metrics** (1977) based on +operator and operand counts extracted from the UAST. ---- - -## Quick Start - -```bash -# Analyze a repository or directory -codefang run -a static/halstead . - -# Human-readable terminal output -codefang run -a static/halstead --format text . - -# Interactive plots -codefang run -a static/halstead --format plot . > halstead.html -``` +For the conceptual model — what operators and operands are and which derived +metrics matter most — see +[Understanding Halstead metrics](../explanation/halstead.md). To run it, see +the [Quick start](../getting-started/quickstart.md). --- -## What It Measures - -The Halstead model treats a program as a sequence of **operators** (keywords, symbols, function calls) and **operands** (variables, literals, constants). From these counts, it derives: - -| Metric | Formula | Description | -|---|---|---| -| **n1** | -- | Number of distinct operators | -| **n2** | -- | Number of distinct operands | -| **N1** | -- | Total number of operators | -| **N2** | -- | Total number of operands | -| **Vocabulary** (n) | n1 + n2 | Distinct tokens used | -| **Length** (N) | N1 + N2 | Total tokens in the program | -| **Volume** (V) | N * log2(n) | Information content in bits | -| **Difficulty** (D) | (n1/2) * (N2/n2) | Error proneness | -| **Effort** (E) | D * V | Mental effort to implement | -| **Time** (T) | E / 18 | Estimated implementation time (seconds) | -| **Bugs** (B) | V / 3000 | Estimated delivered bugs | - -!!! info "Key insight" - **Volume** measures the size of the implementation. **Difficulty** captures how error-prone it is. Their product, **Effort**, is the best single number for comparing overall complexity. - -### Counting policy - -Codefang computes Halstead metrics from UAST with a lexical-first policy: - -- Counts lexical operands (`Identifier`, `Literal`, `Field`) and operator nodes/roles. -- Excludes structural wrappers and declaration-only artifacts from operand counts. -- Uses explicit operator properties when available, then token extraction fallback. - -This improves stability across languages and avoids pseudo-operands like structural `Parameter` nodes. - ---- - -## Configuration Options +## Configuration options The Halstead analyzer uses the UAST directly and has no analyzer-specific configuration options. @@ -62,11 +20,11 @@ The Halstead analyzer uses the UAST directly and has no analyzer-specific config --- -## Output Interpretation +## Output formats ### Terminal (text/summary) -The section now surfaces foundational counts before derived metrics: +The section surfaces foundational counts before derived metrics: - `Distinct Operators (n1)` - `Distinct Operands (n2)` @@ -92,19 +50,9 @@ The plot report includes: - color: low/medium/high risk bucket 3. **Volume Distribution** by bucket (`Low`, `Medium`, `High`, `Very High`) -## Use Cases - -- **Effort estimation**: Use the Effort metric to compare the relative complexity of different modules or features. -- **Bug prediction**: The Bugs metric provides a rough upper bound on expected defects. Modules with high estimated bugs warrant more thorough testing. -- **Code review guidance**: Functions with high Difficulty scores are more error-prone and deserve extra scrutiny. -- **Language comparison**: Halstead metrics allow cross-language comparisons since they are based on abstract operator/operand counts. - --- -## Limitations +## See also -- **Cross-tool variance**: Language-specific tools (for example, Python- or ESTree-only analyzers) can differ in tokenization and counting policy. Compare trends within one toolchain rather than mixing absolute numbers across tools. -- **Operator classification**: UAST-based classification targets cross-language consistency, not byte-for-byte parity with each language parser. -- **Estimation accuracy**: The Bugs and Time formulas are empirical approximations from the 1970s. Treat them as relative indicators, not precise predictions. -- **Macro expansion**: Halstead metrics count tokens as written, not as expanded. Heavy use of macros or code generation can skew results. -- **Comments excluded**: Comments and whitespace are excluded from Halstead counts (by design). +- [Understanding Halstead metrics](../explanation/halstead.md) — the mental model, formulas, counting policy, and limitations. +- [Quick start](../getting-started/quickstart.md) — run static analysis. diff --git a/site/analyzers/imports.md b/site/analyzers/imports.md index 16345c3f..23f59e27 100644 --- a/site/analyzers/imports.md +++ b/site/analyzers/imports.md @@ -1,62 +1,23 @@ -# Imports Analyzer +# Imports analyzer reference -The imports analyzer extracts **import and dependency information** from source code using UAST parsing. It operates in both static mode (single-file analysis) and history mode (tracking import usage per developer over time). +The imports analyzer extracts **import and dependency information** from source +code using UAST parsing. It operates in both static mode (single-file analysis) +and history mode (tracking import usage per developer over time). ---- - -## Quick Start - -=== "Static mode" - - ```bash - uast parse main.go | codefang analyze -a imports - ``` - -=== "History mode" - - ```bash - codefang run -a history/imports . - ``` - ---- - -## Architecture (History Mode) - -The imports history analyzer follows the **TC/Aggregator pattern**: - -1. **Consume phase**: For each commit, `Consume()` extracts imports from changed files via parallel UAST parsing and returns them as `TC{Data: []ImportEntry}`. Each `ImportEntry` carries a language and import path. The analyzer retains no per-commit state; only the UAST parser is kept as working state. -2. **Aggregation phase**: An `imports.Aggregator` collects TCs into a 4-level `Map` (author -> language -> import -> tick -> count) using `SpillStore[Map]`. The `AuthorID` and `Tick` from each TC index the entries correctly. -3. **Serialization phase**: `SerializeTICKs()` merges all tick data back into the full `Map` with metadata (`author_index`, `tick_size`), then delegates to `Serialize()` for JSON, YAML, binary, or HTML plot output. - -This separation enables streaming output, budget-aware memory spilling, and decoupled aggregation. +For the conceptual model — what it measures and how the history-mode pipeline +works — see [Understanding import and dependency +analysis](../explanation/imports.md). To run it, see the +[Quick start](../getting-started/quickstart.md). --- -## What It Measures - -### Static Mode - -- **Import list**: All imports/dependencies declared in each file -- **Language detection**: Automatically detects the language and normalizes import paths -- **Dependency graph**: Maps which files depend on which packages - -### History Mode +## Configuration options -Tracks import usage across Git history, producing a per-developer, per-language, per-tick breakdown of which dependencies each developer introduces or modifies. This reveals: - -- **Dependency adoption timeline**: When new libraries were introduced -- **Developer expertise signals**: Which developers work with which dependencies -- **Technology spread**: How quickly new dependencies propagate across the team - ---- - -## Configuration Options - -### Static Mode +### Static mode No configuration options. Uses UAST directly. -### History Mode +### History mode | Option | Type | Default | Description | |---|---|---|---| @@ -75,7 +36,7 @@ history: --- -## Example Output +## Example output === "Static (JSON)" @@ -111,20 +72,7 @@ history: --- -## Use Cases - -- **Dependency auditing**: List all third-party dependencies used in a project. -- **Developer profiling**: Understand which developers work with which frameworks and libraries. -- **Technology adoption tracking**: Monitor when and how quickly new dependencies spread across the team. -- **License compliance**: Extract the full dependency list for license scanning pipelines. -- **Architecture enforcement**: Detect unauthorized imports from forbidden packages. - ---- - -## Limitations +## See also -- **Language support**: Only languages with UAST parser support are analyzed. Unsupported file types are silently skipped. -- **Dynamic imports**: Runtime or dynamic imports (e.g., Python's `importlib.import_module()`, JavaScript's `import()`) are not detected since they are not present in the static UAST. -- **Transitive dependencies**: Only direct imports are reported. The analyzer does not resolve transitive dependency trees. -- **File size threshold**: Files exceeding `MaxFileSize` (default 1 MB) are skipped to avoid excessive memory usage during parallel extraction. -- **History mode overhead**: History mode creates a UAST parser per fork, which increases memory usage. Tune `Goroutines` based on available memory. +- [Understanding import and dependency analysis](../explanation/imports.md) — the mental model, history-mode architecture, and limitations. +- [Quick start](../getting-started/quickstart.md) — run static and history analysis. diff --git a/site/analyzers/index.md b/site/analyzers/index.md index ea22d777..b8c06520 100644 --- a/site/analyzers/index.md +++ b/site/analyzers/index.md @@ -6,7 +6,7 @@ Codefang ships two families of analyzers that extract quantitative insights from --- -## Static Analyzers +## Static analyzers Static analyzers parse source files into a UAST and compute metrics in a single pass. They require no repository history and can run on individual files or entire directory trees. @@ -19,7 +19,7 @@ Static analyzers parse source files into a UAST and compute metrics in a single | [Imports](imports.md) | `imports` | Import and dependency analysis | | [Clones](clones.md) | `clones` | Cross-file duplicate function detection via MinHash + LSH | -### Running Static Analyzers +### Running static analyzers Pipe a UAST through `codefang analyze`: @@ -36,7 +36,7 @@ codefang analyze -a complexity ./src/ --- -## History Analyzers +## History analyzers History analyzers iterate over Git commits (oldest to newest) and accumulate statistics about how files, developers, and code structure change over time. They require a Git repository. @@ -52,7 +52,7 @@ History analyzers iterate over Git commits (oldest to newest) and accumulate sta | [Typos](typos.md) | `history/typos` | Typo detection dataset builder | | [Anomaly](anomaly.md) | `history/anomaly` | Z-score temporal anomaly detection | -### Running History Analyzers +### Running history analyzers Use `codefang run` with the `-a` flag: @@ -69,7 +69,7 @@ codefang run -a 'history/*' . --- -## Glob Patterns +## Glob patterns You can select analyzers using glob patterns with the `-a` flag: @@ -86,7 +86,7 @@ codefang run -a '*' . --- -## Output Formats +## Output formats All analyzers support multiple output formats via the `-f` flag: diff --git a/site/analyzers/quality.md b/site/analyzers/quality.md index c1dc6a19..72377267 100644 --- a/site/analyzers/quality.md +++ b/site/analyzers/quality.md @@ -1,57 +1,28 @@ -# Code Quality Analyzer +# Code quality analyzer reference -The code quality analyzer tracks **complexity, Halstead metrics, comment quality, and cohesion** across Git history. For each commit, it runs four static analyzers on UAST-parsed changed files and produces per-file quality metrics. +The code quality analyzer tracks **complexity, Halstead metrics, comment +quality, and cohesion** across Git history. For each commit, it runs four +static analyzers on UAST-parsed changed files and produces per-file quality +metrics. ---- - -## Quick Start - -```bash -codefang run -a history/quality . -``` - -!!! note "Requires UAST" - The quality analyzer needs UAST support to parse source code. It is automatically enabled when the UAST pipeline is available. - ---- - -## Architecture - -The quality analyzer follows the **TC/Aggregator pattern**: - -1. **Consume phase**: For each commit, `Consume()` runs four static analyzers (complexity, Halstead, comments, cohesion) on each changed file's UAST, returning per-file metrics as a `TC{Data: *TickQuality}`. The analyzer retains no per-commit state. -2. **Aggregation phase**: A `quality.Aggregator` collects TCs, merges `TickQuality` data by time bucket (tick), and produces `TICK` results. -3. **Serialization phase**: `SerializeTICKs()` converts aggregated TICKs into JSON, YAML, binary, or HTML plot output via `ComputeAllMetrics()`. - -This separation enables streaming output, budget-aware memory spilling, and decoupled aggregation. +For the conceptual model — which metrics it tracks and how it aggregates them +per tick — see [Understanding code quality over +time](../explanation/quality.md). To run it, see the +[Quick start](../getting-started/quickstart.md). --- -## What It Measures - -### Per-File Metrics - -For each changed file in a commit, the analyzer computes: +## Configuration options -| Category | Metrics | -|---|---| -| **Complexity** | Cyclomatic complexity, cognitive complexity, max single-function complexity, function count | -| **Halstead** | Volume, effort, delivered bugs | -| **Comments** | Overall comment score, documentation coverage | -| **Cohesion** | Cohesion score | - -### Statistical Aggregation - -Per-tick statistics are computed from per-file arrays: +The quality analyzer has no configurable options. It uses the UAST pipeline's language detection and parsing configuration. -- **Mean, median, P95, max** for complexity and Halstead volume -- **Sum** for delivered bugs and Halstead volume -- **Min** for comment score and cohesion (worst-case tracking) -- **Total** files analyzed and functions counted +| Option | Type | Default | Description | +|---|---|---|---| +| *(none)* | -- | -- | Uses UAST; no analyzer-specific config | --- -## Output Formats +## Output formats | Format | Flag | Description | |---|---|---| @@ -62,6 +33,7 @@ Per-tick statistics are computed from per-file arrays: --- -## Configuration +## See also -The quality analyzer has no configurable options. It uses the UAST pipeline's language detection and parsing configuration. +- [Understanding code quality over time](../explanation/quality.md) — the mental model, per-file metrics, and architecture. +- [Quick start](../getting-started/quickstart.md) — run history analysis. diff --git a/site/analyzers/sentiment.md b/site/analyzers/sentiment.md index d5c3f472..57e05555 100644 --- a/site/analyzers/sentiment.md +++ b/site/analyzers/sentiment.md @@ -1,119 +1,17 @@ -# Sentiment Analyzer +# Sentiment analyzer reference -The sentiment analyzer classifies **comment sentiment** across Git history. For each commit, it extracts new or changed comments via UAST parsing, filters out noise, and classifies the remaining comments as positive, negative, or neutral. This reveals how developer sentiment evolves over time. +The sentiment analyzer classifies **comment sentiment** across Git history, +extracting new or changed comments via UAST parsing and classifying them as +positive, negative, or neutral. ---- - -## Quick Start - -```bash -codefang run -a history/sentiment . -``` - -!!! note "Requires UAST" - The sentiment analyzer needs UAST support to extract comments. It is automatically enabled when the UAST pipeline is available. +For the conceptual model — how comments are filtered and scored, the +multilingual lexicon, and the software-engineering domain adjustments — see +[Understanding comment sentiment analysis](../explanation/sentiment.md). To run +it, see the [Quick start](../getting-started/quickstart.md). --- -## Architecture - -The sentiment analyzer follows the **TC/Aggregator pattern**: - -1. **Consume phase**: For each commit, `Consume()` extracts and filters comments from UAST changes, returning them as a `TC` (per-commit result). The analyzer retains no per-commit state. -2. **Aggregation phase**: A `sentiment.Aggregator` collects TCs, groups comments by time bucket (tick), computes sentiment scores, and produces `TICK` results. -3. **Serialization phase**: `SerializeTICKs()` converts aggregated TICKs into JSON, YAML, binary, or HTML plot output. - -This separation enables streaming output, budget-aware memory spilling, and decoupled aggregation. - ---- - -## What It Measures - -### Per-Commit Comment Extraction - -For each commit, the analyzer: - -1. Computes the UAST diff between the old and new file versions -2. Extracts comment nodes from the new UAST -3. Merges adjacent comment lines into single blocks -4. Filters out noise (short comments, license headers, function signatures) -5. Returns the filtered comments as a `TC` payload (`sentiment.CommitResult`) - -### Comment Filtering - -Comments are filtered using several heuristics: - -- **Minimum length**: Comments shorter than the threshold are skipped -- **Letter ratio**: At least 60% of characters must be Unicode letters (filters out commented-out code) -- **First character**: Must start with a Unicode letter or digit (supports all scripts) -- **License detection**: Comments matching license/copyright patterns (including UK "licence" spelling) are excluded -- **Function name removal**: Inline function references like `doThing()` are stripped before analysis - -### Multilingual Support - -The analyzer supports comments in any language that uses Unicode letters: - -- **CJK**: Chinese, Japanese, Korean -- **Cyrillic**: Russian, Ukrainian, etc. -- **Arabic**: Right-to-left scripts -- **Latin**: English, French, German, Spanish, etc. -- All other Unicode letter scripts - -Comment filtering uses Unicode-aware regex patterns (`\p{L}` for letters, `\p{N}` for digits) rather than ASCII-only ranges, ensuring comments in non-English languages are not silently dropped. - -!!! info "Multilingual sentiment scoring" - Sentiment scoring uses VADER's English lexicon as the base, extended with **93,000+ multilingual word entries** from the Chen-Skiena lexicon dataset (ACL 2014) covering **32 languages**. Non-ASCII words from the dataset are injected into VADER's lexicon at startup, enabling basic sentiment scoring for comments in Russian, Chinese, Japanese, Korean, Arabic, and 27 other languages. VADER's grammatical rules (negation, intensifiers) still operate on English syntax, so scoring accuracy for non-English comments is lower than for English — but significantly better than no coverage. - -### Sentiment Classification - -Filtered comments are classified as positive, negative, or neutral using **VADER** (Valence Aware Dictionary and sEntiment Reasoner) via the [GoVader](https://github.com/jonreiter/govader) library, enhanced with **software engineering domain adjustments**. - -#### VADER Base Scoring - -VADER is a lexicon and rule-based sentiment analyzer designed for social media and short text. It handles negations, intensifiers, and punctuation. The compound score (-1 to 1) is mapped to our [0, 1] range. - -#### Multilingual Lexicon Extension - -At startup, the analyzer injects ~93,000 multilingual word entries from the [Chen-Skiena lexicon dataset](https://aclanthology.org/P14-2063/) (ACL 2014) into VADER's lexicon. This covers 32 languages: - -| Language Family | Languages | -|---|---| -| **Slavic** | Russian, Ukrainian, Polish, Czech, Slovak, Croatian, Bulgarian | -| **CJK** | Chinese, Japanese, Korean | -| **Romance** | Spanish, French, Portuguese, Italian, Romanian | -| **Germanic** | German, Dutch, Swedish, Danish, Norwegian, Finnish | -| **Other** | Arabic, Hebrew, Hindi, Thai, Turkish, Greek, Hungarian, Indonesian, Malay, Vietnamese, Persian | - -Only non-ASCII words are injected to avoid overriding VADER's curated English entries. Words receive binary valence: +1.5 (positive) or -1.5 (negative), which is the mid-range of VADER's scale. - -To regenerate lexicons from updated source data: - -```bash -go run tools/lexgen/lexgen.go -pos pos_words.txt -neg neg_words.txt \ - -o internal/analyzers/sentiment/lexicons/lexicon_data.gen.go -``` - -#### SE-Domain Adjustments - -VADER frequently misclassifies technical terms that sound negative in natural language but are neutral in software engineering. The analyzer applies domain-specific adjustments: - -**Neutralized terms** (pushed toward neutral): -`kill`, `abort`, `fatal`, `terminate`, `dead`, `destroy`, `panic`, `deprecated`, `obsolete`, `execute`, `exploit`, `conflict`, `revert`, `reject`, `critical` - -**Genuinely negative in SE** (pushed toward negative): -`hack`, `hacky`, `kludge`, `workaround`, `spaghetti`, `nightmare`, `technical debt` - -#### Comment Length Weighting - -Longer comments carry more weight in the sentiment score, as they tend to contain more meaningful sentiment signal. Weight is capped at 3x the average to prevent a single long comment from dominating. - -#### Trend Analysis - -Sentiment trend is computed using **linear regression** (least squares) rather than a simple first-to-last comparison. This makes the trend robust to outliers and intermediate noise. - ---- - -## Configuration Options +## Configuration options | Option | Type | Default | Description | |---|---|---|---| @@ -131,11 +29,13 @@ history: !!! tip "Tuning the gap" A gap of `0.5` is conservative -- only strongly positive or negative comments are flagged. Lower it to `0.3` for more sensitivity, or raise to `0.7` to capture only the most extreme sentiments. +The analyzer requires UAST support to extract comments and is automatically enabled when the UAST pipeline is available. + --- -## Output Formats +## Output formats -### JSON Output +### JSON output ```json { @@ -187,7 +87,7 @@ history: } ``` -### Plot Output +### Plot output The HTML plot includes: @@ -196,7 +96,7 @@ The HTML plot includes: The plot uses semantic colors (green = positive, yellow = neutral, red = negative) and includes interpretive hints. -### Terminal Output +### Terminal output The terminal renderer provides a colored, Unicode-rich summary: @@ -232,56 +132,7 @@ The terminal renderer provides a colored, Unicode-rich summary: --- -## Use Cases - -- **Team morale tracking**: Monitor sentiment trends over time. A sustained drop in sentiment may correlate with deadline pressure, technical debt accumulation, or team issues. -- **Code quality signals**: Negative sentiment spikes often correspond to periods of rushed development, workarounds, or "hack" implementations. -- **Post-mortem analysis**: After incidents, examine whether comment sentiment degraded in the lead-up to the problem. -- **Documentation quality**: Projects with predominantly neutral or positive comments tend to have better documentation culture. -- **Technical debt detection**: Comments containing terms like "hack", "workaround", "kludge" are flagged as genuinely negative in SE context. - ---- - -## Limitations - -- **Non-English scoring accuracy**: While 32 languages have lexicon coverage via the Chen-Skiena dataset, VADER's grammatical rules (negation handling, intensifiers, punctuation effects) are English-specific. Non-English comments get word-level sentiment from the lexicon but miss syntactic nuances. -- **UAST dependency**: Requires UAST parsing support for the target language. Files in unsupported languages are skipped. -- **Sarcasm**: Sarcasm, irony, and context-dependent meaning can mislead the classifier. Comments like "great, another production outage" may be scored as positive. -- **Comment extraction**: Only comments that appear in the UAST are analyzed. Preprocessor directives, build file comments, and non-code files are excluded. -- **CPU intensive**: The sentiment analyzer performs UAST parsing for every modified file in every commit. For large repositories, this is significantly slower than non-UAST analyzers. It benefits from parallel execution via the framework's worker pool. -- **Minimum length filter**: The default minimum of 20 characters filters out many short but potentially meaningful comments (e.g., `// FIXME`, `// HACK`). Lower `MinLength` to capture these, at the cost of more noise. - ---- - -## SE-Domain Lexicon - -The following terms have special handling in the sentiment scorer: - -### Technical terms neutralized (not negative) - -These terms are common in SE but trigger VADER's negative scoring: - -| Term | Context | -|---|---| -| `kill` | Process management | -| `abort` | Transaction/operation cancellation | -| `fatal` | Error severity levels | -| `terminate` | Process/thread lifecycle | -| `panic` | Error handling (Go, Rust) | -| `deprecated` | API lifecycle | -| `execute` | Command/query execution | -| `conflict` | Merge conflicts | -| `critical` | Severity levels | - -### Terms with genuine negative sentiment - -These terms indicate real frustration or code quality issues: +## See also -| Term | Indication | -|---|---| -| `hack` / `hacky` | Quick-and-dirty solutions | -| `kludge` | Inelegant fixes | -| `workaround` | Avoiding root cause | -| `spaghetti` | Poor code structure | -| `nightmare` | Maintenance difficulty | -| `technical debt` | Accumulated shortcuts | +- [Understanding comment sentiment analysis](../explanation/sentiment.md) — the mental model, VADER scoring, multilingual lexicon, SE-domain lexicon, and limitations. +- [Quick start](../getting-started/quickstart.md) — run history analysis. diff --git a/site/analyzers/shotness.md b/site/analyzers/shotness.md index ab55f416..ad1edd52 100644 --- a/site/analyzers/shotness.md +++ b/site/analyzers/shotness.md @@ -1,93 +1,64 @@ -# Shotness Analyzer +# Shotness analyzer reference -The shotness analyzer measures **structural hotness** -- the change frequency of individual code entities (functions, methods, classes) across Git history. Unlike the couples analyzer which operates at file granularity, shotness operates at the UAST node level, providing fine-grained co-change analysis. +The shotness analyzer measures **structural hotness** -- the change frequency of +individual code entities (functions, methods, classes) across Git history, at +the UAST node level. ---- - -## Quick Start - -```bash -codefang run -a history/shotness . -``` - -With custom node selection: - -```bash -codefang run -a history/shotness \ - --shotness-dsl-struct 'filter(.roles has "Function")' \ - --shotness-dsl-name '.props.name' \ - . -``` - -!!! note "Requires UAST" - The shotness analyzer needs UAST support to identify code structures. It is automatically enabled when the UAST pipeline is available. +For the conceptual model — what node-level hotness and coupling mean and how +coupling strength is computed — see +[Understanding structural hotness](../explanation/shotness.md). To run it, see +the [Quick start](../getting-started/quickstart.md). --- -## What It Measures - -### Node Change Frequency - -For each code entity matched by the DSL query (functions by default), the analyzer counts how many commits modified lines within that entity's span. Entities that change frequently are "hot" -- they are likely volatile, complex, or central to the system. - -### Node Co-Change Coupling - -When two code entities are modified in the same commit, their coupling counter is incremented. This produces a fine-grained coupling matrix at the function level, which is more precise than file-level coupling from the couples analyzer. - -### Coupling Strength +## Configuration options -Coupling strength is normalized to a 0-1 scale using the formula: +| Option | Type | Default | Description | +|---|---|---|---| +| `Shotness.DSLStruct` | `string` | `filter(.roles has "Function")` | UAST DSL query to select which code structures to track. | +| `Shotness.DSLName` | `string` | `.props.name` | UAST DSL expression to extract the name from each matched node. | +```yaml +# .codefang.yml +history: + shotness: + dsl_struct: 'filter(.roles has "Function")' + dsl_name: '.props.name' ``` -strength(A, B) = co_changes(A, B) / max(co_changes(A, B), changes(A), changes(B)) -``` - -This ensures the result is always in [0, 1] and provides a meaningful confidence metric. A strength of 1.0 means functions always change together; 0.5 means they co-change half the time relative to the most active function. -### Risk Classification +The `--shotness-dsl-struct` and `--shotness-dsl-name` CLI flags set these options for a single run. -Nodes are classified into risk levels based on absolute change counts: +### Custom DSL examples -| Risk Level | Threshold | Meaning | -|---|---|---| -| **HIGH** | ≥ 20 changes | Requires immediate attention and robust test coverage | -| **MEDIUM** | ≥ 10 changes | Should be monitored and potentially refactored | -| **LOW** | < 10 changes | Normal change frequency | - -### How It Works - -For each commit: - -1. Parse the before and after versions of each changed file into UAST -2. Apply the `dsl_struct` query to select target nodes (e.g., functions) -3. Apply the `dsl_name` query to extract the name of each node -4. Map diff hunks to nodes using line-range overlaps -5. Emit a per-commit TC (Transient Commit result) with touched node deltas and coupling pairs +=== "Track classes instead of functions" -After all commits are processed, the Aggregator accumulates TCs into a final report with sorted nodes and a sparse co-change matrix. + ```yaml + dsl_struct: 'filter(.roles has "Class")' + dsl_name: '.props.name' + ``` -### Architecture +=== "Track both functions and methods" -The shotness analyzer follows the **TC/Aggregator** pattern: + ```yaml + dsl_struct: 'filter(.roles has "Function" or .roles has "Method")' + dsl_name: '.props.name' + ``` -- **Consume phase**: Per-commit processing builds working state (`nodes`, `files` maps for deletion/rename tracking) and emits a `TC{Data: *CommitData}` with node touch deltas and coupling pairs. -- **Aggregation phase**: The `Aggregator` accumulates node counts and coupling matrices from the TC stream. It supports disk-backed spilling via `SpillStore` for memory-bounded operation. -- **Serialization phase**: `SerializeTICKs()` converts aggregated tick data into the `Nodes`/`Counters` report consumed by `ComputeAllMetrics()` and plot generation. +=== "Track interfaces" -The `nodes` map remains in the analyzer as working state because `handleDeletion`, `handleInsertion`, `handleModification`, and `applyRename` read and mutate it during `Consume()`. The aggregator maintains its own separate accumulation of counts and couplings. + ```yaml + dsl_struct: 'filter(.roles has "Interface")' + dsl_name: '.props.name' + ``` --- -## Output Formats +## Output formats The shotness analyzer supports four output formats: JSON, YAML, text, and plot. === "Text" - ```bash - codefang run -a history/shotness -f text . - ``` - Terminal output with color-coded sections: ``` @@ -122,10 +93,6 @@ The shotness analyzer supports four output formats: JSON, YAML, text, and plot. === "JSON" - ```bash - codefang run -a history/shotness -f json . - ``` - ```json { "node_hotness": [ @@ -170,10 +137,6 @@ The shotness analyzer supports four output formats: JSON, YAML, text, and plot. === "YAML" - ```bash - codefang run -a history/shotness -f yaml . - ``` - ```yaml node_hotness: - name: processFile @@ -206,10 +169,6 @@ The shotness analyzer supports four output formats: JSON, YAML, text, and plot. === "Plot" - ```bash - codefang run -a history/shotness -f plot -o shotness.html . - ``` - Generates an interactive HTML dashboard with three visualizations: 1. **Code Hotness TreeMap**: Hierarchical file → function view sized by change frequency @@ -218,49 +177,9 @@ The shotness analyzer supports four output formats: JSON, YAML, text, and plot. --- -## Configuration Options - -| Option | Type | Default | Description | -|---|---|---|---| -| `Shotness.DSLStruct` | `string` | `filter(.roles has "Function")` | UAST DSL query to select which code structures to track. | -| `Shotness.DSLName` | `string` | `.props.name` | UAST DSL expression to extract the name from each matched node. | - -```yaml -# .codefang.yml -history: - shotness: - dsl_struct: 'filter(.roles has "Function")' - dsl_name: '.props.name' -``` - -### Custom DSL Examples - -=== "Track classes instead of functions" - - ```yaml - dsl_struct: 'filter(.roles has "Class")' - dsl_name: '.props.name' - ``` - -=== "Track both functions and methods" - - ```yaml - dsl_struct: 'filter(.roles has "Function" or .roles has "Method")' - dsl_name: '.props.name' - ``` - -=== "Track interfaces" - - ```yaml - dsl_struct: 'filter(.roles has "Interface")' - dsl_name: '.props.name' - ``` - ---- - -## Metrics Reference +## Metrics reference -### Node Hotness +### Node hotness | Field | Type | Description | |---|---|---| @@ -271,7 +190,7 @@ history: | `coupled_nodes` | int | Number of other nodes that co-changed with this node | | `hotness_score` | float | Normalized score [0, 1] relative to the hottest node | -### Node Coupling +### Node coupling | Field | Type | Description | |---|---|---| @@ -293,40 +212,7 @@ history: --- -## Use Cases - -- **Function-level hotspot detection**: Find the most frequently changed functions in the codebase. These are the highest-risk points for bugs. -- **Fine-grained coupling analysis**: Discover which functions always change together. This reveals implicit dependencies that file-level coupling misses. -- **Refactoring prioritization**: Functions that are both hot (high change count) and coupled (always change with others) are the best refactoring candidates. -- **Architecture validation**: Functions from different packages that are highly coupled may indicate a leaking abstraction. -- **Test prioritization**: Focus testing resources on the hottest functions. - ---- - -## Interpreting Results - -### Reading the Coupling Strength - -| Strength | Interpretation | -|---|---| -| 0.8 - 1.0 | Very tight coupling. Functions almost always change together. Consider merging or extracting shared logic. | -| 0.5 - 0.8 | Moderate coupling. There is a significant shared dependency. Review if coupling is intentional. | -| 0.2 - 0.5 | Loose coupling. Occasional co-changes, likely due to shared APIs or data structures. | -| < 0.2 | Minimal coupling. Co-changes are incidental. | - -### Actionable Insights - -1. **High hotness + High coupling**: Core function that drives many changes. Candidate for splitting or stabilizing the interface. -2. **High hotness + Low coupling**: Frequently bugfixed isolated function. Needs better tests and potentially a redesign. -3. **Low hotness + High coupling**: Stable function that always changes with others. Check if coupling is necessary or indicates a design smell. - ---- - -## Limitations +## See also -- **UAST required**: Only languages with UAST parser support are analyzed. Files in unsupported languages are skipped entirely. -- **CPU intensive**: The analyzer performs UAST parsing on both the before and after versions of every changed file in every commit. This makes it one of the most expensive analyzers. It benefits from parallel execution. -- **Name collisions**: If two functions in different files have the same name, they are tracked as distinct nodes (the file path is part of the key). However, if a file is renamed, the analyzer updates all associated nodes. -- **Shallow extraction within a file**: When multiple structural nodes in the same file share the same extracted name (e.g., nested functions with identical names), only one is tracked. The last one encountered wins. Qualified paths (e.g., `OuterClass.innerMethod`) are not built. -- **DSL limitations**: The DSL query must match nodes that have position information (`Pos` field) in the UAST. Nodes without position data cannot be mapped to diff hunks. -- **Large functions**: A change anywhere within a function's line range counts as a change to that function. Very large functions (hundreds of lines) will have inflated change counts. +- [Understanding structural hotness](../explanation/shotness.md) — the mental model, algorithm, architecture, interpretation, and limitations. +- [Quick start](../getting-started/quickstart.md) — run history analysis. diff --git a/site/analyzers/typos.md b/site/analyzers/typos.md index f235d675..131853a3 100644 --- a/site/analyzers/typos.md +++ b/site/analyzers/typos.md @@ -1,62 +1,15 @@ -# Typos Analyzer +# Typos analyzer reference -The typos analyzer detects **typo-fix identifier pairs** from source code in commit diffs using Levenshtein distance. It builds a dataset of probable typos and their corrections by analyzing UAST identifier changes across Git history. +The typos analyzer detects **typo-fix identifier pairs** from source code in +commit diffs using Levenshtein distance. ---- - -## Quick Start - -```bash -codefang run -a history/typos . -``` - -With a custom distance threshold: - -```bash -codefang run -a history/typos --typos-max-distance 3 . -``` - -!!! note "Requires UAST" - The typos analyzer needs UAST support to extract identifiers from source code. It is automatically enabled when the UAST pipeline is available. - ---- - -## Architecture - -The typos analyzer follows the **TC/Aggregator pattern**: - -1. **Consume phase**: For each commit, `Consume()` computes diffs, identifies line pairs within Levenshtein distance, and extracts UAST identifier changes. Per-commit typos are returned as `TC{Data: []Typo}`. The analyzer retains no per-commit state; only the `lcontext` (Levenshtein context) is kept as working state. -2. **Aggregation phase**: A `typos.Aggregator` collects TCs into a `SliceSpillStore[Typo]`. `FlushTick()` deduplicates typos by `wrong|correct` key (keeping the first occurrence), returning a `TickData` with the unique set. -3. **Serialization phase**: `SerializeTICKs()` assembles all tick data into an `analyze.Report{"typos": allTypos}`, then delegates to `ComputeAllMetrics()` for JSON, YAML, binary, or HTML plot output. - -This separation enables streaming output, budget-aware memory spilling, and decoupled aggregation. +For the conceptual model — how detection works and what Levenshtein distance +measures — see [Understanding typo detection](../explanation/typos.md). To run +it, see the [Quick start](../getting-started/quickstart.md). --- -## What It Measures - -### Typo-Fix Pair Detection - -For each commit, the analyzer: - -1. Computes the diff between the old and new versions of each changed file -2. Identifies delete/insert hunk pairs of equal size (same number of lines) -3. Compares corresponding lines using **Levenshtein distance** -4. For line pairs within the distance threshold, extracts UAST identifiers from the old and new versions -5. If exactly one identifier changed between the two versions, it records a typo-fix pair - -### Levenshtein Distance - -The Levenshtein distance is the minimum number of single-character edits (insertions, deletions, substitutions) needed to transform one string into another. A small distance between the old and new line suggests a typo fix rather than a semantic change. - -!!! info "Example" - - `recieve` to `receive` -- distance 2 (transposed characters) - - `lenght` to `length` -- distance 2 (transposed characters) - - `calcualte` to `calculate` -- distance 2 (transposed characters) - ---- - -## Configuration Options +## Configuration options | Option | Type | Default | Description | |---|---|---|---| @@ -74,9 +27,11 @@ history: - **Distance 3-4** (default): Good balance. Catches transposition errors and short misspellings. - **Distance 5+**: Lower confidence. May produce false positives from intentional renames. +The analyzer requires UAST support to extract identifiers from source code. It is automatically enabled when the UAST pipeline is available. The `--typos-max-distance` CLI flag sets `MaximumAllowedDistance` for a single run. + --- -## Example Output +## Example output === "JSON" @@ -128,21 +83,7 @@ history: --- -## Use Cases - -- **Typo dataset building**: Build a corpus of real-world typos and their corrections from your project's history. This can train spell-checking tools or IDE plugins. -- **Code quality auditing**: Identify patterns of common misspellings in your codebase to add to a linting dictionary. -- **API consistency**: Detect identifier typos that may cause confusion (e.g., `getUserByNmae` vs `getUserByName`). -- **Automated fix suggestions**: Use the typo dataset to build automated correction rules for CI pipelines. -- **Research**: Academic research on developer typo patterns and their prevalence across different languages. - ---- - -## Limitations +## See also -- **UAST required**: Only languages with UAST parser support are analyzed. Identifiers in unsupported languages are not extracted. -- **Single-identifier changes only**: The analyzer only records a typo when exactly one identifier changes between the old and new lines. Multi-identifier changes are skipped to avoid false positives. -- **Equal-length hunks only**: Only delete/insert hunk pairs with the same number of lines are considered. A typo fix that also adds or removes lines will be missed. -- **False positives**: Intentional identifier renames with small Levenshtein distance (e.g., `idx` to `jdx`) will be reported as typos. -- **Deduplication**: Typo pairs are deduplicated by the `wrong|correct` key. The same typo fixed in multiple commits is reported only once. -- **CPU intensive**: Like all UAST-based analyzers, the typos analyzer parses both file versions for every changed file in every commit. +- [Understanding typo detection](../explanation/typos.md) — the mental model, Levenshtein distance, architecture, and limitations. +- [Quick start](../getting-started/quickstart.md) — run history analysis. diff --git a/site/architecture/overview.md b/site/architecture/overview.md index 1edd9e87..634a087c 100644 --- a/site/architecture/overview.md +++ b/site/architecture/overview.md @@ -3,13 +3,13 @@ title: Architecture Overview description: System architecture, package structure, and data flow for the Codefang analysis platform. --- -# Architecture Overview +# Architecture overview Codefang follows the **Unix philosophy**: small, focused tools joined by pipes. The project ships two binaries that can be used independently or composed together in pipelines, CI systems, and AI agent workflows. -## Two Binaries +## Two binaries | Binary | Purpose | Entry point | |--------|---------|-------------| @@ -35,16 +35,16 @@ uast parse main.go | codefang run -a static/* --format json --- -## Package Structure +## Package structure -### CLI Layer +### CLI layer | Package | Description | |---------|-------------| | `cmd/codefang/` | CLI entry point for the analyzer. Cobra commands for `run`, `mcp`, etc. | | `cmd/uast/` | CLI entry point for the UAST parser. Commands: `parse`, `query`, `diff`, `explore`, `server`. | -### Core Libraries +### Core libraries | Package | Description | |---------|-------------| @@ -70,7 +70,7 @@ uast parse main.go | codefang run -a static/* --format json ### Analyzers (`internal/analyzers/`) -#### Shared Components (`plumbing/`) +#### Shared components (`plumbing/`) The plumbing package provides the shared pipeline components that all history analyzers depend on. These run as "core" analyzers in the pipeline before any @@ -87,7 +87,7 @@ leaf analyzers consume their output. | `LinesStatsCalculator` | `line_stats.go` | Computes per-commit line addition/deletion stats | | `UASTChangesAnalyzer` | `uast.go` | Parses UAST for changed files | -#### Static Analyzers (`analyze/`) +#### Static analyzers (`analyze/`) | Analyzer | ID | Description | |----------|----|-------------| @@ -101,7 +101,7 @@ The `analyze/` package also contains the static analysis service, analyzer registry, factory, output formatting (JSON, text, compact, YAML, plot, binary, timeseries), and cross-format conversion logic. -#### History Analyzers +#### History analyzers | Analyzer | ID | Description | |----------|----|-------------| @@ -118,7 +118,7 @@ timeseries), and cross-format conversion logic. --- -## Data Flow +## Data flow The following diagram shows how data flows through the system during a combined static + history analysis run. @@ -193,7 +193,7 @@ flowchart TD --- -## Two Analysis Modes +## Two analysis modes Codefang operates in two distinct modes that can run independently or be combined in a single `codefang run` invocation. @@ -230,7 +230,7 @@ flowchart LR MERGE --> OUT["JSON / Text / YAML / Plot / TimeSeries"] ``` -### Static Mode +### Static mode 1. Reads source files from the filesystem. 2. Parses each file into a UAST using Tree-sitter with DSL-based mappings. @@ -239,7 +239,7 @@ flowchart LR Static analysis is fast, parallelized across files, and requires no Git history. -### History Mode +### History mode 1. Opens the Git repository via libgit2 (supports both normal and bare repos). 2. Loads the commit history (optionally filtered by `--limit`, `--since`, `--first-parent`). @@ -249,7 +249,7 @@ Static analysis is fast, parallelized across files, and requires no Git history. 6. For large repositories, the **streaming pipeline** splits commits into memory-bounded chunks with hibernate/boot cycles and optional double-buffered pipelining. The `BaseHistoryAnalyzer` manages state serialization transparently. 7. **Checkpointing** after each chunk enables crash recovery. -### Combined Mode +### Combined mode When both static and history analyzers are selected, Codefang runs both phases sequentially, encodes each phase to an internal binary format, then decodes and @@ -257,7 +257,7 @@ merges the results into a single output document in the requested format. --- -## Pipeline Architecture +## Pipeline architecture The history analysis pipeline is built around a **Runner** that coordinates the full lifecycle: @@ -280,7 +280,7 @@ The **Runner** supports two execution strategies: --- -## Configuration Layers +## Configuration layers Configuration follows a clear priority chain: diff --git a/site/architecture/streaming-pipeline.md b/site/architecture/streaming-pipeline.md index 7ac02553..1080e32f 100644 --- a/site/architecture/streaming-pipeline.md +++ b/site/architecture/streaming-pipeline.md @@ -3,7 +3,7 @@ title: Streaming Pipeline description: Chunk-based streaming execution for analyzing large repositories with bounded memory, including double-buffered pipelining and checkpoint recovery. --- -# Streaming Pipeline +# Streaming pipeline Large Git repositories (tens or hundreds of thousands of commits) can easily exhaust available memory if all analyzer state is accumulated in a single pass. @@ -13,7 +13,7 @@ memory-bounded **chunks**, processing each chunk independently with --- -## The Problem +## The problem History analyzers accumulate state as they process commits. For example: @@ -26,7 +26,7 @@ of memory. Many CI environments and containers are limited to 2-8 GiB. --- -## The Solution: Chunk-Based Processing +## The solution: chunk-based processing The streaming pipeline: @@ -77,7 +77,7 @@ flowchart LR --- -## The Planner +## The planner The `streaming.Planner` calculates optimal chunk boundaries. It lives in `internal/streaming/planner.go`. @@ -98,7 +98,7 @@ The `streaming.Planner` calculates optimal chunk boundaries. It lives in | `AggStatePercent` | 30% | Fraction of remaining budget for aggregator spill budget | | `ChunkMemPercent` | 10% | Fraction of remaining budget for in-flight data | -### Budget Decomposition: P + W + A + S +### Budget decomposition: P + W + A + S The scheduler decomposes the memory budget into four explicit regions: @@ -116,7 +116,7 @@ The `ComputeSchedule()` function in `internal/streaming/planner.go` performs thi decomposition and returns a `Schedule` containing chunk boundaries, chunk size, buffering factor, and the aggregator spill budget. -### Chunk Size Calculation +### Chunk size calculation The planner determines chunk size from the **working state** portion of the budget: @@ -130,7 +130,7 @@ Where: - **`workState`** is 60% of the remaining budget after pipeline overhead and slack. - **`working_state_per_commit`** is the sum of all selected leaf analyzers' declared `WorkingStateSize()` values. Each leaf analyzer declares `WorkingStateSize()` (analyzer-internal data structures) and `AvgTCSize()` (per-commit TC payload). Only `WorkingStateSize()` drives chunk sizing; `AvgTCSize()` is used separately for aggregator budget estimation. -### Memory Sizing Methods +### Memory sizing methods Each `HistoryAnalyzer` declares two per-commit memory estimates: @@ -163,7 +163,7 @@ chunks = ceil(10,000 / 931) = 11 chunks --- -## Buffered Chunk Pipelining +## Buffered chunk pipelining When the memory budget is sufficient and multiple chunks are needed, the pipeline enables **buffered pipelining** to overlap the pipeline stage @@ -171,7 +171,7 @@ of upcoming chunks with the analyzer consumption stage of the current chunk. The scheduler determines the **buffering factor** (1, 2, or 3) based on the memory budget. -### The Insight +### The insight Processing a chunk has two phases: @@ -182,7 +182,7 @@ Processing a chunk has two phases: These two phases use different resources and can overlap. -### Buffering Factor Selection +### Buffering factor selection The `ComputeSchedule()` function iterates buffering factors from `MaxBuffering` (default 3) down to 1, selecting the highest factor where `ChunkSize >= @@ -206,7 +206,7 @@ for bf = maxBuffering; bf >= 1; bf-- { | 512 MiB | 1 | Single buffering (budget too tight) | | Unlimited (0) | 3 | Maximum parallelism | -### How Double-Buffering Works +### How double-buffering works ``` Time ───────────────────────────────────────────> @@ -226,7 +226,7 @@ Chunk 3: |==== Pipeline ====|==== Consume available -- analyzers can consume it immediately without waiting for I/O. 4. The pattern repeats for subsequent chunks. -### How Triple-Buffering Works +### How triple-buffering works With `BufferingFactor >= 3`, the scheduler produces smaller chunks (workState divided by 3), and the prefetch loop naturally overlaps more pipeline phases. @@ -234,7 +234,7 @@ The existing double-buffer loop handles triple-buffering semantics: at each iteration it prefetches the next chunk, so with more (smaller) chunks, the overlap covers a larger fraction of total processing time. -### Memory Budget Split +### Memory budget split The scheduler handles the memory budget split through the buffering factor iteration. With `BufferingFactor = N`, the working state region is divided @@ -248,7 +248,7 @@ chunkSize = workState / (N * effectiveGrowth) This results in smaller chunk sizes (more chunks), but the pipeline overlap compensates by eliminating I/O wait time between chunks. -### Activation Conditions +### Activation conditions The scheduler automatically selects the buffering factor. The pipeline uses double-buffered processing when `BufferingFactor >= 2` and falls back to @@ -257,13 +257,13 @@ always uses single-buffering since it cannot prefetch. --- -## Three-Metric Adaptive Feedback +## Three-metric adaptive feedback After each chunk, the `AdaptivePlanner` examines three independent metrics and re-plans remaining chunks if any metric diverges from its prediction by more than 25%. -### Tracked Metrics +### Tracked metrics | Metric | Source | Purpose | |--------|--------|---------| @@ -275,7 +275,7 @@ Each metric is tracked by its own exponential moving average (EMA) with `alpha = 0.3` (~3-chunk half-life). The `ReplanObservation` struct carries all three per-commit observations to `Replan()`. -### Replan Logic +### Replan logic 1. Update all three EMAs with the chunk's observations (clamped to 1 KiB floor). 2. Compute the predicted effective growth rate: `declared * 1.5` (safety margin). @@ -294,7 +294,7 @@ all three per-commit observations to `Replan()`. --- -## Hibernate / Boot Cycles +## Hibernate / boot cycles Between chunks, the pipeline calls `Hibernate()` on all hibernatable analyzers, then `Boot()` to restore them for the next chunk. @@ -322,7 +322,7 @@ checkpoint to disk. If the process is interrupted (OOM kill, pod eviction, timeout), the next run with `--resume` automatically restarts from the last completed chunk. -### Checkpoint Contents +### Checkpoint contents | Field | Description | |-------|-------------| @@ -341,7 +341,7 @@ On resume, aggregators are recreated and pointed at their saved spill directories so that TCs accumulated before the interruption are preserved. This ensures resumed runs produce identical output to uninterrupted runs. -### Checkpointable Interface +### Checkpointable interface ```go type Checkpointable interface { @@ -355,7 +355,7 @@ type Checkpointable interface { implement `Checkpointable`. If any analyzer lacks support, checkpointing is disabled with a warning. -### CLI Flags +### CLI flags | Flag | Default | Description | |------|---------|-------------| @@ -364,7 +364,7 @@ type Checkpointable interface { | `--checkpoint-dir` | `~/.codefang/checkpoints` | Checkpoint storage directory | | `--clear-checkpoint` | `false` | Clear existing checkpoint before run | -### Crash Recovery Flow +### Crash recovery flow ```mermaid flowchart TD @@ -384,7 +384,7 @@ flowchart TD --- -## Full Pipeline Timeline +## Full pipeline timeline The following diagram shows the complete lifecycle of a streaming analysis run with buffered pipelining (double-buffer shown) and checkpointing: @@ -422,9 +422,9 @@ Hibernate: [hib][boot] [hib][boot] --- -## Configuration Recommendations +## Configuration recommendations -### Memory Budget by Repository Size +### Memory budget by repository size | Repository Size | Recommended `--memory-budget` | Expected Chunks | |-----------------|-------------------------------|-----------------| @@ -433,7 +433,7 @@ Hibernate: [hib][boot] [hib][boot] | 10k -- 100k commits | `4GiB` -- `8GiB` | 10 -- 100+ | | 100k+ commits | `8GiB` | Many, checkpointing essential | -### Tuning Tips +### Tuning tips !!! tip "Let the planner decide" In most cases, setting `--memory-budget` is sufficient. The planner @@ -452,7 +452,7 @@ Hibernate: [hib][boot] [hib][boot] scheduler automatically selects double or triple buffering when the budget allows. -### Reference Benchmarks +### Reference benchmarks Measured on the kubernetes repository (56K first-parent commits, burndown analyzer, `--memory-budget 4GB`): diff --git a/site/architecture/uast.md b/site/architecture/uast.md index be98f8b3..a95640bc 100644 --- a/site/architecture/uast.md +++ b/site/architecture/uast.md @@ -3,7 +3,7 @@ title: UAST System description: Deep dive into the Universal Abstract Syntax Tree parser, Tree-sitter integration, DSL mapping engine, and CLI tools. --- -# UAST System +# UAST system The **Universal Abstract Syntax Tree** (UAST) is Codefang's language-agnostic code representation. It normalizes the wildly different ASTs produced by @@ -46,7 +46,7 @@ Example UAST node for a Go function: --- -## Tree-sitter Under the Hood +## Tree-sitter under the hood Codefang uses [Tree-sitter](https://tree-sitter.github.io/) as the parsing backend. Tree-sitter provides: @@ -67,14 +67,14 @@ flowchart LR DSL --> UAST["Universal AST
normalized nodes"] ``` -### Language Detection +### Language detection File language is detected using the [go-enry](https://github.com/go-enry/go-enry) library, which uses filename, extension, and content heuristics (the same algorithm as GitHub's Linguist). You can override detection with the `--language` / `-l` flag. -### Performance Optimizations +### Performance optimizations The UAST parser includes several performance optimizations for high-throughput analysis: @@ -90,13 +90,13 @@ analysis: --- -## DSL Mapping Engine +## DSL mapping engine The DSL (Domain-Specific Language) is how Tree-sitter's language-specific concrete syntax tree nodes are mapped to universal UAST node types. Each language has a `.uast` mapping file that declares the transformation rules. -### DSL Syntax Overview +### DSL syntax overview A mapping file consists of: @@ -117,7 +117,7 @@ function_declaration <- (function_declaration ) ``` -### Rule Anatomy +### Rule anatomy Each rule has four parts: @@ -131,7 +131,7 @@ Each rule has four parts: | `tree-sitter-pattern` | A Tree-sitter query pattern with named captures (`@name`, `@body`) | | `uast-spec` | The UAST node specification: type, token, roles, props, children | -### UAST Spec Fields +### UAST spec fields | Field | Description | Example | |-------|-------------|---------| @@ -141,7 +141,7 @@ Each rule has four parts: | `props` | Key-value properties | `name: "@name", visibility: "public"` | | `children` | Child capture reference | `"@body"` | -### Token Sources +### Token sources | Source | Description | |--------|-------------| @@ -189,12 +189,12 @@ them. Inheritance is resolved recursively. --- -## Role System +## Role system UAST roles classify what a node **does** semantically, independent of language syntax. A single node can have multiple roles. -### Common Roles +### Common roles | Role | Description | Examples | |------|-------------|---------| @@ -220,11 +220,11 @@ syntax. A single node can have multiple roles. --- -## Writing Custom Mappings +## Writing custom mappings To add or customize language mappings: -### Step 1: Create a Mapping File +### Step 1: create a mapping file Create a `.uast` file for your language: @@ -276,7 +276,7 @@ line_comment <- (line_comment) => uast( ) ``` -### Step 2: Test with the UAST CLI +### Step 2: test with the UAST CLI ```bash # Parse a file with your custom mapping @@ -289,13 +289,13 @@ uast explore main.rs uast query -e 'filter(.type == "Function")' main.rs ``` -### Step 3: Iterate +### Step 3: iterate Use `uast explore` to inspect the Tree-sitter CST and discover which node types and field names are available for your target language. Then write mapping rules to convert them to UAST types. -### Pre-compiled Mappings +### Pre-compiled mappings For production performance, language mappings are pre-compiled into the binary. The DSL is parsed once at startup and the resulting `PatternMatcher` caches @@ -307,7 +307,7 @@ allocation. --- -## UAST CLI Commands +## UAST CLI commands The `uast` binary provides five commands for working with Universal ASTs. @@ -391,7 +391,7 @@ language mappings, with full OpenTelemetry instrumentation. --- -## Architecture Diagram +## Architecture diagram ```mermaid flowchart TB @@ -444,7 +444,7 @@ flowchart TB --- -## Integration with Analyzers +## Integration with analyzers The UAST system is used by both static and history analyzers: diff --git a/site/contributing/changelog.md b/site/contributing/changelog.md index 7b8c76fa..35067000 100644 --- a/site/contributing/changelog.md +++ b/site/contributing/changelog.md @@ -1,97 +1,8 @@ # Changelog -All notable changes to the Codefang project are documented in this file. +The canonical changelog for Codefang lives in the repository root and follows +[Keep a Changelog 1.1.0](https://keepachangelog.com/en/1.1.0/) and +[Semantic Versioning](https://semver.org/spec/v2.0.0.html). -The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), -and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - -## [Unreleased] - -### Added - -- **Temporal Anomaly Detection** analyzer (`history/anomaly`) using Z-score - statistical analysis over sliding windows for detecting sudden quality - degradation in commit history. -- **TimeSeries output format** (`--format timeseries`) that merges all - analyzer outputs into a single chronologically-ordered JSON array. -- **Plot output format** (`--format plot`) generating interactive HTML charts - via go-echarts for visual analysis. -- **MCP Server** (`codefang mcp`) exposing analysis capabilities as tools - for AI agents via the Model Context Protocol (stdio transport, JSON-RPC 2.0). - Three tools: `codefang_analyze`, `uast_parse`, `codefang_history`. -- **Docker support** with multi-stage Dockerfile and `debian:bookworm-slim` - runtime image. Non-root `codefang` user by default. -- **GitHub Actions** integration (`action.yml`) for automated code quality - checks in CI pipelines with configurable analyzers, formats, and quality gates. -- **OpenTelemetry observability** with distributed tracing, RED metrics - (Rate, Errors, Duration), and structured logging with trace context injection. - Supports Jaeger, Prometheus, and OTLP collectors. -- **Streaming pipeline** for bounded-memory processing of large repositories - via chunk-based processing with hibernate/boot cycles. -- **Double-buffered chunk pipelining** overlapping pipeline prefetch with - analyzer consumption for improved throughput on multi-chunk workloads. -- **Checkpointing** with automatic save after each processed chunk and - crash recovery via `--resume` flag. Checkpoint format v2 preserves - aggregator spill state so resumed runs produce identical output. -- **Configuration system** (`.codefang.yaml`) with file, environment variable, - and CLI flag support. Merge priority: CLI > env > file > defaults. -- **Large-scale scanning** support for fleet analysis of thousands of - repositories with bare repo support, GNU Parallel and Kubernetes orchestration - patterns, and DWH loading guides (Athena, Snowflake, Spark). -- **Deep context propagation** eliminating all `context.Background()` calls - in production hot paths for end-to-end tracing. -- **Attribute filter** span processor enforcing allow-list of attribute key - prefixes to prevent PII leakage to collectors. -- **Health check endpoints** (`/healthz`, `/readyz`, `/metrics`) for server - mode deployments. -- **Incremental scanning** with `--since` flag supporting Go durations, - date strings, and RFC3339 timestamps. -- **Memory budget auto-tuning** via `--memory-budget` flag with automatic - chunk size calculation. -- **Watchdog** stall detection with configurable `worker_timeout` for - identifying hung workers. - -### Changed - -- **Legacy cleanup** (specs/legacy-incomplete-cleanup): Removed all backward-compat fallbacks - (quality, anomaly, sentiment, devs ParseReportData); formalized shotness shallow extraction; - integrated VADER (GoVader) for real sentiment analysis; fixed stale comments and dead code. -- **Complete rewrite** from the original `src-d/hercules` project. Modern - Go 1.24+ codebase with idiomatic patterns, clean architecture, and - comprehensive test coverage. -- **Split architecture** into two binaries: `uast` (Universal AST parser - using Tree-sitter for 60+ languages) and `codefang` (analysis engine - for static and history analysis). Unix philosophy: small tools joined by pipes. -- **Tree-sitter based UAST** replacing the original Babelfish/bblfsh parser - with a faster, more reliable, and locally-compiled alternative supporting - 60+ programming languages. -- **DSL-based UAST mappings** with a custom domain-specific language for - transforming Tree-sitter ASTs into standardized UAST nodes. -- **Vendored libgit2** (`third_party/libgit2`) compiled as a static library - for reproducible builds without external dependencies. -- **Structured logging** with `log/slog` replacing all `log.Printf` and - `fmt.Printf` calls. Instance loggers enforced by `sloglint`. - -### Analyzers - -#### Static Analyzers (UAST-based) -- `static/complexity` - Cyclomatic, cognitive complexity, and nesting depth -- `static/cohesion` - LCOM4 metrics for class cohesion analysis -- `static/halstead` - Halstead complexity metrics (volume, difficulty, effort) -- `static/comments` - Documentation coverage, placement quality -- `static/imports` - Import/dependency analysis from source code - -#### History Analyzers (Git-based) -- `history/burndown` - Code survival over time with line ownership tracking -- `history/devs` - Developer activity, language breakdown, bus factor risk -- `history/couples` - File coupling and co-change pattern detection -- `history/file-history` - Per-file lifecycle with rename tracking -- `history/sentiment` - Comment sentiment analysis over commit history -- `history/shotness` - Structural hotspots (function-level change frequency) -- `history/typos` - Typo detection dataset builder via Levenshtein distance -- `history/imports` - Import/dependency evolution over time -- `history/anomaly` - Z-score temporal anomaly detection - ---- - -*For detailed documentation, visit the [Codefang Documentation](../index.md).* +Read it on GitHub: +[CHANGELOG.md](https://github.com/Sumatoshi-tech/codefang/blob/main/CHANGELOG.md). diff --git a/site/contributing/code-of-conduct.md b/site/contributing/code-of-conduct.md index ece8d64b..f29b401c 100644 --- a/site/contributing/code-of-conduct.md +++ b/site/contributing/code-of-conduct.md @@ -1,6 +1,6 @@ -# Code of Conduct +# Code of conduct -## Our Pledge +## Our pledge We as members, contributors, and leaders pledge to make participation in our community a harassment-free experience for everyone, regardless of age, body @@ -12,7 +12,7 @@ identity and orientation. We pledge to act and interact in ways that contribute to an open, welcoming, diverse, inclusive, and healthy community. -## Our Standards +## Our standards Examples of behavior that contributes to a positive environment: @@ -30,7 +30,7 @@ Examples of unacceptable behavior: - Other conduct which could reasonably be considered inappropriate in a professional setting -## Enforcement Responsibilities +## Enforcement responsibilities Community leaders are responsible for clarifying and enforcing our standards of acceptable behavior and will take appropriate and fair corrective action in @@ -44,12 +44,15 @@ an individual is officially representing the community in public spaces. ## Enforcement Instances of abusive, harassing, or otherwise unacceptable behavior may be -reported to the community leaders responsible for enforcement at the -[GitHub Issues](https://github.com/Sumatoshi-tech/codefang/issues) page. +reported **privately** to the community leaders responsible for enforcement by +email at . Do not open a public GitHub issue, +discussion, or pull request to report a violation. -All complaints will be reviewed and investigated promptly and fairly. +All complaints will be reviewed and investigated promptly and fairly. All +community leaders are obligated to respect the privacy and security of the +reporter of any incident. -## Enforcement Guidelines +## Enforcement guidelines Community leaders will follow these guidelines in determining the consequences for any action they deem in violation of this Code of Conduct: @@ -70,7 +73,7 @@ actions. **Consequence**: A warning with consequences for continued behavior. No interaction with the people involved for a specified period of time. -### 3. Temporary Ban +### 3. Temporary ban **Community Impact**: A serious violation of community standards, including sustained inappropriate behavior. @@ -78,7 +81,7 @@ sustained inappropriate behavior. **Consequence**: A temporary ban from any sort of interaction or public communication with the community for a specified period of time. -### 4. Permanent Ban +### 4. Permanent ban **Community Impact**: Demonstrating a pattern of violation of community standards. diff --git a/site/contributing/index.md b/site/contributing/index.md index 02cbb991..0433b85c 100644 --- a/site/contributing/index.md +++ b/site/contributing/index.md @@ -14,18 +14,18 @@ development environment to submitting a pull request. --- -## Getting Started +## Getting started ### Prerequisites | Tool | Version | Purpose | |------|---------|---------| -| **Go** | 1.24+ | Primary language | +| **Go** | 1.26+ | Primary language | | **CMake** | 3.14+ | Building vendored libgit2 | | **Python 3** | 3.10+ | Benchmark scripts, code generation | | **Git** | 2.30+ | Version control | -### Fork, Clone, and Build +### Fork, clone, and build 1. **Fork** the repository on GitHub. @@ -54,7 +54,7 @@ development environment to submitting a pull request. --- -## Development Workflow +## Development workflow Codefang follows a test-driven, branch-based workflow. Here is the recommended sequence for every change: @@ -69,7 +69,7 @@ graph LR F --> G[Open pull request] ``` -### 1. Create a Feature Branch +### 1. Create a feature branch Always branch from `main`: @@ -79,7 +79,7 @@ git pull origin main git checkout -b feat/my-new-feature ``` -### 2. Write Tests First (TDD) +### 2. Write tests first (TDD) Write failing tests that describe the expected behavior **before** writing implementation code. Codefang uses table-driven tests extensively: @@ -106,19 +106,19 @@ func TestMyFeature(t *testing.T) { } ``` -### 3. Implement Changes +### 3. Implement changes Write the minimal code to make the tests pass. Keep functions focused and interfaces small. -### 4. Run Linting and Dead Code Analysis +### 4. Run linting and dead code analysis ```bash make lint make deadcode ``` -### 5. Run Tests (with Race Detector) +### 5. Run tests (with race detector) ```bash make test @@ -128,18 +128,18 @@ make test The test suite runs with the Go race detector enabled in CI. Ensure your code is free of data races before pushing. -### 6. Update Documentation +### 6. Update documentation If your change affects user-facing behavior, CLI flags, configuration options, or analyzer output, update the relevant documentation under `site/`. --- -## Code Standards +## Code standards -### Language and Style +### Language and style -- **Go 1.24+** -- use modern language features where they improve clarity. +- **Go 1.26+** -- use modern language features where they improve clarity. - Write **idiomatic Go** -- follow [Effective Go](https://go.dev/doc/effective_go) and the [Go Code Review Comments](https://github.com/golang/go/wiki/CodeReviewComments). - Run `make fmt` before committing. @@ -150,7 +150,7 @@ or analyzer output, update the relevant documentation under `site/`. - Name test cases descriptively (e.g., `"empty repository returns zero commits"`). - Place test helpers in the same package with a `_test.go` suffix. -### Context Propagation +### Context propagation - Pass `context.Context` as the first parameter through all public APIs. - **Never** use `context.Background()` in hot paths -- propagate the caller's @@ -167,7 +167,7 @@ func Analyze(repo *Repository) (Result, error) { } ``` -### Structured Logging +### Structured logging Use `log/slog` for all logging. Include relevant key-value pairs: @@ -179,7 +179,7 @@ slog.InfoContext(ctx, "analysis complete", ) ``` -### Functional Options +### Functional options Use the functional options pattern for constructors with optional configuration: @@ -199,14 +199,14 @@ func NewRunner(opts ...Option) *Runner { } ``` -### Interface Design +### Interface design - Keep interfaces **small** -- aim for a maximum of **5 methods**. - Define interfaces at the **consumer** site, not the producer. - Prefer standard library interfaces (`io.Reader`, `io.Writer`, `fmt.Stringer`) when they fit. -### Error Handling +### Error handling Wrap errors with context using `fmt.Errorf` and the `%w` verb: @@ -222,7 +222,7 @@ if err != nil { --- -## Commit Conventions +## Commit conventions Codefang uses [Conventional Commits](https://www.conventionalcommits.org/) to produce clean changelogs and enable automated versioning. @@ -261,9 +261,9 @@ produce clean changelogs and enable automated versioning. --- -## Pull Request Process +## Pull request process -### Before Opening a PR +### Before opening a PR - [ ] All tests pass locally (`make test`) - [ ] Linting is clean (`make lint`) @@ -271,7 +271,7 @@ produce clean changelogs and enable automated versioning. - [ ] Commit messages follow conventional commits - [ ] Documentation is updated if needed -### PR Description Template +### PR description template When opening a pull request, include: @@ -286,7 +286,7 @@ When opening a pull request, include: --- -## Make Targets Reference +## Make targets reference The project `Makefile` provides a comprehensive set of targets for development: @@ -324,7 +324,7 @@ The project `Makefile` provides a comprehensive set of targets for development: --- -## Reporting Bugs +## Reporting bugs Found a bug? Please [open a GitHub Issue](https://github.com/Sumatoshi-tech/codefang/issues/new) with the following information: @@ -343,7 +343,7 @@ with the following information: --- -## Feature Requests +## Feature requests Have an idea for a new analyzer, output format, or integration? We welcome feature requests. Please [open a GitHub Issue](https://github.com/Sumatoshi-tech/codefang/issues/new) @@ -357,7 +357,7 @@ and include: --- -## Thank You +## Thank you Every contribution, no matter how small, helps make Codefang a better tool. Thank you for being part of the community. diff --git a/site/examples/index.md b/site/examples/index.md index 4b1605e2..74b7d927 100644 --- a/site/examples/index.md +++ b/site/examples/index.md @@ -2,7 +2,7 @@ All examples below are generated from the [Kubernetes](https://github.com/kubernetes/kubernetes) repository — one of the largest open-source Go projects with 135k+ commits and thousands of contributors. -## History Analyzers +## History analyzers ### Burndown @@ -65,7 +65,7 @@ See the [Couples analyzer docs](../analyzers/couples.md) for configuration optio --- -### File History +### File history The file history chart shows the most frequently modified files across the repository's entire commit history. @@ -95,7 +95,7 @@ See the [File History analyzer docs](../analyzers/file-history.md) for configura --- -### Anomaly Detection +### Anomaly detection The anomaly chart detects unusual patterns in commit activity — spikes or drops that deviate significantly from the repository's normal rhythm. @@ -123,7 +123,7 @@ See the [Anomaly Detection analyzer docs](../analyzers/anomaly.md) for configura --- -### Imports (History) +### Imports (history) The imports history chart tracks how import dependencies evolve over the repository's commit history. @@ -296,7 +296,7 @@ codefang run -a history/quality --format plot /path/to/repo > quality.html --- -## Static Analyzers +## Static analyzers ### Complexity @@ -433,7 +433,7 @@ See the [Comments analyzer docs](../analyzers/comments.md) for configuration opt --- -### Imports (Static) +### Imports (static) The static imports charts show the most used import packages and their categorization across the codebase. diff --git a/site/explanation/anomaly.md b/site/explanation/anomaly.md new file mode 100644 index 00000000..f8f1d7d8 --- /dev/null +++ b/site/explanation/anomaly.md @@ -0,0 +1,93 @@ +# Understanding temporal anomaly detection + +This page explains the mental model behind the anomaly analyzer: how Z-score +analysis over a sliding window flags unusual activity, which metrics it tracks, +and how the aggregation pipeline is structured. For configuration keys and the +output schema, see the [Anomaly reference](../analyzers/anomaly.md). + +--- + +## What it measures + +The anomaly analyzer performs **temporal anomaly detection** using Z-score +analysis over a sliding window of per-tick commit metrics. It detects sudden +quality degradation -- unusual spikes in churn, file changes, or other metrics +that deviate significantly from the historical baseline. + +### Z-score analysis + +For each time tick, the analyzer computes Z-scores for six metrics using a trailing sliding window: + +| Metric | Description | +|---|---| +| **Net churn** | `lines_added - lines_removed` | +| **Files changed** | Number of files modified in the tick | +| **Lines added** | Total lines added across all commits in the tick | +| **Lines removed** | Total lines removed across all commits in the tick | +| **Language diversity** | Number of distinct programming languages modified | +| **Author count** | Number of distinct developers active in the tick | + +A Z-score measures how many standard deviations a value is from the rolling mean. When any metric's absolute Z-score exceeds the threshold, the tick is flagged as anomalous. + +!!! info "Z-score interpretation" + - **|Z| < 2.0**: Within normal variation + - **|Z| 2.0 - 3.0**: Unusual, worth investigating + - **|Z| > 3.0**: Highly anomalous, likely a significant event + +### Sliding window + +The Z-score is computed against a trailing window of the previous N ticks (default 20). This means the "normal" baseline adapts to gradual trends, and only sudden deviations are flagged. + +### Anomaly records + +Each detected anomaly includes: + +- **Tick index**: When it occurred +- **Z-scores**: Per-metric Z-scores showing which dimension is anomalous +- **Max absolute Z-score**: The highest Z-score across all metrics (used for severity ranking) +- **Raw metrics**: The actual values for that tick +- **Files**: List of files changed in the anomalous tick + +Anomalies are sorted by severity (highest absolute Z-score first). + +--- + +## Architecture + +The anomaly analyzer follows the **TC/Aggregator pattern**: + +1. **Consume phase**: For each commit, `Consume()` extracts file changes, line stats, language diversity, and author ID, returning per-commit metrics as a `TC{Data: *CommitAnomalyData}`. The analyzer retains no per-commit state. +2. **Aggregation phase**: An `anomaly.Aggregator` collects TCs, merges `CommitAnomalyData` into per-tick `TickMetrics` (aggregating files, lines, languages, and unique authors), and produces `TICK` results. +3. **Z-score detection**: `ticksToReport()` runs sliding-window Z-score analysis on six dimensions over the aggregated tick metrics to detect anomalies. +4. **Serialization phase**: `SerializeTICKs()` converts aggregated TICKs into JSON, YAML, binary, or HTML plot output via `ComputeAllMetrics()`. + +This separation enables streaming output, budget-aware memory spilling, and decoupled aggregation. + +--- + +## Use cases + +- **Quality gates**: Flag releases or sprints that contain anomalous commit patterns as higher risk. +- **Incident correlation**: Overlay anomaly timelines with production incident timelines to find patterns. +- **Process monitoring**: Detect process breakdowns -- e.g., a sudden spike in single-author commits to many files may indicate bypassed code review. +- **Onboarding safety**: Monitor anomaly rates during onboarding periods. New developers may produce unusual patterns. +- **Seasonal pattern detection**: Over long time periods, the analyzer reveals cyclical patterns (release crunches, holiday slowdowns). +- **Multi-metric alerts**: Unlike simple threshold alerts on individual metrics, the Z-score approach adapts to the repository's natural variance. + +--- + +## Limitations + +- **Cold start**: The first `window_size` ticks have no or limited history for Z-score computation. Anomalies during this period are less reliable. +- **Gradual drift**: Z-scores detect sudden deviations, not gradual trends. A slow but steady increase in churn will update the rolling mean and never trigger an anomaly. +- **Zero-variance sentinel**: When the window has zero standard deviation (all values identical) and the current value differs, a sentinel Z-score of 100 is assigned. This ensures detection but may inflate severity rankings. +- **Tick granularity**: Metrics are aggregated per tick (default 24-hour window). Sub-tick anomalies are not detected. +- **Not causal**: The analyzer detects statistical anomalies, not root causes. Investigation is required to determine whether an anomaly represents a real problem. +- **Multi-dimensional**: An anomaly is flagged when *any* single metric exceeds the threshold. This means false positive rates increase with the number of metrics (currently six). + +--- + +## See also + +- [Anomaly reference](../analyzers/anomaly.md) — configuration keys and output schema. +- [Quick start](../getting-started/quickstart.md) — run history analysis. diff --git a/site/explanation/burndown.md b/site/explanation/burndown.md new file mode 100644 index 00000000..3b6cc438 --- /dev/null +++ b/site/explanation/burndown.md @@ -0,0 +1,97 @@ +# Understanding code burndown + +This page explains the mental model behind the burndown analyzer: what code +survival means, the per-file and per-developer breakdowns, and how the +line-tracking algorithm works. For configuration keys and the output schema, +see the [Burndown reference](../analyzers/burndown.md). + +--- + +## What it measures + +The burndown analyzer tracks **code survival over time** by following every +line of code through Git history. It produces burndown charts showing how code +written at different points in time persists, is modified, or is deleted. +Optionally, it tracks per-file and per-developer breakdowns. + +### Global code survival + +A time-series matrix where each row is a sampling point and each column is an age band. The value at `[sample][band]` is the number of lines that were last edited during that band and still survive at that sample point. + +This matrix produces the classic **burndown chart**: stacked area plots showing how much code from each era remains. + +### Per-file burndown + +When `--burndown-files` is enabled, the analyzer produces a separate survival matrix for each file, enabling file-level burndown charts. + +### Per-developer burndown + +When `--burndown-people` is enabled, the analyzer tracks which developer last edited each line. This reveals: + +- **Developer survival rates**: How much of each developer's code persists +- **Interaction matrix**: Which developers modify each other's code + +### File ownership + +When both `--burndown-files` and `--burndown-people` are enabled, the analyzer computes per-file ownership by iterating the live line segments in each file's internal tree. Each segment stores a packed `[author|tick]` value, from which the author ID is extracted to produce a `file -> author -> line_count` mapping. + +!!! note "Ownership requires developer tracking" + File ownership data is only available when `--burndown-people` is enabled. Without developer tracking, no author information is stored in the line segments, so file ownership will be empty. + +!!! warning "Memory usage" + Per-file and per-developer tracking significantly increases memory usage. For repositories with more than 100k commits, consider enabling hibernation (on by default). + +--- + +## How it works + +### Algorithm overview + +1. **Commit traversal**: Commits are processed sequentially in topological order. For each commit, the analyzer diffs the parent tree against the current tree to find inserted, deleted, and modified lines. + +2. **Line tracking**: Each line in every file is tracked using an augmented balanced binary tree (treap). Each tree node stores a contiguous range of lines with a packed value encoding `[author_id | creation_tick]`. When lines are inserted or deleted, the tree is split and merged to maintain the correct line-to-value mapping. + +3. **Sparse history accumulation**: At each commit, the analyzer records deltas in a sparse history structure: `map[currentTick]map[creationTick]lineCountDelta`. This captures how many lines created at `creationTick` are alive at `currentTick`. + +4. **Tick aggregation**: Sparse deltas from individual commits are accumulated into tick-level snapshots by a streaming aggregator. At each sampling boundary, the aggregator emits a `TickResult` containing the full sparse history state. + +5. **Dense matrix conversion**: The sparse history is converted to a dense matrix (`DenseHistory`) via `groupSparseHistory()`. This involves: + - **Tick normalization**: Map arbitrary tick values to sorted indices + - **Granularity grouping**: Collapse adjacent creation ticks into age bands of width `granularity` + - **Forward-fill**: For each age band, carry forward the last known value across sampling gaps to produce a complete matrix + +6. **Metrics computation**: The dense matrix is used to compute aggregate statistics (survival rate, peak lines, current lines), per-developer survival, and the interaction matrix. + +### Key parameters + +- **Granularity** controls the width of each age band (in ticks). Higher values produce fewer, wider bands. +- **Sampling** controls how frequently snapshots are taken (in ticks). Higher values reduce the number of data points. +- **Tick size** defaults to a 24-hour window. All commits within the same day share one tick. + +--- + +## Use cases + +- **Project health monitoring**: Track the overall code survival rate. A declining rate may indicate churn or instability. +- **Developer contribution analysis**: Understand whose code persists and who rewrites existing code. +- **Code age visualization**: Generate burndown charts showing how much ancient code remains in the codebase. +- **Refactoring impact**: Measure how much code a refactoring effort actually replaced. +- **Team dynamics**: The interaction matrix reveals collaboration patterns -- who reviews and modifies whose code. + +--- + +## Limitations + +- **Sequential processing**: Burndown tracks cumulative per-line state across all commits and must process commits sequentially. It cannot be parallelized across commits (though per-file processing within a commit is parallelized via goroutines). +- **Memory intensive**: Every line in every file is tracked throughout history. Large repositories (100k+ commits, 10k+ files) can require several GB of RAM. Use hibernation options to manage memory. +- **Binary files excluded**: Binary files are automatically skipped since they cannot be meaningfully diff'd line-by-line. +- **Rename tracking**: File renames are tracked using Git's rename detection. If Git does not detect a rename (e.g., content changed significantly), the file appears as a deletion + insertion. +- **Tick resolution**: The default 24-hour tick means that all commits within the same day share one tick. Sub-day granularity is not supported. +- **File ownership without developer tracking**: File ownership data requires `--burndown-people`. Without it, the `file_survival` array will have empty ownership maps. + +--- + +## See also + +- [Burndown reference](../analyzers/burndown.md) — configuration keys and output schema. +- [Quick start](../getting-started/quickstart.md) — run history analysis. diff --git a/site/explanation/clones.md b/site/explanation/clones.md new file mode 100644 index 00000000..44dd5cee --- /dev/null +++ b/site/explanation/clones.md @@ -0,0 +1,107 @@ +# Understanding clone detection + +This page explains the mental model behind the clone detection analyzer: what +a clone pair is, the clone-type taxonomy, and why the two-stage MinHash + LSH +design keeps cross-file detection fast. For configuration keys and the output +schema, see the [Clones reference](../analyzers/clones.md). + +--- + +## What it measures + +The clone detection analyzer finds **duplicate and near-duplicate functions** +across your entire codebase using MinHash signatures and Locality-Sensitive +Hashing (LSH). It operates on the UAST representation and detects clones +**cross-file** — not just within a single file. + +### Clone pairs + +A clone pair is two functions whose AST structure is similar enough to be considered duplicated code. Each pair includes a **similarity score** (0.0–1.0) and a **clone type** classification. + +### Clone types + +| Type | Similarity | Description | +|---|---|---| +| **Type-1 (Exact)** | = 1.0 | Identical AST structure and tokens | +| **Type-2 (Renamed)** | ≥ 0.8 | Identical structure, different variable/function names | +| **Type-3 (Near-miss)** | ≥ 0.5 | Similar but modified structure | + +### Clone ratio + +The ratio of detected clone pairs to total functions, computed globally across all files: + +``` +clone_ratio = total_clone_pairs / total_functions +``` + +!!! info "Interpretation" + - **0.0** (green): No duplication detected + - **≤ 0.1** (yellow): Low duplication — few clone pairs + - **≤ 0.3** (red): Moderate duplication — consider refactoring + - **> 0.3**: High duplication — significant refactoring recommended + +--- + +## Methodology + +### Stage 1: per-file signature extraction + +For each source file, the analyzer: + +1. **Parses** the file into a UAST and finds all function/method nodes. +2. **Shingles** each function by performing a pre-order traversal of the AST subtree and extracting sliding windows of `k` consecutive node types (k=5 by default). For example, a function with nodes `[Function, Block, Assignment, Identifier, Call, Return]` produces shingles like `Function|Block|Assignment|Identifier|Call` and `Block|Assignment|Identifier|Call|Return`. +3. **MinHash** compresses each function's shingle set into a fixed-size signature of 128 hash values. Two functions with identical structure produce identical signatures; similar functions produce similar signatures. + +Signatures are exported per-file along with function names. At this stage, no clone detection occurs — comparison is deferred to the aggregation stage. + +### Stage 2: cross-file clone detection + +After all files have been processed, the aggregator: + +1. **Qualifies** function names with their source file path (`path/to/file.go::FuncName`) so that same-named functions across files are distinguishable. +2. **Builds a global LSH index** (16 bands, 8 rows per band) from all collected signatures. LSH hashes each signature's bands into buckets — functions whose signatures share any bucket are candidate clones. +3. **Queries** the LSH index for each function, retrieves candidates above the similarity threshold (0.5), computes exact Jaccard similarity from the full MinHash signatures, and classifies each pair by clone type. +4. **Computes metrics** globally: `clone_ratio = total_pairs / total_functions`. + +This two-stage design means clone detection works across file boundaries. A controller duplicated in `pkg/a/handler.go` and `pkg/b/handler.go` is detected even though each file is parsed independently. + +### Why MinHash + LSH? + +Pairwise comparison of N functions is O(N²). For a project with 100K functions this is infeasible. MinHash compresses each function into a 1 KB signature (128 hashes × 8 bytes), and LSH reduces candidate pair generation to near-linear time by only comparing functions that hash into the same bucket. The combination provides probabilistic guarantees on recall while keeping the analysis fast. + +--- + +## Use cases + +- **Refactoring targets**: Identify exact and near-duplicate functions that should be consolidated into shared utilities. +- **Code review gates**: Flag pull requests that introduce new clone pairs, especially Type-1 (exact) duplicates. +- **Copy-paste detection**: Find controllers, handlers, or boilerplate that were copy-pasted across packages instead of being extracted. +- **Technical debt tracking**: Monitor clone ratio over time to ensure duplication does not grow unbounded. +- **Cross-team coordination**: Detect when multiple teams independently implement the same logic in different packages. + +--- + +## Methodology references + +- **MinHash**: Broder, A. Z. (1997). *On the resemblance and containment of documents*. Proceedings of the Compression and Complexity of Sequences. MinHash estimates Jaccard similarity between sets using compact fixed-size signatures. +- **LSH (Locality-Sensitive Hashing)**: Indyk, P. & Motwani, R. (1998). *Approximate nearest neighbors: towards removing the curse of dimensionality*. STOC. Banded LSH reduces candidate pair generation from O(N²) to near-linear time. +- **AST-based clone detection**: Jiang, L. et al. (2007). *DECKARD: Scalable and accurate tree-based detection of code clones*. ICSE. Tree-based shingling captures structural similarity independent of identifier names. +- **Clone taxonomy**: Roy, C. K. & Cordy, J. R. (2007). *A survey on software clone detection research*. The Type-1/Type-2/Type-3 classification follows the standard clone taxonomy. + +--- + +## Limitations + +- **Language scope**: Works with any language supported by the UAST parser. Unsupported files are silently skipped. +- **Structural only**: Detection is based on AST node types, not token values. Two functions with identical structure but completely different semantics (e.g., add vs multiply with the same nesting shape) may appear as clones. +- **Minimum function size**: Functions smaller than the shingle window (5 AST nodes) produce no shingles and are excluded from clone detection. They are still counted in `total_functions`. +- **Probabilistic recall**: LSH is a probabilistic algorithm. Clone pairs with similarity just above the threshold (0.5) may occasionally be missed. High-similarity pairs (Type-1 and Type-2) have near-perfect recall. +- **Generated code**: The analyzer does not distinguish hand-written code from generated code. Consider excluding generated directories. +- **No Type-4 clones**: Semantic clones (different structure, same behavior) are not detected. The analyzer operates purely on syntactic structure. + +--- + +## See also + +- [Clones reference](../analyzers/clones.md) — configuration keys and output schema. +- [Quick start](../getting-started/quickstart.md) — run your first analysis. diff --git a/site/explanation/cohesion.md b/site/explanation/cohesion.md new file mode 100644 index 00000000..ab52c417 --- /dev/null +++ b/site/explanation/cohesion.md @@ -0,0 +1,108 @@ +# Understanding cohesion metrics + +This page explains the mental model behind the cohesion analyzer: what +LCOM-HS measures, how the variable-sharing ratio captures communicational +cohesion, and how to read the scores. For configuration keys and the output +schema, see the [Cohesion reference](../analyzers/cohesion.md). + +--- + +## What it measures + +The cohesion analyzer computes **LCOM-HS (Henderson-Sellers)** and **variable +sharing ratio** metrics to identify files and modules with low internal +cohesion. Low cohesion indicates functions that are poorly related to each +other — a strong signal for refactoring. + +### LCOM-HS (lack of cohesion of methods — Henderson-Sellers) + +LCOM-HS is the industry-standard cohesion metric used by NDepend, JArchitect, and CppDepend. + +**Formula:** + +``` +LCOM = 1 - sum(mA) / (m × a) +``` + +Where: + +- `m` = number of functions in the file +- `a` = number of distinct variables across all functions +- `mA` = for each variable, the count of functions that reference it +- `sum(mA)` = sum of access counts across all variables + +**Range:** 0.0 (perfect cohesion) to 1.0 (no cohesion). + +!!! info "Interpretation" + - **LCOM ≤ 0.3** (green): Excellent cohesion — functions share most variables. + - **LCOM ≤ 0.6** (yellow): Moderate cohesion — some functions may be loosely related. + - **LCOM > 0.6** (red): Poor cohesion — functions share few variables and may belong in separate files. + +### Cohesion score + +A convenience inversion of LCOM-HS: `cohesion_score = 1.0 - LCOM`. Higher is better. + +| Cohesion Score | Assessment | +|---|---| +| ≥ 0.7 | Excellent | +| ≥ 0.4 | Good | +| ≥ 0.3 | Fair | +| < 0.3 | Poor | + +### Function-level cohesion (variable sharing ratio) + +For each function, the analyzer computes what fraction of its variables are shared with at least one other function in the same file. This measures **communicational cohesion** (Yourdon-Constantine level 5). + +**Formula:** + +``` +function_cohesion = shared_vars / total_unique_vars +``` + +Where: + +- `shared_vars` = variables in this function that also appear in at least one other function +- `total_unique_vars` = total distinct variables in this function +- Functions with no variables receive a score of 1.0 (trivial, no penalty) + +**Range:** 0.0 (completely isolated function) to 1.0 (all variables shared). + +| Function Cohesion | Assessment | +|---|---| +| ≥ 0.6 | Excellent — function shares most variables with the module | +| ≥ 0.4 | Good — reasonable sharing with room for improvement | +| ≥ 0.3 | Fair — consider refactoring | +| < 0.3 | Poor — function is isolated from the module | + +--- + +## Use cases + +- **Refactoring targets**: Find functions with low sharing ratio that do unrelated work and should be split. +- **Architecture reviews**: Validate that files follow the Single Responsibility Principle by checking LCOM-HS scores. +- **Code quality tracking**: Monitor cohesion trends over time to catch degradation. +- **Code review**: Use per-function cohesion scores to identify newly added functions that lack cohesion with existing code. + +--- + +## Methodology references + +- **LCOM-HS**: Henderson-Sellers, B. (1996). *Object-Oriented Metrics: Measures of Complexity*. Prentice Hall. The Henderson-Sellers variant normalizes LCOM to [0, 1] and is the variant used by NDepend, JArchitect, and CppDepend. +- **Variable Sharing Ratio**: Measures communicational cohesion per the Yourdon-Constantine classification. A function that shares all its variables with other functions in the module exhibits high communicational cohesion. + +--- + +## Limitations + +- **Language scope**: Works with any language supported by the UAST. Best results with languages that have explicit variable declarations and function definitions. +- **Accessor methods**: Simple getters/setters may inflate cohesion since each touches only one shared field. +- **Variable naming**: The analyzer uses lexical variable names. Different variables with the same name across functions will be counted as shared. +- **Single-function files**: Files with only one function always receive perfect cohesion (LCOM = 0.0, cohesion = 1.0) since there are no other functions to compare against. +- **Trivial functions**: Functions with no variables receive a cohesion score of 1.0 to avoid penalizing simple utility functions. + +--- + +## See also + +- [Cohesion reference](../analyzers/cohesion.md) — configuration keys and output schema. +- [Quick start](../getting-started/quickstart.md) — run your first analysis. diff --git a/site/explanation/comments.md b/site/explanation/comments.md new file mode 100644 index 00000000..53888e5b --- /dev/null +++ b/site/explanation/comments.md @@ -0,0 +1,59 @@ +# Understanding documentation coverage + +This page explains the mental model behind the comments analyzer: what +documentation coverage means, how the analyzer distinguishes comment types, +and what a healthy comment density looks like. For configuration keys and the +output schema, see the [Comments reference](../analyzers/comments.md). + +--- + +## What it measures + +The comments analyzer evaluates **documentation coverage** by examining comment +presence, placement, and density across your source code. It identifies +functions and classes that lack documentation and measures overall +documentation quality. + +### Documentation coverage + +The percentage of public functions, methods, and classes that have an associated documentation comment. A comment is "associated" when it immediately precedes the declaration. + +### Comment placement quality + +Not all comments are equally useful. The analyzer distinguishes: + +- **Doc comments**: Comments immediately before a function/class declaration +- **Inline comments**: Comments within function bodies +- **Orphan comments**: Comments not associated with any code structure + +### Comment density + +The ratio of comment lines to total lines of code. Provides a quick overall measure of how well-documented a file is. + +!!! info "Healthy ratios" + A comment density between **15-30%** is generally considered healthy. Below 10% suggests under-documentation; above 40% may indicate stale or redundant comments. + +--- + +## Use cases + +- **Documentation enforcement**: Require minimum documentation coverage in CI/CD pipelines. +- **Onboarding assessment**: Measure how well-documented a codebase is before new team members join. +- **API surface quality**: Ensure all exported/public functions have doc comments. +- **Stale comment detection**: Identify files with unusually high comment density that may contain outdated comments. + +--- + +## Limitations + +- **Comment quality**: The analyzer measures comment *presence* and *placement*, not comment *quality*. A comment saying `// do the thing` counts the same as a thorough description. +- **Language conventions**: Documentation comment conventions vary by language (e.g., `///` in Rust, `"""` in Python, `//` in Go). The UAST normalizes these, but edge cases in less common languages may be missed. +- **Auto-generated docs**: Comments generated by tools (e.g., protobuf stubs) are counted the same as hand-written documentation. +- **Multilingual comments**: Comment text is not analyzed for language quality or correctness. + +--- + +## See also + +- [Comments reference](../analyzers/comments.md) — configuration keys and output schema. +- [Quick start](../getting-started/quickstart.md) — run your first analysis. diff --git a/site/explanation/complexity.md b/site/explanation/complexity.md new file mode 100644 index 00000000..7fae0e6a --- /dev/null +++ b/site/explanation/complexity.md @@ -0,0 +1,84 @@ +# Understanding complexity metrics + +This page explains the mental model behind the complexity analyzer: what +each metric means, why the analyzer measures three dimensions, and how +cyclomatic and cognitive complexity differ. For configuration keys and the +output schema, see the [Complexity reference](../analyzers/complexity.md). + +--- + +## What it measures + +The complexity analyzer measures three dimensions of code complexity from +source code: **cyclomatic complexity**, **cognitive complexity**, and +**nesting depth**. It operates on the UAST representation of your source +files. + +### Cyclomatic complexity (McCabe, 1976) + +Counts the number of linearly independent paths through a function's control flow graph. + +For Go code, the analyzer follows the same practical counting model used by `gocyclo`: + +- Base score is `1` +- `if`, `for` / `range` add `+1` +- Non-default `case` adds `+1` +- Each `&&` and `||` adds `+1` +- `default` case does **not** add complexity + +!!! info "Interpretation" + - **1-10**: Simple, low risk + - **11-20**: Moderate complexity + - **21-50**: High complexity, consider refactoring + - **51+**: Very high risk, untestable + +### Cognitive complexity (SonarSource, 2017) + +Measures how difficult code is for a human to read and understand. Unlike cyclomatic complexity, it penalizes nested structures more heavily and rewards linear sequences of logic. + +Key differences from cyclomatic complexity: + +- Nesting increments add to the score (deeper nesting = higher penalty) +- `else if` adds structural complexity, but avoids extra nesting penalty vs deeply nested `if` +- Logical operator sequences account for readability, not just path count +- Direct recursion adds a penalty + +### Nesting depth + +Tracks the maximum depth of nested control structures within each function. Deep nesting is a strong signal for refactoring. + +--- + +## Validation against golden implementations + +The complexity analyzer is continuously validated against battle-tested Go references: + +- **Cyclomatic parity target:** `gocyclo` (v0.6.0) +- **Cognitive parity target:** `gocognit` (v1.2.1) + +For stabilization, we run a controlled methodology sample through both references and `codefang`, then align discrepancies until parity is reached. + +--- + +## Use cases + +- **Code review gates**: Reject pull requests where any function exceeds a cyclomatic complexity threshold. +- **Refactoring prioritization**: Sort functions by cognitive complexity to find the hardest-to-understand code. +- **Technical debt tracking**: Monitor complexity trends across releases. +- **Test planning**: Functions with high cyclomatic complexity need more test cases for full path coverage. + +--- + +## Limitations + +- **Language coverage**: Only languages supported by the UAST parser are analyzed. Unsupported files are silently skipped. +- **Generated code**: The analyzer does not distinguish hand-written code from generated code. Consider excluding generated directories. +- **Macros and metaprogramming**: Complexity within macros or template metaprogramming may not be fully captured, since the UAST represents the source as written, not as expanded. +- **Cognitive complexity model**: The cognitive complexity scoring follows the SonarSource specification. Other tools may use slightly different weightings. + +--- + +## See also + +- [Complexity reference](../analyzers/complexity.md) — configuration keys and output schema. +- [Quick start](../getting-started/quickstart.md) — run your first analysis. diff --git a/site/explanation/couples.md b/site/explanation/couples.md new file mode 100644 index 00000000..ea97ae76 --- /dev/null +++ b/site/explanation/couples.md @@ -0,0 +1,99 @@ +# Understanding file and developer coupling + +This page explains the mental model behind the couples analyzer: what file and +developer coupling mean, how coupling strength is computed, and how the +aggregation pipeline is structured. For configuration keys and the output +schema, see the [Couples reference](../analyzers/couples.md). + +--- + +## What it measures + +The couples analyzer identifies **file coupling** and **developer coupling** by +analyzing co-change patterns across Git history. Files that frequently change +together are likely coupled; developers who frequently touch the same files are +collaborating (or competing). + +### File coupling + +A square matrix where each cell `[i][j]` counts the number of commits in which files `i` and `j` both appeared. High co-change counts indicate tight coupling between files. + +**Coupling strength** is computed using the code-maat formula: + +``` +strength = co_changes / average(revisions_file1, revisions_file2) +``` + +Where `revisions_fileN` is the diagonal element of the file matrix (the file's self-change count). The result is capped at 1.0. This normalizes coupling by the average activity of both files, so a pair of files that change together 10 times out of 20 average revisions scores 50%. + +!!! info "Changeset size filter" + Commits touching more than **1000 files** are excluded from coupling analysis. Mass changes (e.g., formatting, license updates, dependency bumps) produce noise rather than meaningful coupling signal. + +### Developer coupling + +A matrix where each cell `[i][j]` counts the number of times developers `i` and `j` committed to the same file. Developer coupling strength uses the same code-maat formula normalized by each developer's individual commit activity. + +### File ownership + +For each tracked file, the analyzer reports its line count and the number of distinct contributors. Single-owner files are flagged as bus-factor risks. + +### Rename tracking + +The analyzer tracks file renames to avoid counting a renamed file as both a deletion and an insertion, which would break coupling chains. + +--- + +## Use cases + +- **Architecture analysis**: Highly coupled file pairs that span package boundaries may indicate architectural violations. +- **Dependency discovery**: Find implicit dependencies that are not captured by import statements. If two files always change together, they are coupled even without explicit imports. +- **Module boundary validation**: Files within the same module should be more coupled than files across modules. Cross-module coupling is a design smell. +- **Team topology**: Developer coupling reveals who collaborates with whom. This can inform team structure decisions. +- **Change impact prediction**: When modifying a file, the coupling matrix predicts which other files are likely to need changes. +- **Bus factor assessment**: File ownership data identifies single-owner files that represent knowledge concentration risks. + +--- + +## Architecture + +The couples analyzer follows the **TC/Aggregator pattern**: + +1. **Consume phase**: Each commit produces a `TC{Data: *CommitData}` containing the coupling context (list of co-changed files), per-file author touch counts (always 1 per file per commit), rename pairs, and whether the author's commit count was incremented. Commits exceeding `CouplesMaximumMeaningfulContextSize` (1000 files) are skipped, producing an empty coupling context. + +2. **Aggregation phase**: The `Aggregator` accumulates the file co-occurrence matrix using `SpillStore[map[string]int]`, along with per-person file touch counts (`people`), commit counts (`peopleCommits`), and rename tracking. When memory pressure exceeds `SpillBudget`, the file coupling matrix is spilled to disk via gob encoding. `Collect()` merges spilled data using additive merge semantics. + +3. **Serialization phase**: `ticksToReport()` reconstructs the full report (PeopleMatrix, PeopleFiles, Files, FilesLines, FilesMatrix, ReversedPeopleDict) from aggregated TICKs, then `ComputeAllMetrics()` produces the final typed output for any format (text, plot, JSON, YAML, binary). + +**Working state** (`merges`, `seenFiles`) stays in the analyzer for merge-mode dedup across commits. **Accumulated output** (file couplings, people maps, renames) is owned entirely by the aggregator. + +--- + +## Methodology + +The coupling strength formula follows the **code-maat** academic standard (Adam Tornhill, "Your Code as a Crime Scene"): + +``` +degree = shared_revisions / average(revisions_A, revisions_B) +``` + +This measures what fraction of a file pair's average activity is shared. A coupling strength of 80% means the pair changes together in 80% of their average revision count. + +The aggregate `avg_coupling_strength` is the mean of all per-pair coupling strengths (not the mean of raw co-change counts). + +**Highly coupled pairs** are those with 10 or more co-changes (raw count threshold). + +--- + +## Limitations + +- **Large commits excluded**: Commits touching more than 1000 files are skipped to avoid noise from mass changes (formatting, license headers, dependency updates). +- **No temporal decay**: All commits are weighted equally. A coupling from long ago counts the same as a recent one. Consider filtering by date range for recent coupling analysis. +- **Merge commits**: Merge commits are processed only once (first encounter) to avoid double-counting changes that appear in multiple branches. +- **File deletions**: Deleted files are included in the coupling matrix during the analysis window but may not appear in the final output if they no longer exist at HEAD. + +--- + +## See also + +- [Couples reference](../analyzers/couples.md) — configuration keys and output schema. +- [Quick start](../getting-started/quickstart.md) — run history analysis. diff --git a/site/explanation/developers.md b/site/explanation/developers.md new file mode 100644 index 00000000..10ce0576 --- /dev/null +++ b/site/explanation/developers.md @@ -0,0 +1,102 @@ +# Understanding developer analysis + +This page explains the mental model behind the developers analyzer: what +per-developer statistics it computes, how the bus factor is derived, and how +the aggregation pipeline is structured. For configuration keys and the output +schema, see the [Developers reference](../analyzers/developers.md). + +--- + +## What it measures + +The developers analyzer computes **per-developer contribution statistics** +across Git history, including commit counts, line changes, language breakdown, +bus factor risk, and activity time series. + +### Developer statistics + +For each developer (identified by the identity detector): + +- **Commits**: Total number of commits +- **Lines added / removed / changed**: Aggregate line statistics +- **Net lines**: `added - removed`, showing net contribution +- **Languages**: Breakdown of line changes per programming language +- **Active period**: First and last ticks of activity, number of active ticks + +### Language statistics + +Aggregated across all developers: + +- **Total lines per language**: Lines added (for backward compatibility) +- **Total contribution per language**: Lines added + removed, used for bus factor and ownership calculations +- **Contributors per language**: Which developers contribute to each language, measured by total contribution (added + removed) + +### Bus factor + +Knowledge concentration risk per language, following the [CHAOSS Contributor Absence Factor](https://chaoss.community/kb/metric-bus-factor/) methodology. + +For each language, the analyzer computes: + +- **Bus factor number**: The smallest number of contributors responsible for 50% of total contributions (added + removed). This is the CHAOSS standard metric. +- **Total contributors**: Number of unique contributors to that language. +- **Primary/secondary owner**: The top two contributors and their ownership percentages. +- **Risk level**: Based on primary owner concentration. + +!!! danger "Risk levels" + - **CRITICAL** (>= 90%): A single developer owns nearly all the code + - **HIGH** (>= 80%): Very concentrated ownership + - **MEDIUM** (>= 60%): Moderate concentration + - **LOW** (< 60%): Healthy distribution + +A **project-level bus factor** is also computed across all developers and all languages, reported in the aggregate section. + +### Activity time series + +Per-tick commit counts broken down by developer. Shows contribution velocity over time. + +### Code churn (line velocity) + +Per-tick lines added and removed. This measures raw line velocity — the volume of code changes over time. High values may indicate refactoring, feature development, or instability. + +!!! note "Terminology" + In academic literature, "code churn" specifically refers to recently-written code that is quickly rewritten. This analyzer measures the broader concept of line velocity (total additions and removals per time period). + +--- + +## Architecture + +The developers analyzer is built on the **BaseHistoryAnalyzer** and **GenericAggregator** foundations: + +1. **Consume phase**: `Consume()` extracts author ID, line stats, and language breakdown, delegating state storage to the generic aggregator framework. The analyzer retains minimal internal state. +2. **Aggregation phase**: The `GenericAggregator` automatically handles per-commit memory spilling and per-tick grouping using pure function hooks (`extractTC`, `mergeState`, `buildTick`), eliminating custom state management boilerplate. +3. **Metrics & UI phase**: A pure function pipeline (`ComputeAllMetrics`) generates the output structures for JSON, YAML, and text, while declarative chart builders (`plotpage.BuildBarChart`, `plotpage.BuildLineChart`) render the HTML visualizations. + +This unified approach significantly reduces boilerplate while maintaining full support for streaming output, decoupled aggregation, and budget-aware memory spilling. + +--- + +## Use cases + +- **Team assessment**: Understand who contributes what, in which languages, and when. +- **Bus factor analysis**: Identify languages or components where a single developer departure would create critical knowledge gaps. The CHAOSS bus factor number tells you how many people need to leave before 50% of knowledge is lost. +- **Activity monitoring**: Track developer engagement over time. Declining activity may signal burnout or attrition risk. +- **Language migration tracking**: Monitor the adoption of a new language by watching language statistics over time. +- **Onboarding evaluation**: Measure how quickly new team members ramp up by comparing their activity curves. +- **Code churn analysis**: Detect periods of high line velocity that may correlate with instability or deadline pressure. + +--- + +## Limitations + +- **Identity resolution**: Developer identity is determined by the identity detector (email-based by default). Multiple email addresses for the same person will appear as separate developers unless a mailmap is configured. +- **Merge commits**: By default, merge commits are processed only once (first encounter). Trivial merges are skipped unless `ConsiderEmptyCommits` is enabled. +- **Line attribution**: Lines are attributed to the commit author, not the committer. In workflows with heavy rebasing, this may differ from expectations. +- **Contribution measurement**: Contributions are measured as lines added + lines removed. This gives fair credit to refactoring work but does not distinguish between code complexity or quality. +- **Active developer threshold**: "Active developers" are those with commits in the most recent 90-day window (when tick size is known). Falls back to the recent 30% of the analysis period when tick size is unavailable. This threshold is not configurable. + +--- + +## See also + +- [Developers reference](../analyzers/developers.md) — configuration keys and output schema. +- [Quick start](../getting-started/quickstart.md) — run developer analysis. diff --git a/site/explanation/file-history.md b/site/explanation/file-history.md new file mode 100644 index 00000000..cb303d4c --- /dev/null +++ b/site/explanation/file-history.md @@ -0,0 +1,68 @@ +# Understanding file history analysis + +This page explains the mental model behind the file history analyzer: what +per-file lifecycle data it records, how hotspots and churn are derived, and how +rename detection maintains continuity. For configuration keys and the output +schema, see the [File history reference](../analyzers/file-history.md). + +--- + +## What it measures + +The file history analyzer tracks the **lifecycle of every file** through Git +history, recording which commits touched each file, which developers modified +it, and aggregating line statistics per contributor. It supports rename +detection to maintain continuity across file moves. + +### Per-file commit history + +For each file present in the repository at HEAD, the analyzer records the ordered list of commits that created, modified, or deleted it. + +### Per-file contributor breakdown + +For each file, a map of developer IDs to line statistics (added, removed, changed). This shows who contributed what to each file. + +### Code hotspots + +Files with high commit counts are flagged as hotspots with risk levels: + +!!! warning "Hotspot risk levels" + - **CRITICAL**: >= 50 commits + - **HIGH**: >= 30 commits + - **MEDIUM**: >= 15 commits + - Files below 15 commits are not flagged + +### File churn + +A composite score combining commit frequency and line change volume. High-churn files may indicate instability or areas of active development. + +### Rename support + +When Git detects a file rename (e.g., `old/path.go` to `new/path.go`), the analyzer transfers the full history from the old path to the new path, maintaining a continuous record. + +--- + +## Use cases + +- **Hotspot identification**: Find the files that change most often. These are the highest-risk files for bugs and the best candidates for extra test coverage and code review scrutiny. +- **Code ownership mapping**: Determine who is the primary contributor for each file to establish code ownership. +- **Onboarding guides**: New team members can see which files each developer owns to know who to ask about specific code. +- **Refactoring ROI**: Identify files with both high churn and many contributors -- refactoring these produces the largest payoff. +- **Risk assessment**: Files with a single contributor and high commit counts are both hot and concentrated -- a bus factor risk. + +--- + +## Limitations + +- **HEAD-only output**: Only files present at HEAD are included in the final output. Files that were deleted before the last commit are tracked during analysis but excluded from results. +- **Merge handling**: Merge commits are processed only once to avoid double-counting. Non-merge context changes in merge commits are skipped. +- **Rename detection**: Depends on Git's rename detection heuristics. If a file is both renamed and heavily modified in the same commit, Git may not detect the rename. +- **Churn score formula**: The churn score is `commit_count + (added + removed + changed) / 100`. This is a simple heuristic, not a rigorous statistical measure. +- **No time-series**: Unlike the burndown analyzer, file history does not produce time-series data. It provides aggregate per-file statistics only. + +--- + +## See also + +- [File history reference](../analyzers/file-history.md) — configuration keys and output schema. +- [Quick start](../getting-started/quickstart.md) — run history analysis. diff --git a/site/explanation/halstead.md b/site/explanation/halstead.md new file mode 100644 index 00000000..64032360 --- /dev/null +++ b/site/explanation/halstead.md @@ -0,0 +1,69 @@ +# Understanding Halstead metrics + +This page explains the mental model behind the Halstead analyzer: what +operators and operands are, which derived metrics matter most, and how the +counting policy keeps results stable across languages. For configuration keys +and the output schema, see the [Halstead reference](../analyzers/halstead.md). + +--- + +## What it measures + +The Halstead analyzer computes **Halstead complexity metrics** (1977) based on +operator and operand counts extracted from the UAST. These metrics provide an +objective, quantitative assessment of program size and complexity. + +The Halstead model treats a program as a sequence of **operators** (keywords, symbols, function calls) and **operands** (variables, literals, constants). From these counts, it derives: + +| Metric | Formula | Description | +|---|---|---| +| **n1** | -- | Number of distinct operators | +| **n2** | -- | Number of distinct operands | +| **N1** | -- | Total number of operators | +| **N2** | -- | Total number of operands | +| **Vocabulary** (n) | n1 + n2 | Distinct tokens used | +| **Length** (N) | N1 + N2 | Total tokens in the program | +| **Volume** (V) | N * log2(n) | Information content in bits | +| **Difficulty** (D) | (n1/2) * (N2/n2) | Error proneness | +| **Effort** (E) | D * V | Mental effort to implement | +| **Time** (T) | E / 18 | Estimated implementation time (seconds) | +| **Bugs** (B) | V / 3000 | Estimated delivered bugs | + +!!! info "Key insight" + **Volume** measures the size of the implementation. **Difficulty** captures how error-prone it is. Their product, **Effort**, is the best single number for comparing overall complexity. + +### Counting policy + +Codefang computes Halstead metrics from UAST with a lexical-first policy: + +- Counts lexical operands (`Identifier`, `Literal`, `Field`) and operator nodes/roles. +- Excludes structural wrappers and declaration-only artifacts from operand counts. +- Uses explicit operator properties when available, then token extraction fallback. + +This improves stability across languages and avoids pseudo-operands like structural `Parameter` nodes. + +--- + +## Use cases + +- **Effort estimation**: Use the Effort metric to compare the relative complexity of different modules or features. +- **Bug prediction**: The Bugs metric provides a rough upper bound on expected defects. Modules with high estimated bugs warrant more thorough testing. +- **Code review guidance**: Functions with high Difficulty scores are more error-prone and deserve extra scrutiny. +- **Language comparison**: Halstead metrics allow cross-language comparisons since they are based on abstract operator/operand counts. + +--- + +## Limitations + +- **Cross-tool variance**: Language-specific tools (for example, Python- or ESTree-only analyzers) can differ in tokenization and counting policy. Compare trends within one toolchain rather than mixing absolute numbers across tools. +- **Operator classification**: UAST-based classification targets cross-language consistency, not byte-for-byte parity with each language parser. +- **Estimation accuracy**: The Bugs and Time formulas are empirical approximations from the 1970s. Treat them as relative indicators, not precise predictions. +- **Macro expansion**: Halstead metrics count tokens as written, not as expanded. Heavy use of macros or code generation can skew results. +- **Comments excluded**: Comments and whitespace are excluded from Halstead counts (by design). + +--- + +## See also + +- [Halstead reference](../analyzers/halstead.md) — configuration keys and output schema. +- [Quick start](../getting-started/quickstart.md) — run your first analysis. diff --git a/site/explanation/imports.md b/site/explanation/imports.md new file mode 100644 index 00000000..5ae6572d --- /dev/null +++ b/site/explanation/imports.md @@ -0,0 +1,67 @@ +# Understanding import and dependency analysis + +This page explains the mental model behind the imports analyzer: what it +measures in static and history modes, and how the history-mode aggregation +pipeline works. For configuration keys and the output schema, see the +[Imports reference](../analyzers/imports.md). + +--- + +## What it measures + +The imports analyzer extracts **import and dependency information** from source +code using UAST parsing. It operates in both static mode (single-file analysis) +and history mode (tracking import usage per developer over time). + +### Static mode + +- **Import list**: All imports/dependencies declared in each file +- **Language detection**: Automatically detects the language and normalizes import paths +- **Dependency graph**: Maps which files depend on which packages + +### History mode + +Tracks import usage across Git history, producing a per-developer, per-language, per-tick breakdown of which dependencies each developer introduces or modifies. This reveals: + +- **Dependency adoption timeline**: When new libraries were introduced +- **Developer expertise signals**: Which developers work with which dependencies +- **Technology spread**: How quickly new dependencies propagate across the team + +--- + +## Architecture (history mode) + +The imports history analyzer follows the **TC/Aggregator pattern**: + +1. **Consume phase**: For each commit, `Consume()` extracts imports from changed files via parallel UAST parsing and returns them as `TC{Data: []ImportEntry}`. Each `ImportEntry` carries a language and import path. The analyzer retains no per-commit state; only the UAST parser is kept as working state. +2. **Aggregation phase**: An `imports.Aggregator` collects TCs into a 4-level `Map` (author -> language -> import -> tick -> count) using `SpillStore[Map]`. The `AuthorID` and `Tick` from each TC index the entries correctly. +3. **Serialization phase**: `SerializeTICKs()` merges all tick data back into the full `Map` with metadata (`author_index`, `tick_size`), then delegates to `Serialize()` for JSON, YAML, binary, or HTML plot output. + +This separation enables streaming output, budget-aware memory spilling, and decoupled aggregation. + +--- + +## Use cases + +- **Dependency auditing**: List all third-party dependencies used in a project. +- **Developer profiling**: Understand which developers work with which frameworks and libraries. +- **Technology adoption tracking**: Monitor when and how quickly new dependencies spread across the team. +- **License compliance**: Extract the full dependency list for license scanning pipelines. +- **Architecture enforcement**: Detect unauthorized imports from forbidden packages. + +--- + +## Limitations + +- **Language support**: Only languages with UAST parser support are analyzed. Unsupported file types are silently skipped. +- **Dynamic imports**: Runtime or dynamic imports (e.g., Python's `importlib.import_module()`, JavaScript's `import()`) are not detected since they are not present in the static UAST. +- **Transitive dependencies**: Only direct imports are reported. The analyzer does not resolve transitive dependency trees. +- **File size threshold**: Files exceeding `MaxFileSize` (default 1 MB) are skipped to avoid excessive memory usage during parallel extraction. +- **History mode overhead**: History mode creates a UAST parser per fork, which increases memory usage. Tune `Goroutines` based on available memory. + +--- + +## See also + +- [Imports reference](../analyzers/imports.md) — configuration keys and output schema. +- [Quick start](../getting-started/quickstart.md) — run your first analysis. diff --git a/site/explanation/quality.md b/site/explanation/quality.md new file mode 100644 index 00000000..f8ef8285 --- /dev/null +++ b/site/explanation/quality.md @@ -0,0 +1,54 @@ +# Understanding code quality over time + +This page explains the mental model behind the code quality analyzer: which +static metrics it tracks per commit and how it aggregates them per tick. For +configuration keys and the output schema, see the +[Quality reference](../analyzers/quality.md). + +--- + +## What it measures + +The code quality analyzer tracks **complexity, Halstead metrics, comment +quality, and cohesion** across Git history. For each commit, it runs four +static analyzers on UAST-parsed changed files and produces per-file quality +metrics. + +### Per-file metrics + +For each changed file in a commit, the analyzer computes: + +| Category | Metrics | +|---|---| +| **Complexity** | Cyclomatic complexity, cognitive complexity, max single-function complexity, function count | +| **Halstead** | Volume, effort, delivered bugs | +| **Comments** | Overall comment score, documentation coverage | +| **Cohesion** | Cohesion score | + +### Statistical aggregation + +Per-tick statistics are computed from per-file arrays: + +- **Mean, median, P95, max** for complexity and Halstead volume +- **Sum** for delivered bugs and Halstead volume +- **Min** for comment score and cohesion (worst-case tracking) +- **Total** files analyzed and functions counted + +--- + +## Architecture + +The quality analyzer follows the **TC/Aggregator pattern**: + +1. **Consume phase**: For each commit, `Consume()` runs four static analyzers (complexity, Halstead, comments, cohesion) on each changed file's UAST, returning per-file metrics as a `TC{Data: *TickQuality}`. The analyzer retains no per-commit state. +2. **Aggregation phase**: A `quality.Aggregator` collects TCs, merges `TickQuality` data by time bucket (tick), and produces `TICK` results. +3. **Serialization phase**: `SerializeTICKs()` converts aggregated TICKs into JSON, YAML, binary, or HTML plot output via `ComputeAllMetrics()`. + +This separation enables streaming output, budget-aware memory spilling, and decoupled aggregation. + +--- + +## See also + +- [Quality reference](../analyzers/quality.md) — configuration keys and output schema. +- [Quick start](../getting-started/quickstart.md) — run history analysis. diff --git a/site/explanation/sentiment.md b/site/explanation/sentiment.md new file mode 100644 index 00000000..c796cf16 --- /dev/null +++ b/site/explanation/sentiment.md @@ -0,0 +1,161 @@ +# Understanding comment sentiment analysis + +This page explains the mental model behind the sentiment analyzer: how it +extracts and filters comments, how VADER and the multilingual lexicon score +them, and the software-engineering domain adjustments that keep technical +language from skewing the result. For configuration keys and the output schema, +see the [Sentiment reference](../analyzers/sentiment.md). + +--- + +## What it measures + +The sentiment analyzer classifies **comment sentiment** across Git history. For +each commit, it extracts new or changed comments via UAST parsing, filters out +noise, and classifies the remaining comments as positive, negative, or neutral. +This reveals how developer sentiment evolves over time. + +### Per-commit comment extraction + +For each commit, the analyzer: + +1. Computes the UAST diff between the old and new file versions +2. Extracts comment nodes from the new UAST +3. Merges adjacent comment lines into single blocks +4. Filters out noise (short comments, license headers, function signatures) +5. Returns the filtered comments as a `TC` payload (`sentiment.CommitResult`) + +### Comment filtering + +Comments are filtered using several heuristics: + +- **Minimum length**: Comments shorter than the threshold are skipped +- **Letter ratio**: At least 60% of characters must be Unicode letters (filters out commented-out code) +- **First character**: Must start with a Unicode letter or digit (supports all scripts) +- **License detection**: Comments matching license/copyright patterns (including UK "licence" spelling) are excluded +- **Function name removal**: Inline function references like `doThing()` are stripped before analysis + +### Multilingual support + +The analyzer supports comments in any language that uses Unicode letters: + +- **CJK**: Chinese, Japanese, Korean +- **Cyrillic**: Russian, Ukrainian, etc. +- **Arabic**: Right-to-left scripts +- **Latin**: English, French, German, Spanish, etc. +- All other Unicode letter scripts + +Comment filtering uses Unicode-aware regex patterns (`\p{L}` for letters, `\p{N}` for digits) rather than ASCII-only ranges, ensuring comments in non-English languages are not silently dropped. + +!!! info "Multilingual sentiment scoring" + Sentiment scoring uses VADER's English lexicon as the base, extended with **93,000+ multilingual word entries** from the Chen-Skiena lexicon dataset (ACL 2014) covering **32 languages**. Non-ASCII words from the dataset are injected into VADER's lexicon at startup, enabling basic sentiment scoring for comments in Russian, Chinese, Japanese, Korean, Arabic, and 27 other languages. VADER's grammatical rules (negation, intensifiers) still operate on English syntax, so scoring accuracy for non-English comments is lower than for English — but significantly better than no coverage. + +### Sentiment classification + +Filtered comments are classified as positive, negative, or neutral using **VADER** (Valence Aware Dictionary and sEntiment Reasoner) via the [GoVader](https://github.com/jonreiter/govader) library, enhanced with **software engineering domain adjustments**. + +#### VADER base scoring + +VADER is a lexicon and rule-based sentiment analyzer designed for social media and short text. It handles negations, intensifiers, and punctuation. The compound score (-1 to 1) is mapped to our [0, 1] range. + +#### Multilingual lexicon extension + +At startup, the analyzer injects ~93,000 multilingual word entries from the [Chen-Skiena lexicon dataset](https://aclanthology.org/P14-2063/) (ACL 2014) into VADER's lexicon. This covers 32 languages: + +| Language Family | Languages | +|---|---| +| **Slavic** | Russian, Ukrainian, Polish, Czech, Slovak, Croatian, Bulgarian | +| **CJK** | Chinese, Japanese, Korean | +| **Romance** | Spanish, French, Portuguese, Italian, Romanian | +| **Germanic** | German, Dutch, Swedish, Danish, Norwegian, Finnish | +| **Other** | Arabic, Hebrew, Hindi, Thai, Turkish, Greek, Hungarian, Indonesian, Malay, Vietnamese, Persian | + +Only non-ASCII words are injected to avoid overriding VADER's curated English entries. Words receive binary valence: +1.5 (positive) or -1.5 (negative), which is the mid-range of VADER's scale. + +To regenerate lexicons from updated source data: + +```bash +go run tools/lexgen/lexgen.go -pos pos_words.txt -neg neg_words.txt \ + -o internal/analyzers/sentiment/lexicons/lexicon_data.gen.go +``` + +#### SE-domain adjustments + +VADER frequently misclassifies technical terms that sound negative in natural language but are neutral in software engineering. The analyzer applies domain-specific adjustments: + +**Neutralized terms** (pushed toward neutral): +`kill`, `abort`, `fatal`, `terminate`, `dead`, `destroy`, `panic`, `deprecated`, `obsolete`, `execute`, `exploit`, `conflict`, `revert`, `reject`, `critical` + +**Genuinely negative in SE** (pushed toward negative): +`hack`, `hacky`, `kludge`, `workaround`, `spaghetti`, `nightmare`, `technical debt` + +#### Comment length weighting + +Longer comments carry more weight in the sentiment score, as they tend to contain more meaningful sentiment signal. Weight is capped at 3x the average to prevent a single long comment from dominating. + +#### Trend analysis + +Sentiment trend is computed using **linear regression** (least squares) rather than a simple first-to-last comparison. This makes the trend robust to outliers and intermediate noise. + +--- + +## SE-domain lexicon + +The following terms have special handling in the sentiment scorer: + +### Technical terms neutralized (not negative) + +These terms are common in SE but trigger VADER's negative scoring: + +| Term | Context | +|---|---| +| `kill` | Process management | +| `abort` | Transaction/operation cancellation | +| `fatal` | Error severity levels | +| `terminate` | Process/thread lifecycle | +| `panic` | Error handling (Go, Rust) | +| `deprecated` | API lifecycle | +| `execute` | Command/query execution | +| `conflict` | Merge conflicts | +| `critical` | Severity levels | + +### Terms with genuine negative sentiment + +These terms indicate real frustration or code quality issues: + +| Term | Indication | +|---|---| +| `hack` / `hacky` | Quick-and-dirty solutions | +| `kludge` | Inelegant fixes | +| `workaround` | Avoiding root cause | +| `spaghetti` | Poor code structure | +| `nightmare` | Maintenance difficulty | +| `technical debt` | Accumulated shortcuts | + +--- + +## Use cases + +- **Team morale tracking**: Monitor sentiment trends over time. A sustained drop in sentiment may correlate with deadline pressure, technical debt accumulation, or team issues. +- **Code quality signals**: Negative sentiment spikes often correspond to periods of rushed development, workarounds, or "hack" implementations. +- **Post-mortem analysis**: After incidents, examine whether comment sentiment degraded in the lead-up to the problem. +- **Documentation quality**: Projects with predominantly neutral or positive comments tend to have better documentation culture. +- **Technical debt detection**: Comments containing terms like "hack", "workaround", "kludge" are flagged as genuinely negative in SE context. + +--- + +## Limitations + +- **Non-English scoring accuracy**: While 32 languages have lexicon coverage via the Chen-Skiena dataset, VADER's grammatical rules (negation handling, intensifiers, punctuation effects) are English-specific. Non-English comments get word-level sentiment from the lexicon but miss syntactic nuances. +- **UAST dependency**: Requires UAST parsing support for the target language. Files in unsupported languages are skipped. +- **Sarcasm**: Sarcasm, irony, and context-dependent meaning can mislead the classifier. Comments like "great, another production outage" may be scored as positive. +- **Comment extraction**: Only comments that appear in the UAST are analyzed. Preprocessor directives, build file comments, and non-code files are excluded. +- **CPU intensive**: The sentiment analyzer performs UAST parsing for every modified file in every commit. For large repositories, this is significantly slower than non-UAST analyzers. It benefits from parallel execution via the framework's worker pool. +- **Minimum length filter**: The default minimum of 20 characters filters out many short but potentially meaningful comments (e.g., `// FIXME`, `// HACK`). Lower `MinLength` to capture these, at the cost of more noise. + +--- + +## See also + +- [Sentiment reference](../analyzers/sentiment.md) — configuration keys and output schema. +- [Quick start](../getting-started/quickstart.md) — run history analysis. diff --git a/site/explanation/shotness.md b/site/explanation/shotness.md new file mode 100644 index 00000000..197860c1 --- /dev/null +++ b/site/explanation/shotness.md @@ -0,0 +1,113 @@ +# Understanding structural hotness + +This page explains the mental model behind the shotness analyzer: what +node-level change frequency and coupling mean, how coupling strength is +computed, and how the aggregation pipeline is structured. For configuration +keys and the output schema, see the +[Shotness reference](../analyzers/shotness.md). + +--- + +## What it measures + +The shotness analyzer measures **structural hotness** -- the change frequency of +individual code entities (functions, methods, classes) across Git history. +Unlike the couples analyzer which operates at file granularity, shotness +operates at the UAST node level, providing fine-grained co-change analysis. + +### Node change frequency + +For each code entity matched by the DSL query (functions by default), the analyzer counts how many commits modified lines within that entity's span. Entities that change frequently are "hot" -- they are likely volatile, complex, or central to the system. + +### Node co-change coupling + +When two code entities are modified in the same commit, their coupling counter is incremented. This produces a fine-grained coupling matrix at the function level, which is more precise than file-level coupling from the couples analyzer. + +### Coupling strength + +Coupling strength is normalized to a 0-1 scale using the formula: + +``` +strength(A, B) = co_changes(A, B) / max(co_changes(A, B), changes(A), changes(B)) +``` + +This ensures the result is always in [0, 1] and provides a meaningful confidence metric. A strength of 1.0 means functions always change together; 0.5 means they co-change half the time relative to the most active function. + +### Risk classification + +Nodes are classified into risk levels based on absolute change counts: + +| Risk Level | Threshold | Meaning | +|---|---|---| +| **HIGH** | ≥ 20 changes | Requires immediate attention and robust test coverage | +| **MEDIUM** | ≥ 10 changes | Should be monitored and potentially refactored | +| **LOW** | < 10 changes | Normal change frequency | + +### How it works + +For each commit: + +1. Parse the before and after versions of each changed file into UAST +2. Apply the `dsl_struct` query to select target nodes (e.g., functions) +3. Apply the `dsl_name` query to extract the name of each node +4. Map diff hunks to nodes using line-range overlaps +5. Emit a per-commit TC (Transient Commit result) with touched node deltas and coupling pairs + +After all commits are processed, the Aggregator accumulates TCs into a final report with sorted nodes and a sparse co-change matrix. + +### Architecture + +The shotness analyzer follows the **TC/Aggregator** pattern: + +- **Consume phase**: Per-commit processing builds working state (`nodes`, `files` maps for deletion/rename tracking) and emits a `TC{Data: *CommitData}` with node touch deltas and coupling pairs. +- **Aggregation phase**: The `Aggregator` accumulates node counts and coupling matrices from the TC stream. It supports disk-backed spilling via `SpillStore` for memory-bounded operation. +- **Serialization phase**: `SerializeTICKs()` converts aggregated tick data into the `Nodes`/`Counters` report consumed by `ComputeAllMetrics()` and plot generation. + +The `nodes` map remains in the analyzer as working state because `handleDeletion`, `handleInsertion`, `handleModification`, and `applyRename` read and mutate it during `Consume()`. The aggregator maintains its own separate accumulation of counts and couplings. + +--- + +## Use cases + +- **Function-level hotspot detection**: Find the most frequently changed functions in the codebase. These are the highest-risk points for bugs. +- **Fine-grained coupling analysis**: Discover which functions always change together. This reveals implicit dependencies that file-level coupling misses. +- **Refactoring prioritization**: Functions that are both hot (high change count) and coupled (always change with others) are the best refactoring candidates. +- **Architecture validation**: Functions from different packages that are highly coupled may indicate a leaking abstraction. +- **Test prioritization**: Focus testing resources on the hottest functions. + +--- + +## Interpreting results + +### Reading the coupling strength + +| Strength | Interpretation | +|---|---| +| 0.8 - 1.0 | Very tight coupling. Functions almost always change together. Consider merging or extracting shared logic. | +| 0.5 - 0.8 | Moderate coupling. There is a significant shared dependency. Review if coupling is intentional. | +| 0.2 - 0.5 | Loose coupling. Occasional co-changes, likely due to shared APIs or data structures. | +| < 0.2 | Minimal coupling. Co-changes are incidental. | + +### Actionable insights + +1. **High hotness + High coupling**: Core function that drives many changes. Candidate for splitting or stabilizing the interface. +2. **High hotness + Low coupling**: Frequently bugfixed isolated function. Needs better tests and potentially a redesign. +3. **Low hotness + High coupling**: Stable function that always changes with others. Check if coupling is necessary or indicates a design smell. + +--- + +## Limitations + +- **UAST required**: Only languages with UAST parser support are analyzed. Files in unsupported languages are skipped entirely. +- **CPU intensive**: The analyzer performs UAST parsing on both the before and after versions of every changed file in every commit. This makes it one of the most expensive analyzers. It benefits from parallel execution. +- **Name collisions**: If two functions in different files have the same name, they are tracked as distinct nodes (the file path is part of the key). However, if a file is renamed, the analyzer updates all associated nodes. +- **Shallow extraction within a file**: When multiple structural nodes in the same file share the same extracted name (e.g., nested functions with identical names), only one is tracked. The last one encountered wins. Qualified paths (e.g., `OuterClass.innerMethod`) are not built. +- **DSL limitations**: The DSL query must match nodes that have position information (`Pos` field) in the UAST. Nodes without position data cannot be mapped to diff hunks. +- **Large functions**: A change anywhere within a function's line range counts as a change to that function. Very large functions (hundreds of lines) will have inflated change counts. + +--- + +## See also + +- [Shotness reference](../analyzers/shotness.md) — configuration keys and output schema. +- [Quick start](../getting-started/quickstart.md) — run history analysis. diff --git a/site/explanation/typos.md b/site/explanation/typos.md new file mode 100644 index 00000000..6ed0ceae --- /dev/null +++ b/site/explanation/typos.md @@ -0,0 +1,73 @@ +# Understanding typo detection + +This page explains the mental model behind the typos analyzer: how it detects +typo-fix identifier pairs, what Levenshtein distance measures, and how the +aggregation pipeline is structured. For configuration keys and the output +schema, see the [Typos reference](../analyzers/typos.md). + +--- + +## What it measures + +The typos analyzer detects **typo-fix identifier pairs** from source code in +commit diffs using Levenshtein distance. It builds a dataset of probable typos +and their corrections by analyzing UAST identifier changes across Git history. + +### Typo-fix pair detection + +For each commit, the analyzer: + +1. Computes the diff between the old and new versions of each changed file +2. Identifies delete/insert hunk pairs of equal size (same number of lines) +3. Compares corresponding lines using **Levenshtein distance** +4. For line pairs within the distance threshold, extracts UAST identifiers from the old and new versions +5. If exactly one identifier changed between the two versions, it records a typo-fix pair + +### Levenshtein distance + +The Levenshtein distance is the minimum number of single-character edits (insertions, deletions, substitutions) needed to transform one string into another. A small distance between the old and new line suggests a typo fix rather than a semantic change. + +!!! info "Example" + - `recieve` to `receive` -- distance 2 (transposed characters) + - `lenght` to `length` -- distance 2 (transposed characters) + - `calcualte` to `calculate` -- distance 2 (transposed characters) + +--- + +## Architecture + +The typos analyzer follows the **TC/Aggregator pattern**: + +1. **Consume phase**: For each commit, `Consume()` computes diffs, identifies line pairs within Levenshtein distance, and extracts UAST identifier changes. Per-commit typos are returned as `TC{Data: []Typo}`. The analyzer retains no per-commit state; only the `lcontext` (Levenshtein context) is kept as working state. +2. **Aggregation phase**: A `typos.Aggregator` collects TCs into a `SliceSpillStore[Typo]`. `FlushTick()` deduplicates typos by `wrong|correct` key (keeping the first occurrence), returning a `TickData` with the unique set. +3. **Serialization phase**: `SerializeTICKs()` assembles all tick data into an `analyze.Report{"typos": allTypos}`, then delegates to `ComputeAllMetrics()` for JSON, YAML, binary, or HTML plot output. + +This separation enables streaming output, budget-aware memory spilling, and decoupled aggregation. + +--- + +## Use cases + +- **Typo dataset building**: Build a corpus of real-world typos and their corrections from your project's history. This can train spell-checking tools or IDE plugins. +- **Code quality auditing**: Identify patterns of common misspellings in your codebase to add to a linting dictionary. +- **API consistency**: Detect identifier typos that may cause confusion (e.g., `getUserByNmae` vs `getUserByName`). +- **Automated fix suggestions**: Use the typo dataset to build automated correction rules for CI pipelines. +- **Research**: Academic research on developer typo patterns and their prevalence across different languages. + +--- + +## Limitations + +- **UAST required**: Only languages with UAST parser support are analyzed. Identifiers in unsupported languages are not extracted. +- **Single-identifier changes only**: The analyzer only records a typo when exactly one identifier changes between the old and new lines. Multi-identifier changes are skipped to avoid false positives. +- **Equal-length hunks only**: Only delete/insert hunk pairs with the same number of lines are considered. A typo fix that also adds or removes lines will be missed. +- **False positives**: Intentional identifier renames with small Levenshtein distance (e.g., `idx` to `jdx`) will be reported as typos. +- **Deduplication**: Typo pairs are deduplicated by the `wrong|correct` key. The same typo fixed in multiple commits is reported only once. +- **CPU intensive**: Like all UAST-based analyzers, the typos analyzer parses both file versions for every changed file in every commit. + +--- + +## See also + +- [Typos reference](../analyzers/typos.md) — configuration keys and output schema. +- [Quick start](../getting-started/quickstart.md) — run history analysis. diff --git a/site/getting-started/first-analysis.md b/site/getting-started/first-analysis.md index 3f2f74b7..2993e33f 100644 --- a/site/getting-started/first-analysis.md +++ b/site/getting-started/first-analysis.md @@ -1,4 +1,4 @@ -# Your First Analysis +# Your first analysis This guide walks you through a complete analysis of a real Go project, explains the two analysis modes, shows you how to interpret results, and @@ -6,7 +6,7 @@ covers every output format. --- -## Two Modes of Analysis +## Two modes of analysis Codefang has two fundamentally different analysis engines. Understanding when to use each one is key to getting the most out of the tool. @@ -72,7 +72,7 @@ to use each one is key to getting the most out of the tool. --- -## Step-by-Step: Analyzing a Go Project +## Step-by-step: analyzing a Go project Let's analyze the Codefang repository itself. Clone it if you have not already: @@ -82,7 +82,7 @@ git clone https://github.com/Sumatoshi-tech/codefang.git cd codefang ``` -### Step 1 -- Static Complexity Analysis +### Step 1 -- static complexity analysis Start with a complexity scan to find the most complex functions: @@ -113,7 +113,7 @@ codefang run -a static/complexity --format text . candidates for refactoring. Cognitive complexity captures nesting depth and control-flow breaks that make code hard for humans to understand. -### Step 2 -- Halstead Metrics +### Step 2 -- Halstead metrics Measure the information-theoretic complexity of your modules: @@ -137,7 +137,7 @@ codefang run -a static/halstead --format text . code. The *estimated bugs* metric (Volume / 3000) gives a rough defect prediction. -### Step 3 -- Burndown History +### Step 3 -- burndown history Now switch to history analysis. The burndown analyzer shows how code ages: @@ -150,7 +150,7 @@ codefang run -a history/burndown --format yaml . repositories, use `--limit 1000` to cap the number of commits or `--since 24h` to restrict the time window. -### Step 4 -- Combine Static and History +### Step 4 -- combine static and history Run both engines in a single invocation: @@ -163,12 +163,12 @@ groups, runs each pipeline, and merges the output. --- -## Understanding Output Formats +## Understanding output formats The `--format` flag controls how results are rendered. Every analyzer supports the universal formats; some also support format-specific extras. -### Universal Formats +### Universal formats | Format | Flag | Description | Best for | |--------|------|-------------|----------| @@ -178,14 +178,14 @@ the universal formats; some also support format-specific extras. | **Binary** | `--format bin` | Compact binary encoding | Storage, conversion | | **Time Series** | `--format timeseries` | Merged JSON array keyed by commit | Trend analysis | -### Static-Only Formats +### Static-only formats | Format | Flag | Description | Best for | |--------|------|-------------|----------| | **Text** | `--format text` | Pretty-printed report with headers | Terminal reading | | **Compact** | `--format compact` | One line per analyzer per file | Grep-friendly scanning | -### Format Examples +### Format examples === "JSON (default)" @@ -289,7 +289,7 @@ the universal formats; some also support format-specific extras. --- -## Configuration File +## Configuration file Instead of passing flags every time, create a `.codefang.yaml` file in your repository root (or `$HOME`): @@ -337,7 +337,7 @@ checkpoint: --- -## Useful Flags for History Analysis +## Useful flags for history analysis | Flag | Example | Description | |------|---------|-------------| @@ -352,7 +352,7 @@ checkpoint: --- -## Converting Between Formats +## Converting between formats You can convert an existing report to a different format without re-running the analysis: @@ -375,7 +375,7 @@ codefang run --input report.bin --format plot > dashboard.html --- -## Next Steps +## Next steps You now know how to run both analysis engines, interpret their output, and control the output format. Continue exploring: diff --git a/site/getting-started/installation.md b/site/getting-started/installation.md index 7a4a01b6..5e74293a 100644 --- a/site/getting-started/installation.md +++ b/site/getting-started/installation.md @@ -23,7 +23,7 @@ Before you begin, make sure the following tools are available on your system: --- -## Install from Source (Recommended) +## Install from source (recommended) The fastest way to get both binaries: @@ -45,7 +45,7 @@ export PATH="$(go env GOPATH)/bin:$PATH" --- -## Build from Source +## Build from source For contributors or when you need a custom build: @@ -78,7 +78,7 @@ graph LR 3. **Go build** -- Both binaries are compiled with version/commit metadata injected via `-ldflags`. -### Building libgit2 Separately +### Building libgit2 separately If you only need to rebuild the native library: @@ -119,7 +119,7 @@ docker run --rm -v "$(pwd):/repo" codefang run -a history/burndown --format yaml --- -## Verify Installation +## Verify installation After installing, confirm both binaries are working: @@ -150,7 +150,7 @@ echo "$HOME/.local/bin" --- -## Platform Notes +## Platform notes ### Linux @@ -199,7 +199,7 @@ distribution and follow the Linux instructions above. --- -## Next Steps +## Next steps With both binaries installed, head to the [Quick Start](quickstart.md) guide to run your first analysis. diff --git a/site/getting-started/quickstart.md b/site/getting-started/quickstart.md index 714eceef..7b619eaa 100644 --- a/site/getting-started/quickstart.md +++ b/site/getting-started/quickstart.md @@ -1,4 +1,4 @@ -# Quick Start +# Quick start This page shows the fastest path to useful results. In five minutes you will parse code, run static analysis, and mine Git history. @@ -9,7 +9,7 @@ parse code, run static analysis, and mine Git history. --- -## 1. Parse Source Code into a UAST +## 1. Parse source code into a UAST Start by parsing a single file into a Universal Abstract Syntax Tree: @@ -53,7 +53,7 @@ cat main.go | uast parse - # read from stdin --- -## 2. Static Analysis +## 2. Static analysis Run structural analysis on a folder. The `codefang run` command with `static/*` analyzers walks the directory, parses every supported file, and @@ -86,7 +86,7 @@ codefang run -a static/complexity --format text . --- -## 3. History Analysis -- Burndown +## 3. History analysis -- burndown Analyze how code ages and survives over your repository's lifetime: @@ -118,7 +118,7 @@ column shows how many of those lines survive into subsequent windows. --- -## 4. Developer Analysis +## 4. Developer analysis See per-developer contribution statistics with a single commit snapshot: @@ -150,7 +150,7 @@ codefang run -a history/devs --head --format yaml . --- -## 5. Run Everything at Once +## 5. Run everything at once Use glob patterns to run all history analyzers (or all analyzers) in a single invocation: @@ -183,7 +183,7 @@ invocation: --- -## 6. Interactive HTML Plots +## 6. Interactive HTML plots Generate a self-contained HTML report with interactive charts: @@ -202,7 +202,7 @@ xdg-open burndown.html # Linux --- -## 7. Pipe into AI (MCP Integration) +## 7. Pipe into AI (MCP integration) Codefang ships a built-in **Model Context Protocol** server so AI agents can query analysis results programmatically: @@ -225,7 +225,7 @@ other MCP-compatible clients can connect to. See the --- -## What's Next? +## What's next? | Goal | Page | |------|------| diff --git a/site/glossary.md b/site/glossary.md new file mode 100644 index 00000000..f2f6cf40 --- /dev/null +++ b/site/glossary.md @@ -0,0 +1,161 @@ +--- +title: Glossary +description: Definitions of the project-specific terms Codefang introduces, for human readers and as a spell-check dictionary seed. +--- + + + +# Glossary + +This glossary defines the terms Codefang introduces or uses in a specific way. +Terms are listed alphabetically. + +## Abstract syntax tree (AST) + +A tree representation of source code that captures its structure rather than its +raw text. Codefang analyzes the AST so it can reason about meaning, not just +characters. + +## Analyzer + +A pluggable unit of analysis. Each analyzer carries an ID (for example +`static/complexity` or `history/burndown`) and produces metrics. Static analyzers +consume parsed code; history analyzers consume commit history. + +## Bare repository + +A Git repository without a working tree. Codefang's history mode opens both +normal and bare repositories via libgit2. + +## Blob + +The raw content of a file at a specific Git object hash. History analysis reads +blobs to compute diffs and parse code. + +## Blob cache + +A core plumbing analyzer (`BlobCacheAnalyzer`) that caches blob content so the +pipeline can re-read the same file content without fetching it from Git again. +It wraps an LRU cache. + +## Blob pipeline + +One of the three parallel stages the Coordinator runs. The blob pipeline loads +blob content from Git for the commits in a chunk. + +## Burndown + +A history analysis that tracks code age and survival: how many lines written in +a given period are still present in the codebase over time. The result is often +shown as a burndown chart. + +## Checkpoint + +A serialized snapshot of analyzer state written after each streaming chunk so an +interrupted run can recover instead of restarting from the first commit. + +## Chunk + +A memory-bounded batch of commits. The streaming pipeline splits a large commit +history into chunks so memory stays within budget. + +## Cohesion + +A static metric measuring how strongly the members of a class or module belong +together. Codefang reports cohesion using metrics such as LCOM. + +## Coordinator + +The component that orchestrates a worker pool across the three pipeline stages +(blob, diff, and UAST) for each chunk of commits. + +## Couples + +A history analysis that detects co-change coupling: files that tend to change +together in the same commits. Useful for risk-aware refactoring. + +## Diff pipeline + +One of the three parallel stages the Coordinator runs. The diff pipeline computes +per-commit tree and file diffs. + +## Halstead + +A family of static software-science metrics (volume, difficulty, effort, and +related measures) derived from the operators and operands in code. + +## Hibernate/boot cycle + +The state-management cycle of the streaming pipeline. Between chunks a +hibernatable analyzer serializes (hibernates) its state to a compact form, then +reboots to resume. This keeps memory bounded across a long history walk. + +## History mode + +The analysis mode that walks a Git repository's commit history and runs history +analyzers. Contrast with static mode, which reads source files from disk. + +## Identity detection + +A core plumbing step (`IdentityDetector`) that maps commit authors to canonical +identities so a single developer with multiple names or emails counts as one +person. + +## Leaf analyzer + +A history analyzer that consumes the output of the core plumbing analyzers — for +example burndown, couples, or devs. Leaf analyzers run after plumbing in the +pipeline. + +## Pipeline stage + +One of the three parallel work types the Coordinator dispatches per chunk: blob +loading, diff computation, and UAST parsing. + +## Plumbing + +The shared core analyzers that all history analyzers depend on (tree diff, blob +cache, identity detection, tick assignment, line stats, language detection, UAST +changes). Plumbing runs before any leaf analyzer. + +## Runner + +The component that coordinates the full history-analysis lifecycle: initialize, +process each chunk, then finalize with aggregators. It supports single-pass and +streaming execution strategies. + +## Sentiment + +A history analysis that scores the sentiment of code comments over time. + +## Shotness + +A history analysis that measures function-level change frequency — which +functions are touched most often (structural hotness). + +## Static mode + +The analysis mode that reads source files from disk, parses them into UASTs, and +runs static analyzers. It requires no Git history. + +## Streaming pipeline + +The execution strategy that processes a large history in memory-bounded chunks +with hibernate/boot cycles and optional double-buffered pipelining, planned by +the streaming planner. + +## Tick + +A discrete time index assigned to commits so analyses can build time series. The +`TicksSinceStart` plumbing analyzer assigns a tick to each commit relative to the +start of the history. + +## Typos + +A history analysis that detects likely identifier typos in diffs. + +## UAST (Universal Abstract Syntax Tree) + +A language-neutral abstract syntax tree. Codefang parses 60+ languages into a +single UAST shape via Tree-sitter, so one analyzer (for example complexity) works +across every supported language. diff --git a/site/guide/cli-reference.md b/site/guide/cli-reference.md index 3fe4f812..222aa3eb 100644 --- a/site/guide/cli-reference.md +++ b/site/guide/cli-reference.md @@ -1,4 +1,4 @@ -# CLI Reference +# CLI reference Codefang ships two binaries: **`codefang`**, the analyzer engine, and **`uast`**, the Universal Abstract Syntax Tree parser. This page is the exhaustive reference for @@ -22,7 +22,7 @@ codefang run [path] [flags] If `path` is omitted the current directory (or the value of `--path`) is used. -#### Analyzer Selection +#### Analyzer selection | Flag | Short | Type | Default | Description | |------|-------|------|---------|-------------| @@ -59,7 +59,7 @@ codefang run -a '*' . `history/quality`, `history/sentiment`, `history/shotness`, `history/typos` -#### Output Flags +#### Output flags | Flag | Short | Type | Default | Description | |------|-------|------|---------|-------------| @@ -79,7 +79,7 @@ codefang run -a 'history/*' --format plot . codefang run -a 'history/devs,history/sentiment' --format timeseries . ``` -#### Path & Input Flags +#### Path & input flags | Flag | Short | Type | Default | Description | |------|-------|------|---------|-------------| @@ -97,7 +97,7 @@ codefang run -a 'history/devs,history/sentiment' --format timeseries . codefang run -a 'history/*' --input report.bin --format plot ``` -#### Git History Flags +#### Git history flags | Flag | Type | Default | Description | |------|------|---------|-------------| @@ -128,7 +128,7 @@ codefang run -a history/couples --limit 500 . The burndown analyzer automatically enables `--first-parent` when selected. This is required for correct line-tracking across merge commits. -#### Language Filtering +#### Language filtering | Flag | Type | Default | Description | |------|------|---------|-------------| @@ -174,7 +174,7 @@ codefang run -a 'history/devs' --languages dockerfile . See `specs/optimize-lang/PROPOSAL.md` for the architecture and acceptance-gate numbers. -#### Vendor & Generated Exclusion +#### Vendor & generated exclusion By default, Codefang excludes **vendored dependencies** and **auto-generated files** from analysis. This matches the convention of every major @@ -225,7 +225,7 @@ See `specs/exclude-vendored/PROPOSAL.md` for the full cross-phase design and `specs/frds/FRD-20260419-exclude-vendored.md` for implementation details. -#### Pipeline Tuning Flags +#### Pipeline tuning flags | Flag | Type | Default | Description | |------|------|---------|-------------| @@ -248,14 +248,14 @@ codefang run -a 'static/*' --memory-budget 512MB . codefang run -a 'history/*' --blob-cache-size 2GB --diff-cache-size 50000 . ``` -#### GC Tuning Flags +#### GC tuning flags | Flag | Type | Default | Description | |------|------|---------|-------------| | `--gogc` | `int` | `0` | GC target percentage (`0` = Go default 100) | | `--ballast-size` | `string` | `"0"` | GC ballast allocation (`0` = disabled) | -#### Checkpoint & Resume Flags +#### Checkpoint & resume flags | Flag | Type | Default | Description | |------|------|---------|-------------| @@ -275,7 +275,7 @@ codefang run -a 'history/*' --resume . codefang run -a 'history/*' --clear-checkpoint . ``` -#### Profiling & Debug Flags +#### Profiling & debug flags | Flag | Type | Default | Description | |------|------|---------|-------------| diff --git a/site/guide/configuration.md b/site/guide/configuration.md index a3b9df4b..b739e1ac 100644 --- a/site/guide/configuration.md +++ b/site/guide/configuration.md @@ -6,9 +6,9 @@ and they merge with a well-defined priority order. --- -## Configuration Sources +## Configuration sources -### Search Order +### Search order When no explicit `--config` flag is provided, Codefang searches for a `.codefang.yaml` file in the following locations (first match wins): @@ -18,7 +18,7 @@ When no explicit `--config` flag is provided, Codefang searches for a | 1 | Current working directory (`./.codefang.yaml`) | | 2 | User home directory (`$HOME/.codefang.yaml`) | -### Merge Priority +### Merge priority When the same setting is specified in multiple sources, the highest-priority source wins: @@ -33,7 +33,7 @@ CLI flags > Environment variables > Config file > Compiled defaults `--workers 8`. The effective value is **8** because CLI flags have the highest priority. -### Explicit Config Path +### Explicit config path Use `--config` to point at a specific file: @@ -43,7 +43,7 @@ codefang run -a 'history/*' --config /etc/codefang/production.yaml . --- -## Environment Variables +## Environment variables All configuration keys can be set via environment variables using the `CODEFANG_` prefix. Nested keys use `_` as a separator. @@ -65,7 +65,7 @@ codefang run -a 'history/*' . --- -## Full Configuration Reference +## Full configuration reference Below is the complete `.codefang.yaml` file with all fields set to their compiled defaults: @@ -189,7 +189,7 @@ checkpoint: --- -## Section Reference +## Section reference ### `analyzers` @@ -432,7 +432,7 @@ Controls checkpoint and resume behavior for long-running history analyses. --- -## Minimal Examples +## Minimal examples === "CI / Headless" diff --git a/site/guide/data-analytics.md b/site/guide/data-analytics.md index 4e60676a..ac7f0ac5 100644 --- a/site/guide/data-analytics.md +++ b/site/guide/data-analytics.md @@ -1,4 +1,4 @@ -# Data Analytics & DWH Integration +# Data analytics & DWH integration Codefang produces richly structured JSON output designed for loading into columnar data warehouses (ClickHouse, Greenplum, BigQuery, Snowflake) and @@ -7,7 +7,7 @@ repository analysis to production dashboards. --- -## Quick Start +## Quick start Analyze a repository and produce DWH-ready output: @@ -24,7 +24,7 @@ codefang run --format json --per-file --limit 5000 /path/to/repo > report.json --- -## Output Format Selection +## Output format selection | Repo Size | Recommended Format | Reason | |-----------|-------------------|--------| @@ -33,7 +33,7 @@ codefang run --format json --per-file --limit 5000 /path/to/repo > report.json | 10K-50K files | `ndjson` | JSON gets multi-GB; NDJSON streams | | 50K+ files | `ndjson` + `--limit` | Bound history for practical runtimes | -### JSON Format +### JSON format ```bash codefang run --format json --per-file /repo > report.json @@ -60,7 +60,7 @@ Produces a single JSON object with versioned envelope: } ``` -### NDJSON Format +### NDJSON format ```bash codefang run --format ndjson --per-file /repo > report.ndjson @@ -89,7 +89,7 @@ cat report.ndjson | clickhouse-client --query "INSERT INTO codefang_raw FORMAT J --- -## Memory Budget +## Memory budget **Always set `--memory-budget`** for repos with history analysis. Without it, the streaming pipeline uses a conservative 2GB default that may OOM on large @@ -117,7 +117,7 @@ codefang run --format ndjson --per-file --memory-budget 8GB ~/sources/kubernetes --- -## Commit Limiting +## Commit limiting Use `--limit N` to analyze only the most recent N commits. This is useful for: @@ -138,7 +138,7 @@ codefang run --format json --per-file --memory-budget 8GB /repo > full.json --- -## Key Fields for Analytics +## Key fields for analytics Every function-level record includes fields designed for DWH joins and aggregation: @@ -156,7 +156,7 @@ aggregation: --- -## Schema Manifest +## Schema manifest Every analyzer section includes a `schema` field describing its output: @@ -196,7 +196,7 @@ for analyzer in data['analyzers']: --- -## Star Schema Design +## Star schema design ### Dimensions @@ -237,7 +237,7 @@ CREATE TABLE dim_tick ( ) ENGINE = MergeTree() ORDER BY (repo_id, tick); ``` -### Fact Tables +### Fact tables ```sql -- Static analysis facts (per-function grain) @@ -291,7 +291,7 @@ ORDER BY (repo_id, file1, file2); --- -## ETL Pipeline +## ETL pipeline ### Python (with dbt or standalone) @@ -326,7 +326,7 @@ coupling = analyzers['history/couples']['file_coupling'] # Each has: file1, file2, co_changes, coupling_strength ``` -### ClickHouse Direct Load +### ClickHouse direct load ```bash # Extract function complexity from NDJSON @@ -342,12 +342,12 @@ grep '"history/sentiment"' report.ndjson \ --- -## Recommended Analyzer Selection +## Recommended analyzer selection Not all 17 analyzers are needed for every use case. Select based on your dashboard needs: -### Code Quality Dashboard +### Code quality dashboard ```bash codefang run \ @@ -359,7 +359,7 @@ codefang run \ **Produces**: Function-level metrics, quality trend over time. **Row count**: ~200K functions + ~4K tick entries for a medium repo. -### Developer Analytics Dashboard +### Developer analytics dashboard ```bash codefang run \ @@ -370,7 +370,7 @@ codefang run \ **Produces**: Developer profiles, coupling networks, sentiment trends. **Row count**: ~500 developers + ~5K coupling pairs + ~4K ticks. -### File Health Dashboard +### File health dashboard ```bash codefang run \ @@ -382,7 +382,7 @@ codefang run \ **Produces**: Per-file complexity, churn hotspots, coupling networks. **Row count**: ~30K files + ~100K coupling pairs. -### Full Analysis (Everything) +### Full analysis (everything) ```bash codefang run --format ndjson --per-file --memory-budget 8GB /repo @@ -392,9 +392,9 @@ codefang run --format ndjson --per-file --memory-budget 8GB /repo --- -## Performance Tuning +## Performance tuning -### Static Analysis Workers +### Static analysis workers Control parallelism for the UAST parsing phase: @@ -405,7 +405,7 @@ codefang run --static-workers 16 --format json /repo More workers = faster static phase but higher peak memory. -### History Analysis +### History analysis The streaming pipeline auto-tunes chunk sizes based on `--memory-budget`. No manual tuning needed. Key parameters: @@ -429,13 +429,13 @@ codefang run --since 6m --first-parent --format json /repo --- -## Incremental Analysis & Checkpointing +## Incremental analysis & checkpointing Codefang supports two persistence mechanisms for long-running analysis: **incremental caching** (skip already-processed commits) and **checkpointing** (crash recovery). -### Incremental Cache +### Incremental cache The incremental cache stores analysis results keyed by repository root SHA and branch. On subsequent runs, only new commits since the last cached position @@ -474,7 +474,7 @@ rewrite), the cache is automatically invalidated. Each daily run only processes the new commits since yesterday, cutting analysis time from hours to minutes. -### Checkpointing (Crash Recovery) +### Checkpointing (crash recovery) For very long runs (e.g., full kubernetes at ~3 hours), checkpointing saves progress periodically so a crash doesn't lose all work. @@ -517,7 +517,7 @@ The checkpoint stores: For DWH pipelines, you want **both**: `--cache-dir` for incremental loads and `--checkpoint` for resilience. -### Production Pipeline Example +### Production pipeline example A daily cron job that incrementally analyzes a repository: @@ -552,7 +552,7 @@ cat "$OUTPUT_DIR/report-$(date +%Y%m%d).ndjson" \ | clickhouse-client --query "INSERT INTO codefang_raw FORMAT JSONEachRow" ``` -### Advanced Tuning for History Pipeline +### Advanced tuning for history pipeline Fine-tune the history streaming pipeline for specific hardware: @@ -582,7 +582,7 @@ codefang run \ --- -## Row Count Estimates +## Row count estimates Use these to plan DWH capacity: @@ -599,7 +599,7 @@ Compressed in ClickHouse: ~200MB. --- -## Materialized Views +## Materialized views Pre-aggregate for common dashboard queries: @@ -633,7 +633,7 @@ GROUP BY repo_id, week; ## Troubleshooting -### OOM Kills +### OOM kills **Symptom**: Process killed during history analysis. **Fix**: Set `--memory-budget` explicitly. @@ -646,7 +646,7 @@ free -h codefang run --memory-budget 4GB --format ndjson /repo ``` -### Empty History Analyzers +### Empty history analyzers Some analyzers require specific conditions: @@ -656,7 +656,7 @@ Some analyzers require specific conditions: | `history/imports` | Requires UAST-enabled pipeline mode | | `history/typos` | Requires UAST-enabled pipeline mode | -### Large File Coupling Tables +### Large file coupling tables `file_coupling` can produce millions of rows for large repos. Filter in your ETL: @@ -675,7 +675,7 @@ ORDER BY coupling_strength DESC LIMIT 1000; ``` -### Missing Language/Directory on Some Records +### Missing language/directory on some records The `language` and `directory` fields are populated by the UAST parser. If a file's language is not supported by the parser, these fields will be empty. diff --git a/site/guide/output-formats.md b/site/guide/output-formats.md index 3b908d02..b52e255b 100644 --- a/site/guide/output-formats.md +++ b/site/guide/output-formats.md @@ -1,4 +1,4 @@ -# Output Formats +# Output formats Codefang supports six output formats. Each is suited to a different use case, from human review to CI pipelines to interactive exploration. Select a format @@ -10,7 +10,7 @@ codefang run -a static/complexity --format text . --- -## Format Overview +## Format overview | Format | Flag Value | Content Type | Best For | |--------|-----------|--------------|----------| @@ -304,7 +304,7 @@ cat output.ndjson | clickhouse-client --query "INSERT INTO codefang FORMAT JSONE --- -## Time Series +## Time series **Flag:** `--format timeseries` @@ -444,7 +444,7 @@ open ./report/index.html --- -## Format Comparison +## Format comparison The following table summarizes which formats are available for which analyzer categories: @@ -477,7 +477,7 @@ categories: --- -## Cross-Format Conversion +## Cross-format conversion You can convert a previously generated report to a different format without re-running analysis. First, generate a binary (or JSON) report, then convert: diff --git a/site/how-to/analyze-a-bare-repo.md b/site/how-to/analyze-a-bare-repo.md new file mode 100644 index 00000000..ebf99259 --- /dev/null +++ b/site/how-to/analyze-a-bare-repo.md @@ -0,0 +1,70 @@ +# How to analyze a bare repository + + + +## Goal + +Run history analysis directly against a bare Git repository — such as a GitLab +backup, a Gitolite mirror, or a `git clone --bare` mirror — without first +checking out a working tree. + +## Prerequisites + +- `codefang` installed. +- A bare repository on disk (a directory ending in `.git` with no working tree). +- For GitLab backups, enough disk space to extract the `repositories/` tree. + +## Steps + +1. Point `codefang run` directly at the bare repository path. Codefang uses + libgit2, which opens bare and non-bare repositories transparently, so no + clone or checkout is required: + + ```bash + codefang run /data/backups/repositories/@hashed/ab/cd/abcdef.git \ + -a 'history/*' --format json --silent + ``` + +2. To scan a GitLab backup, extract the repositories tree from the tarball: + + ```bash + tar xf backup.tar -C /data/extract repositories/ + ``` + +3. Build a list of real project repos, skipping wiki and design repos: + + ```bash + find /data/extract/repositories -name "*.git" \ + ! -name "*.wiki.git" \ + ! -name "*.design.git" \ + -type d > /tmp/repo_paths.txt + ``` + +4. Run a single bare repo with a memory budget so the streaming pipeline stays + bounded on large histories: + + ```bash + codefang run /data/extract/repositories/@hashed/ab/cd/abcdef.git \ + -a 'history/*' --memory-budget 4GiB --workers 4 --format json --silent + ``` + +5. For periodic scans, analyze only new commits with `--since` rather than + re-walking the whole history each run: + + ```bash + codefang run /data/extract/repositories/@hashed/ab/cd/abcdef.git \ + -a 'history/*' --since 168h --format json --silent + ``` + +## Result + +You get the same history report you would get from a normal working clone — +burndown, developers, couples, and more — produced straight from the bare repo +with no checkout. A non-empty JSON report on stdout confirms the bare repo was +opened and analyzed. + +## See also + +- [Large-scale repository scanning](../operations/large-scale-scanning.md) +- [How to analyze a monorepo](analyze-a-monorepo.md) +- [CLI reference](../guide/cli-reference.md) diff --git a/site/how-to/analyze-a-monorepo.md b/site/how-to/analyze-a-monorepo.md new file mode 100644 index 00000000..869f3c47 --- /dev/null +++ b/site/how-to/analyze-a-monorepo.md @@ -0,0 +1,79 @@ +# How to analyze a monorepo + + + +## Goal + +Run Codefang against a single large repository that holds many languages and +services, and get a clean per-service report without drowning in vendored or +generated noise. + +## Prerequisites + +- `codefang` and `uast` installed. See [Installation](../getting-started/installation.md). +- A monorepo checked out locally, with full Git history (history analyzers need + every commit, not a shallow clone). +- `jq` if you want to post-process the JSON output. + +## Steps + +1. Run static analysis across the whole tree first. By default Codefang already + excludes vendored dependencies (`vendor/`, `node_modules/`, `third_party/`, + `testdata/`) and generated files, so the report covers your own code only: + + ```bash + codefang run -a 'static/*' --format json --silent . + ``` + +2. Narrow a noisy polyglot repo to the languages you care about. The + `--languages` filter applies to both the static and history phases and skips + non-matching files before they are parsed: + + ```bash + codefang run -a 'static/*' --languages go,typescript,python --format json . + ``` + +3. Scope a run to a single service directory instead of the whole repo by + passing its path (or `--path`): + + ```bash + codefang run -a 'static/complexity,static/cohesion' --format json ./services/api + ``` + +4. Add extra exclusion prefixes for tooling directories that Linguist's + heuristics do not catch, such as a Python virtualenv or a Rust target dir: + + ```bash + codefang run -a '*' --extra-excluded-prefixes '.venv/,target/,build/' --format json . + ``` + +5. Constrain memory on a large monorepo so the streaming pipeline auto-tunes + worker count and spill thresholds rather than consuming all available RAM: + + ```bash + codefang run -a 'history/*' --memory-budget 4GB --workers 4 --format json --silent . + ``` + +6. If you need only a subset of the history, limit the commit range with + `--since` or `--limit` to keep the run bounded: + + ```bash + codefang run -a 'history/devs' --since 2025-01-01 --format json . + ``` + +## Result + +You have a JSON report scoped to your own source code (vendored and generated +files excluded) and, when you narrow by `--languages` or path, scoped to the +service you care about. Confirm the scope by inspecting the report — for +example, count the files Codefang actually analyzed: + +```bash +codefang run -a 'static/complexity' --languages go --format json . | jq '.["static/complexity"] | length' +``` + +## See also + +- [How to run Codefang in CI](run-in-ci.md) +- [Large-scale repository scanning](../operations/large-scale-scanning.md) +- [CLI reference](../guide/cli-reference.md) diff --git a/site/how-to/attach-an-mcp-client.md b/site/how-to/attach-an-mcp-client.md new file mode 100644 index 00000000..27d8a040 --- /dev/null +++ b/site/how-to/attach-an-mcp-client.md @@ -0,0 +1,88 @@ +# How to attach an MCP client + + + +## Goal + +Connect an AI agent to Codefang over the Model Context Protocol (MCP) so the +agent can run code analysis as a tool. + +## Prerequisites + +- `codefang` installed and on your `PATH`. +- An MCP-compatible client (for example, Claude Code or another agent that + launches MCP servers over stdio). +- For the `codefang_history` tool, an absolute path to a local Git repository. + +## Steps + +1. Confirm the MCP server starts. It communicates over stdio using JSON-RPC, the + standard transport for local MCP tool use: + + ```bash + codefang mcp + ``` + + Press `Ctrl-C` to stop it — the agent will launch its own copy. + +2. Register Codefang as an MCP server in your client. For Claude Code, add it to + `~/.claude/settings.json` (user scope) or `.claude/settings.json` (project + scope): + + ```json + { + "mcpServers": { + "codefang": { + "command": "codefang", + "args": ["mcp"], + "env": {} + } + } + } + ``` + +3. Restart the client so it picks up the new server. The agent then discovers + three tools via the standard MCP `tools/list` method: `codefang_analyze` + (static analysis on inline code), `uast_parse` (parse source into a UAST), + and `codefang_history` (history analysis on a local repo). + +4. Ask the agent to run history analysis. The `codefang_history` tool requires + an absolute `repo_path`; a relative path is rejected: + + ```json + { + "name": "codefang_history", + "arguments": { + "repo_path": "/home/user/projects/myapp", + "analyzers": ["burndown", "devs"], + "limit": 500 + } + } + ``` + +5. If a call misbehaves, enable debug logging to stderr by adding the `--debug` + argument to the server registration: + + ```json + { + "mcpServers": { + "codefang": { + "command": "codefang", + "args": ["mcp", "--debug"], + "env": {} + } + } + } + ``` + +## Result + +The agent lists the three Codefang tools and returns structured analysis when it +calls them. A successful `codefang_analyze` or `codefang_history` call that comes +back without `isError: true` confirms the client is attached. + +## See also + +- [MCP server reference](../integrations/mcp.md) +- [AI agents](../integrations/ai-agents.md) +- [How to read a burndown chart](read-a-burndown-chart.md) diff --git a/site/how-to/read-a-burndown-chart.md b/site/how-to/read-a-burndown-chart.md new file mode 100644 index 00000000..aae95923 --- /dev/null +++ b/site/how-to/read-a-burndown-chart.md @@ -0,0 +1,79 @@ +# How to read a burndown chart + + + +## Goal + +Generate a code-survival burndown for a repository and interpret the result so +you can see how much code from each era still survives at the latest sample. + +## Prerequisites + +- `codefang` installed and a Git repository checked out with full history. +- A web browser to view the interactive HTML chart. +- `jq` if you want to inspect the raw matrix. + +## Steps + +1. Run the burndown analyzer and write the YAML output so you can read the raw + matrix: + + ```bash + codefang run -a history/burndown --format yaml . + ``` + +2. Read the `project` matrix. Each row is a sampling point in time; each value + in a row is the number of lines, created in a given age band, that still + survive at that sample. As you read down the rows, watch a band's values + shrink — that is code being rewritten or deleted over time: + + ```yaml + burndown: + granularity: 30 + sampling: 30 + project: + - [0, 1200, 1100, 980, 920, 870] + - [0, 0, 320, 290, 270, 250] + ticks: + - "2024-01-15" + - "2024-02-14" + ``` + +3. Use `ticks` to map matrix positions back to calendar dates. The `granularity` + key is the width of each age band (in ticks) and `sampling` is how often a + snapshot is taken; the default tick is a 24-hour window. + +4. Add a per-developer breakdown to see whose code survives, by enabling + developer tracking: + + ```bash + codefang run -a history/burndown --burndown-people --format yaml . + ``` + +5. Generate the interactive stacked-area chart and open it in a browser. This is + the visual form of the same matrix: + + ```bash + codefang run -a history/burndown --format plot . > burndown.html + xdg-open burndown.html # Linux + open burndown.html # macOS + ``` + +6. Extract just the tick axis for scripting or a custom plot: + + ```bash + codefang run -a history/burndown --format json . | jq '.burndown.ticks' + ``` + +## Result + +You can point at any value in the chart or matrix and state what it means: "this +many lines, written around this date, still survive at this later date." A +declining survival curve signals churn; a flat top band signals stable, +long-lived code. + +## See also + +- [Understanding code burndown](../explanation/burndown.md) — the mental model. +- [Burndown reference](../analyzers/burndown.md) — configuration keys and output schema. +- [How to run Codefang in CI](run-in-ci.md) diff --git a/site/how-to/run-in-ci.md b/site/how-to/run-in-ci.md new file mode 100644 index 00000000..9399e1e1 --- /dev/null +++ b/site/how-to/run-in-ci.md @@ -0,0 +1,95 @@ +# How to run Codefang in CI + + + +## Goal + +Add Codefang to a continuous-integration pipeline so every push and pull request +runs code analysis automatically, and optionally fail the build when analysis +detects issues. + +## Prerequisites + +- A repository hosted on GitHub (this guide uses the published GitHub Action). +- Permission to add workflow files under `.github/workflows/`. +- For history analyzers, the checkout step must fetch full history + (`fetch-depth: 0`); a shallow clone only sees the most recent commits. + +## Steps + +1. Create `.github/workflows/codefang.yml` and add the action with the + analyzers you want. Static analysis works on a shallow clone: + + ```yaml + name: Code Quality + on: [push, pull_request] + + jobs: + analyze: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + + - name: Run Codefang + id: codefang + uses: Sumatoshi-tech/codefang@main + with: + analyzers: "static/*" + format: "json" + ``` + +2. Turn the run into a quality gate by setting `fail-on-error: "true"`. The step + fails the workflow when analysis detects issues: + + ```yaml + - name: Run Codefang + uses: Sumatoshi-tech/codefang@main + with: + analyzers: "static/complexity,static/comments" + fail-on-error: "true" + ``` + +3. For history analyzers, fetch the full Git history in the checkout step, + otherwise the analysis only sees the shallow clone: + + ```yaml + - uses: actions/checkout@v4 + with: + fetch-depth: 0 + + - name: Run Codefang History + uses: Sumatoshi-tech/codefang@main + with: + analyzers: "history/burndown,history/devs" + format: "json" + ``` + +4. Read the action outputs in a later step. `pass` is `true` when analysis + completed without errors, and `report` holds the full report content: + + ```yaml + - name: Check results + run: | + echo "Pass: ${{ steps.codefang.outputs.pass }}" + echo "${{ steps.codefang.outputs.report }}" + ``` + +5. If a large repository runs out of memory in CI, constrain it with a memory + budget by running the binary directly inside the job, or in Docker: + + ```bash + docker run --rm -v "$(pwd):/workspace:ro" \ + codefang run -a 'history/*' --memory-budget 2GiB --format json --silent /workspace + ``` + +## Result + +Open the Actions tab after a push. The Codefang job runs, prints the report, and +— when `fail-on-error` is set — marks the check red if analysis detected issues. +A green check with a populated `report` output confirms the pipeline works. + +## See also + +- [Docker & GitHub Actions reference](../integrations/docker-and-actions.md) +- [How to analyze a monorepo](analyze-a-monorepo.md) +- [Output formats](../guide/output-formats.md) diff --git a/site/index.md b/site/index.md index e55ad01f..f3e571dc 100644 --- a/site/index.md +++ b/site/index.md @@ -53,7 +53,7 @@ analysis of entire repositories. One tool, one config, complete understanding. --- -## Key Metrics +## Key metrics
@@ -96,7 +96,7 @@ analysis of entire repositories. One tool, one config, complete understanding. --- -## Quick Install +## Quick install === "Go Install" @@ -130,7 +130,7 @@ uast --version --- -## Quick Example +## Quick example ``` $ codefang run -a static/complexity,static/halstead --format text . @@ -163,9 +163,9 @@ codefang (v2): --- -## Analyzers at a Glance +## Analyzers at a glance -### Static Analyzers +### Static analyzers | Analyzer | Description | Key Metric | |----------|-------------|------------| @@ -175,7 +175,7 @@ codefang (v2): | `comments` | Comment density and documentation ratios | Density % | | `imports` | Import/dependency graph analysis | Coupling score | -### History Analyzers +### History analyzers | Analyzer | Description | Key Metric | |----------|-------------|------------| @@ -190,7 +190,7 @@ codefang (v2): --- -## Explore the Docs +## Explore the docs
diff --git a/site/integrations/ai-agents.md b/site/integrations/ai-agents.md index 0bf4fd64..c7589588 100644 --- a/site/integrations/ai-agents.md +++ b/site/integrations/ai-agents.md @@ -3,7 +3,7 @@ title: AI Agent Integration description: Patterns and workflows for using Codefang with AI coding agents for self-correcting generation, architectural context, and risk-aware refactoring. --- -# Using Codefang with AI Agents +# Using Codefang with AI agents Codefang's MCP server and CLI tools enable powerful integration with AI coding agents. Rather than treating code analysis as a one-off check, agents can use @@ -14,7 +14,7 @@ This guide covers proven patterns for agent-assisted development with Codefang. --- -## Integration Methods +## Integration methods | Method | Best For | Setup | |--------|----------|-------| @@ -28,12 +28,12 @@ See the [MCP Server guide](mcp.md) for setup instructions. --- -## Pattern 1: Self-Correcting Code Generation +## Pattern 1: self-correcting code generation **Goal**: Generate code, check its quality, and automatically fix issues in a feedback loop. -### How It Works +### How it works ```mermaid flowchart TD @@ -49,7 +49,7 @@ flowchart TD CHECK -->|Yes| DONE ``` -### Agent Workflow +### Agent workflow 1. The agent generates code for the requested feature. 2. It calls `codefang_analyze` with the generated code. @@ -60,7 +60,7 @@ flowchart TD 4. If any metric is out of bounds, the agent refactors and re-analyzes. 5. The loop repeats until all metrics pass. -### Example Agent Prompt +### Example agent prompt ``` Generate a Go function that parses CSV files with configurable delimiters @@ -69,7 +69,7 @@ check complexity. If any function has cyclomatic complexity > 10, refactor it into smaller functions and re-check. ``` -### What the Agent Sees +### What the agent sees ```json // Agent calls codefang_analyze with the generated code @@ -88,11 +88,11 @@ exactly which functions need attention. --- -## Pattern 2: Architectural Context Injection +## Pattern 2: architectural context injection **Goal**: Help the agent understand existing code structure before making changes. -### How It Works +### How it works Before modifying a codebase, the agent parses key files to understand the architecture -- function signatures, class hierarchies, import graphs, and @@ -109,7 +109,7 @@ flowchart LR TASK --> PARSE --> CONTEXT --> PLAN --> IMPLEMENT ``` -### Agent Workflow +### Agent workflow 1. Agent receives a task like "Add caching to the database layer." 2. It calls `uast_parse` on the relevant source files. @@ -121,7 +121,7 @@ flowchart LR 4. With this structural understanding, it plans and implements changes that are consistent with the existing codebase. -### Example: Understanding Module Structure +### Example: understanding module structure ```json { @@ -139,11 +139,11 @@ surface without reading every line. --- -## Pattern 3: Risk-Aware Refactoring +## Pattern 3: risk-aware refactoring **Goal**: Understand file coupling and change risk before refactoring. -### How It Works +### How it works Before making structural changes, the agent runs `codefang_history` with the `couples` analyzer to understand which files tend to change together. This @@ -164,7 +164,7 @@ flowchart TD PROCEED --> VERIFY ``` -### Agent Workflow +### Agent workflow 1. Agent plans to refactor `pkg/database/queries.go`. 2. It calls `codefang_history` with the `couples` analyzer. @@ -191,18 +191,18 @@ to identify blast radius. --- -## Pattern 4: Style Enforcement +## Pattern 4: style enforcement **Goal**: Establish and enforce code style baselines using cohesion and comment density metrics. -### How It Works +### How it works 1. Analyze the existing codebase to establish baseline metrics. 2. When generating new code, ensure it meets or exceeds the baseline. 3. Flag deviations for review. -### Agent Workflow +### Agent workflow ```mermaid flowchart LR @@ -218,7 +218,7 @@ flowchart LR PASS -->|Yes| ACCEPT ``` -### Baseline Metrics +### Baseline metrics | Metric | Source | Typical Baseline | |--------|--------|-----------------| @@ -227,7 +227,7 @@ flowchart LR | Cohesion score | `static/cohesion` | > 0.5 (LCOM4) | | Halstead difficulty | `static/halstead` | < 30 per function | -### Example Agent Prompt +### Example agent prompt ``` Before writing new code for this module, analyze the existing files in @@ -238,16 +238,16 @@ matching comment density, complexity levels, and cohesion scores. --- -## Pattern 5: Developer Onboarding Context +## Pattern 5: developer onboarding context **Goal**: Help agents understand team dynamics and code ownership. -### How It Works +### How it works The `devs` and `file-history` analyzers reveal who owns which parts of the codebase, how active different areas are, and what languages are used where. -### Agent Workflow +### Agent workflow 1. Agent needs to modify an unfamiliar module. 2. It calls `codefang_history` with `devs` and `file-history` analyzers. @@ -273,12 +273,12 @@ codebase, how active different areas are, and what languages are used where. --- -## Pattern 6: Anomaly Detection in Generated Code +## Pattern 6: anomaly detection in generated code **Goal**: Use temporal anomaly detection to flag unusual patterns in ongoing development. -### Agent Workflow +### Agent workflow 1. After a sprint of AI-assisted development, run `codefang_history` with the `anomaly` analyzer. @@ -299,9 +299,9 @@ development. --- -## MCP Integration Best Practices +## MCP integration best practices -### Tool Selection +### Tool selection | Task | Recommended Tool | Analyzers | |------|-----------------|-----------| @@ -314,7 +314,7 @@ development. | Find function hotspots | `codefang_history` | `shotness` | | Audit imports | `codefang_analyze` | `imports` | -### Performance Tips +### Performance tips - **Limit history depth**: Use the `limit` parameter (default: 1000) to avoid analyzing the entire history when recent context is sufficient. @@ -325,7 +325,7 @@ development. - **Cache UAST results**: If analyzing multiple aspects of the same file, parse once and reuse the UAST tree. -### Error Recovery +### Error recovery Agents should handle common errors gracefully: @@ -340,7 +340,7 @@ Agents should handle common errors gracefully: --- -## End-to-End Example: Feature Implementation +## End-to-end example: feature implementation Here is a complete agent workflow for implementing a new feature: diff --git a/site/integrations/docker-and-actions.md b/site/integrations/docker-and-actions.md index 7f0b8541..462e1b2d 100644 --- a/site/integrations/docker-and-actions.md +++ b/site/integrations/docker-and-actions.md @@ -3,7 +3,7 @@ title: Docker & GitHub Actions description: Running Codefang in Docker containers and GitHub Actions CI pipelines for automated code quality checks. --- -# Docker & GitHub Actions +# Docker & GitHub actions Codefang ships a multi-stage Dockerfile and a GitHub Actions `action.yml` for automated code quality checks in CI pipelines. @@ -12,7 +12,7 @@ for automated code quality checks in CI pipelines. ## Docker -### Building the Image +### Building the image Using the Makefile: @@ -27,7 +27,7 @@ Or directly with Docker: docker build -t codefang . ``` -#### Build Arguments +#### Build arguments Embed version metadata into the binary at build time: @@ -39,9 +39,9 @@ docker build \ -t codefang:v1.0.0 . ``` -### Running in a Container +### Running in a container -#### Static Analysis +#### Static analysis Analyze a local directory (read-only mount): @@ -51,7 +51,7 @@ docker run --rm \ codefang run -a static/* --format json --silent /workspace ``` -#### History Analysis +#### History analysis Analyze a Git repository's commit history: @@ -61,7 +61,7 @@ docker run --rm \ codefang run -a history/burndown --format json --silent /workspace ``` -#### Combined Analysis +#### Combined analysis Run both static and history analyzers in one pass: @@ -71,7 +71,7 @@ docker run --rm \ codefang run -a static/*,history/devs --format json --silent /workspace ``` -#### Custom Configuration +#### Custom configuration Mount a config file into the container: @@ -83,7 +83,7 @@ docker run --rm \ --format json --silent /workspace ``` -#### With Observability +#### With observability Export traces to an OTLP collector: @@ -106,7 +106,7 @@ docker run --rm \ codefang parse /workspace/main.go ``` -### Testing the Docker Image +### Testing the Docker image ```bash make docker-test @@ -115,7 +115,7 @@ make docker-test This builds the image and runs a static complexity analysis on the Codefang source tree inside the container. -### Image Details +### Image details | Property | Value | |----------|-------| @@ -135,9 +135,9 @@ source tree inside the container. --- -## GitHub Actions +## GitHub actions -### Quick Start +### Quick start Add Codefang to any workflow: @@ -170,7 +170,7 @@ jobs: History analyzers require full Git history. Set `fetch-depth: 0` in the checkout step, or history analysis will only see the shallow clone commits. -### Action Inputs +### Action inputs | Input | Description | Default | |-------|-------------|---------| @@ -180,7 +180,7 @@ jobs: | `format` | Output format: `json`, `text`, `compact`, `yaml`, `timeseries` | `json` | | `fail-on-error` | Fail the workflow step if analysis detects issues | `false` | -### Action Outputs +### Action outputs | Output | Description | |--------|-------------| @@ -189,9 +189,9 @@ jobs: --- -### Example Workflows +### Example workflows -#### Static Analysis with Quality Gate +#### Static analysis with quality gate Fail the CI pipeline if static analysis detects issues: @@ -212,7 +212,7 @@ jobs: fail-on-error: "true" ``` -#### History Analysis on Push +#### History analysis on push Run history analysis on every push to main: @@ -244,7 +244,7 @@ jobs: path: report.json ``` -#### Post Results as PR Comment +#### Post results as PR comment Annotate pull requests with analysis results: @@ -282,7 +282,7 @@ jobs: }) ``` -#### All Analyzers with Custom Config +#### All analyzers with custom config Run the full analyzer suite with a project config file: @@ -308,7 +308,7 @@ jobs: format: "json" ``` -#### Complexity Trend Tracking +#### Complexity trend tracking Track complexity over time by uploading JSON reports as artifacts: @@ -342,7 +342,7 @@ jobs: retention-days: 90 ``` -#### Scheduled Fleet Scan +#### Scheduled fleet scan Run nightly analysis across multiple paths: @@ -375,7 +375,7 @@ jobs: ## Troubleshooting -### Common Issues +### Common issues !!! failure "Error: repository path does not exist" The checkout step did not run, or the `path` input points to a @@ -390,7 +390,7 @@ jobs: memory usage. The streaming pipeline will automatically chunk the commit history. -### Self-Test Workflow +### Self-test workflow The repository includes `.github/workflows/action-test.yml` which validates the Docker build and action functionality on every change to the Dockerfile, diff --git a/site/integrations/mcp.md b/site/integrations/mcp.md index d0a01986..1fa9bf8e 100644 --- a/site/integrations/mcp.md +++ b/site/integrations/mcp.md @@ -3,7 +3,7 @@ title: MCP Server description: Model Context Protocol server for AI agent integration, exposing Codefang analysis tools over stdio transport. --- -# MCP Server +# MCP server Codefang includes a built-in [Model Context Protocol](https://modelcontextprotocol.io/) (MCP) server that exposes its analysis capabilities as tools that AI agents @@ -26,7 +26,7 @@ MCP-compatible agent can use without manual setup. --- -## Quick Start +## Quick start Start the MCP server: @@ -38,7 +38,7 @@ The server communicates over **stdio** (stdin/stdout) using JSON-RPC, which is the standard MCP transport for local tool use. Agents connect by launching the process and reading/writing to its stdio streams. -### Debug Mode +### Debug mode Enable debug logging to stderr for troubleshooting: @@ -53,7 +53,7 @@ Debug mode enables: --- -## Available Tools +## Available tools The MCP server exposes three tools: @@ -65,7 +65,7 @@ The MCP server exposes three tools: --- -## Tool Reference +## Tool reference ### `codefang_analyze` @@ -80,7 +80,7 @@ metrics. | `language` | string | **Yes** | -- | Programming language (e.g., `go`, `python`, `javascript`) | | `analyzers` | string[] | No | all | List of analyzer names to run | -#### Available Analyzers +#### Available analyzers | Analyzer | Description | |----------|-------------| @@ -148,7 +148,7 @@ returns analyzer-specific results. | `since` | string | No | -- | Only analyze commits after this time (e.g., `24h`, `2024-01-01`, RFC3339) | | `first_parent` | boolean | No | `false` | Follow only the first parent of merge commits | -#### Available Analyzers +#### Available analyzers | Analyzer | Description | |----------|-------------| @@ -177,9 +177,9 @@ returns analyzer-specific results. --- -## Agent Configuration +## Agent configuration -### Claude Code +### Claude code Add Codefang as an MCP server in your Claude Code settings: @@ -241,7 +241,7 @@ Add Codefang as an MCP server in your Claude Code settings: } ``` -### Generic MCP Client +### Generic MCP client Any MCP-compatible client can connect by launching the process: @@ -255,7 +255,7 @@ Clients call `tools/call` with the tool name and arguments. --- -## Error Handling +## Error handling The MCP server returns structured errors for common failure modes: diff --git a/site/operations/large-scale-scanning.md b/site/operations/large-scale-scanning.md index 9829a068..5abaf288 100644 --- a/site/operations/large-scale-scanning.md +++ b/site/operations/large-scale-scanning.md @@ -1,4 +1,4 @@ -# Large-Scale Repository Scanning +# Large-scale repository scanning This guide covers using Codefang to periodically scan large heterogeneous codebases — thousands of repositories — and feed results into a data warehouse @@ -26,7 +26,7 @@ graph LR The streaming pipeline, checkpointing, and bounded-memory execution make each individual run safe on large repositories without manual tuning. -## Bare Repository Support +## Bare repository support Codefang uses libgit2 which opens both normal and bare repositories transparently. No cloning is required. @@ -43,7 +43,7 @@ This is relevant for: - **Gitolite mirrors** — bare repos by convention - **`git clone --bare`** mirrors -### GitLab Backup Layout +### GitLab backup layout GitLab backup tarballs contain bare repositories at predictable paths: @@ -69,7 +69,7 @@ find /data/extract/repositories -name "*.git" \ -type d > /tmp/repo_paths.txt ``` -## Output Format for DWH Ingestion +## Output format for DWH ingestion Use `--format json` for structured output suitable for data warehouse loading: @@ -81,7 +81,7 @@ codefang run /path/to/repo -a history/* --format json --silent The `--silent` flag suppresses progress output on stderr, keeping stdout clean for piping. -### Adding Metadata +### Adding metadata Wrap the output with repository metadata using `jq` before uploading: @@ -91,7 +91,7 @@ codefang run "$REPO_PATH" -a history/* --format json --silent \ '{repo: $repo, scan_date: $date, results: .}' ``` -### TimeSeries Format +### TimeSeries format For time-series analytics, use `--format timeseries` to merge all analyzer outputs into a single chronologically-ordered JSON array: @@ -103,7 +103,7 @@ codefang run /path/to/repo -a history/devs,history/burndown \ ## Orchestration -### GNU Parallel (Simple) +### GNU parallel (simple) For a flat list of repository paths: @@ -128,7 +128,7 @@ cat "$REPOS" | parallel -j 8 --joblog /tmp/codefang-jobs.log ' This produces `s3://analytics/codefang/2025-01-15/repo-name.json`, partitioned by date. -### Kubernetes Jobs +### Kubernetes jobs For large fleets (thousands of repos), use Kubernetes Jobs or Argo Workflows: @@ -183,7 +183,7 @@ spec: - **Parallelism**: 50-100 concurrent pods is typical for 60k repositories - **Scheduling**: use a CronJob or Argo CronWorkflow for periodic scans -## Memory Budget and Streaming +## Memory budget and streaming The `--memory-budget` flag controls how Codefang splits a repository's commit history into chunks. See [Streaming Pipeline](../architecture/streaming-pipeline.md) @@ -198,7 +198,7 @@ for details. When unset, the budget defaults to 50% of system memory (capped at 4 GiB). -## Incremental Scanning with `--since` +## Incremental scanning with `--since` For periodic scans, use `--since` to analyze only new commits since the last run: @@ -220,7 +220,7 @@ codefang run /path/to/repo -a history/* --format json --since 2025-01-01T00:00:0 | Date only | `2025-01-01` | Midnight UTC on that date | | RFC3339 | `2025-01-01T00:00:00Z` | Exact timestamp | -## Checkpointing and Crash Recovery +## Checkpointing and crash recovery Checkpointing is enabled by default. After each fully processed chunk, the pipeline saves analyzer state to disk. If a run is interrupted (OOM kill, pod @@ -239,7 +239,7 @@ codefang run /path/to/repo \ !!! tip "Kubernetes" Mount the checkpoint directory on a PVC to survive pod restarts. -## DWH Loading +## DWH loading ### Amazon Athena @@ -286,7 +286,7 @@ devs_df = df.select("repo", explode("results.devs.authors").alias("author")) devs_df.write.partitionBy("scan_date").parquet("s3://warehouse/codefang/devs/") ``` -## Static Analysis Memory Monitoring +## Static analysis memory monitoring The static analysis phase logs progress events at regular intervals (every 1000 files). By default, only phase and file count are shown: @@ -340,7 +340,7 @@ codefang run /path/to/repo -a history/* --format json --silent | `codefang.errors.total` | Track failure rate across the fleet | | `codefang.cache.hits` / `codefang.cache.misses` | Tune cache sizes | -## Sizing Estimates +## Sizing estimates | Parameter | Typical Value | |-----------|---------------| diff --git a/site/operations/observability.md b/site/operations/observability.md index 2a573d6b..ab987536 100644 --- a/site/operations/observability.md +++ b/site/operations/observability.md @@ -12,7 +12,7 @@ and production-ready when enabled. --- -## Quick Start +## Quick start By default, observability uses no-op providers with zero export overhead. To enable telemetry export, set the OTLP endpoint: @@ -32,7 +32,7 @@ This forces 100% trace sampling and prints the `trace_id` on completion. --- -## Local Development Stack +## Local development stack A pre-configured OTel stack (Jaeger + OTel Collector + Prometheus) is available via Docker Compose: @@ -53,7 +53,7 @@ make otel-down # Stop the stack ## Configuration -### Environment Variables +### Environment variables | Variable | Description | Default | |----------|-------------|---------| @@ -63,13 +63,13 @@ make otel-down # Stop the stack | `OTEL_TRACES_SAMPLER` | Sampler type (see [Sampling](#sampling)) | (uses config) | | `OTEL_TRACES_SAMPLER_ARG` | Sampler argument (ratio for ratio-based samplers) | `1.0` | -### CLI Flags +### CLI flags | Flag | Description | |------|-------------| | `--debug-trace` | Force 100% sampling, enable debug logging | -### MCP Mode +### MCP mode The MCP server (`codefang mcp`) uses JSON-structured logs on stderr by default. When `--debug` is enabled, debug-level logging and full trace sampling are @@ -79,7 +79,7 @@ activated: codefang mcp --debug ``` -### No-Op Mode +### No-op mode When `OTEL_EXPORTER_OTLP_ENDPOINT` is empty (the default), the SDK returns no-op tracer and meter providers: @@ -90,7 +90,7 @@ no-op tracer and meter providers: --- -## Span Hierarchy +## Span hierarchy The entire analysis pipeline is instrumented with hierarchical spans: @@ -131,7 +131,7 @@ codefang.run (CLI entry point) +-- codefang.uast.changes ``` -### Span Attributes +### Span attributes | Span | Key Attributes | |------|---------------| @@ -161,11 +161,11 @@ Pipeline cache statistics are also recorded on the analysis span: --- -## RED Metrics +## RED metrics When connected to an OTLP collector, the following metrics are exported: -### Core RED Metrics +### Core RED metrics | Metric | Type | Unit | Description | |--------|------|------|-------------| @@ -174,7 +174,7 @@ When connected to an OTLP collector, the following metrics are exported: | `codefang.errors.total` | Counter | `{error}` | Total number of errors (labeled by `op`) | | `codefang.inflight.requests` | UpDownCounter | `{request}` | Currently in-flight requests (labeled by `op`) | -### Analysis Metrics +### Analysis metrics | Metric | Type | Unit | Description | |--------|------|------|-------------| @@ -184,7 +184,7 @@ When connected to an OTLP collector, the following metrics are exported: | `codefang.analysis.cache.hits.total` | Counter | `{hit}` | Cache hits (labeled by `cache`: `blob` or `diff`) | | `codefang.analysis.cache.misses.total` | Counter | `{miss}` | Cache misses (labeled by `cache`: `blob` or `diff`) | -### Histogram Buckets +### Histogram buckets Duration histograms use these bucket boundaries (in seconds), covering sub-second static checks through multi-minute history pipelines: @@ -204,7 +204,7 @@ Trace sampling is configurable with the following precedence (highest first): 3. **`Config.SampleRatio`**: Ratio-based sampling via config file. 4. **Default**: `ParentBased(AlwaysOn)` -- samples all root spans, respects parent decisions. -### Supported Sampler Values +### Supported sampler values | `OTEL_TRACES_SAMPLER` | Description | |------------------------|-------------| @@ -215,7 +215,7 @@ Trace sampling is configurable with the following precedence (highest first): | `parentbased_always_off` | Parent-based with always-off root sampler | | `parentbased_traceidratio` | Parent-based with ratio root sampler | -### Example: 10% Sampling +### Example: 10% sampling ```bash export OTEL_TRACES_SAMPLER=traceidratio @@ -223,7 +223,7 @@ export OTEL_TRACES_SAMPLER_ARG=0.1 codefang run -a history/burndown . ``` -### Production Recommendation +### Production recommendation Use `parentbased_traceidratio` with a low ratio (1-5%) to limit trace volume while preserving distributed context propagation: @@ -233,7 +233,7 @@ export OTEL_TRACES_SAMPLER=parentbased_traceidratio export OTEL_TRACES_SAMPLER_ARG=0.01 ``` -### Tail Sampling (Collector-Side) +### Tail sampling (collector-side) Head-based sampling decides at trace start whether to sample. This means slow or errored traces may be dropped. **Tail sampling** defers the decision until @@ -288,13 +288,13 @@ representative sample of normal traffic. --- -## Attribute Filter +## Attribute filter All exported spans pass through an `AttributeFilter` span processor that enforces an allow-list of attribute key prefixes. This prevents PII and high-cardinality data from reaching the collector. -### Allowed Prefixes +### Allowed prefixes | Prefix | Covers | |--------|--------| @@ -311,7 +311,7 @@ high-cardinality data from reaching the collector. | `runner.` | Runner attributes | | `cache` | Cache statistics | -### Blocked Keys +### Blocked keys | Pattern | Reason | |---------|--------| @@ -326,7 +326,7 @@ development. --- -## HTTP Middleware +## HTTP middleware The `observability.HTTPMiddleware` wraps HTTP handlers with: @@ -342,12 +342,12 @@ that send a `traceparent` header have their traces continued by Codefang. --- -## Health Checks +## Health checks For server mode deployments (Kubernetes), the observability package provides HTTP handlers: -### Liveness Probe +### Liveness probe ``` GET /healthz -> HTTP 200 {"status": "ok"} @@ -355,7 +355,7 @@ GET /healthz -> HTTP 200 {"status": "ok"} Always returns 200. Use as a Kubernetes liveness probe. -### Readiness Probe +### Readiness probe ``` GET /readyz -> HTTP 200 {"status": "ok"} @@ -370,7 +370,7 @@ mux.Handle("/healthz", observability.HealthHandler()) mux.Handle("/readyz", observability.ReadyHandler(dbCheck, cacheCheck)) ``` -### Prometheus Metrics Endpoint +### Prometheus metrics endpoint ``` GET /metrics -> Prometheus exposition format @@ -387,7 +387,7 @@ mux.Handle("/metrics", metricsHandler) --- -## Structured Logging +## Structured logging All framework-level log messages use `log/slog` with instance loggers (not the global `slog.Default()`). The `TracingHandler` automatically injects @@ -401,7 +401,7 @@ trace context into every log record: | `mode` | Operating mode (`cli`, `mcp`) | | `env` | Deployment environment | -### Log Modes +### Log modes | Mode | Handler | Use case | |------|---------|----------| @@ -409,7 +409,7 @@ trace context into every log record: | MCP | `slog.JSONHandler` | Machine-parseable structured logs | | Debug | Level `DEBUG` | Verbose output for troubleshooting | -### Linter Enforcement +### Linter enforcement The following linters enforce observability correctness in `.golangci.yml`: @@ -421,12 +421,12 @@ The following linters enforce observability correctness in `.golangci.yml`: --- -## Deep Context Propagation +## Deep context propagation All hot-path `context.Background()` calls have been eliminated. Context flows end-to-end from the CLI entry point through every layer: -### Propagation Path +### Propagation path ``` CLI (codefang.run) @@ -452,7 +452,7 @@ CLI (codefang.run) +-> parser.Parse(ctx) ``` -### Key Interfaces +### Key interfaces The `Consume` method on history analyzers accepts context: @@ -482,7 +482,7 @@ func TreeDiff(ctx context.Context, repo *Repository, oldTree, newTree *Tree) (Ch --- -## Integration Points Summary +## Integration points summary | Component | Tracing | Metrics | Logging | |-----------|---------|---------|---------| diff --git a/site/release-notes/index.md b/site/release-notes/index.md new file mode 100644 index 00000000..917e2b95 --- /dev/null +++ b/site/release-notes/index.md @@ -0,0 +1,18 @@ + + +# Release notes + +Release notes summarize what changed in each Codefang release, how to upgrade, and any known issues. They expand on the entries in the canonical [changelog](../contributing/changelog.md), which follows [Keep a Changelog 1.1.0](https://keepachangelog.com/en/1.1.0/) and [Semantic Versioning](https://semver.org/spec/v2.0.0.html). + +## How releases are cut + +Codefang releases are automated. Pushing a `v*` tag (for example `v1.0.0`) triggers the GoReleaser pipeline in the [CI/CD workflow](https://github.com/Sumatoshi-tech/codefang/blob/main/.github/workflows/ci.yml), which builds the cross-platform binaries and publishes a [GitHub Release](https://github.com/Sumatoshi-tech/codefang/releases). Each release page here mirrors the GitHub Release and the matching changelog section. + +## Releases + +| Release | Notes | +|---|---| +| Upcoming release | [Release notes](unreleased.md) | + +!!! note "No tagged release yet" + Codefang has not cut its first tagged release. The [upcoming release](unreleased.md) page tracks everything staged in the `[Unreleased]` section of the changelog. Once the first `v*` tag ships, this page gains a dated row per release. diff --git a/site/release-notes/unreleased.md b/site/release-notes/unreleased.md new file mode 100644 index 00000000..5930921f --- /dev/null +++ b/site/release-notes/unreleased.md @@ -0,0 +1,99 @@ + + + + + +# Upcoming release + +These notes describe the upcoming first release line of Codefang. No `v*` tag has been cut yet, so this page tracks everything staged in the `[Unreleased]` section of the [changelog](../contributing/changelog.md). When the first tag ships, the GoReleaser pipeline in the [CI/CD workflow](https://github.com/Sumatoshi-tech/codefang/blob/main/.github/workflows/ci.yml) publishes the matching [GitHub Release](https://github.com/Sumatoshi-tech/codefang/releases), and these notes are promoted to a versioned page. + +## Highlights + +- Codefang is a complete rewrite of the original `src-d/hercules` project into a modern, idiomatic Go codebase with clean architecture and comprehensive tests. +- The tool ships as two composable binaries: `uast` (a Tree-sitter-based Universal AST parser for 60+ languages) and `codefang` (the static and history analysis engine). +- Several output formats target both humans and data warehouses: `timeseries`, `plot` (interactive HTML), and `ndjson` (streaming, warehouse-friendly). +- An MCP server (`codefang mcp`) exposes analysis to AI agents over the Model Context Protocol. +- A streaming pipeline with checkpointing and crash recovery makes bounded-memory analysis of very large repositories practical. +- OpenTelemetry observability provides distributed tracing, RED metrics, and structured logging. + +## Upgrade notes + +This is the first release line, so there is no prior version to upgrade from. If you tracked an unreleased build, note these behavior changes before adopting the release: + +- **Default file filtering changed.** Analysis now excludes vendored and generated files by default, matching mature multi-language analyzers. To restore the previous behavior, pass `--include-vendored --include-generated`. +- **`--languages` is now cross-phase.** It narrows both static and history analysis through a single source of truth and fails fast on unknown language tokens. Pipelines that relied on static analysis ignoring this flag must drop it or supply valid tokens. +- **Output schema changes.** Several fields moved from maps to sorted arrays so they can be UNNEST'd in columnar warehouses: `developers[].languages`, `activity[].by_developer`, and `file_contributors[].contributors`. Clone-pair `func_a` / `func_b` paths are now relative. Update any consumers that read these fields. +- **Deprecated flags.** `--skip-blacklist` is now a no-op, and `--blacklisted-prefixes` is superseded by `--extra-excluded-prefixes` (identical semantics). Both emit a deprecation warning when passed. + +## Added + +- Temporal anomaly detection analyzer (`history/anomaly`) using Z-score statistical analysis over sliding windows to detect sudden quality degradation in commit history. +- TimeSeries output format (`--format timeseries`) that merges all analyzer outputs into a single chronologically ordered JSON array. +- Plot output format (`--format plot`) generating interactive HTML charts via go-echarts. +- NDJSON output format (`--format ndjson`) emitting one JSON line per analyzer result (with an optional metadata line) for streaming ingestion into warehouses such as ClickHouse. +- MCP server (`codefang mcp`) exposing analysis capabilities as tools for AI agents via the Model Context Protocol (stdio transport, JSON-RPC 2.0), with the `codefang_analyze`, `uast_parse`, and `codefang_history` tools. +- Docker support with a multi-stage Dockerfile and a `debian:bookworm-slim` runtime image that runs as a non-root `codefang` user by default. +- GitHub Actions integration (`action.yml`) for automated code-quality checks in CI pipelines with configurable analyzers, formats, and quality gates. +- OpenTelemetry observability with distributed tracing, RED metrics (rate, errors, duration), and structured logging with trace-context injection. Supports Jaeger, Prometheus, and OTLP collectors. +- Streaming pipeline for bounded-memory processing of large repositories via chunk-based processing with hibernate/boot cycles. +- Double-buffered chunk pipelining that overlaps pipeline prefetch with analyzer consumption for higher throughput on multi-chunk workloads. +- Checkpointing with automatic save after each processed chunk and crash recovery via the `--resume` flag. Checkpoint format v2 preserves aggregator spill state so resumed runs produce identical output. +- Configuration system (`.codefang.yaml`) with file, environment-variable, and CLI-flag support. Merge priority: CLI > env > file > defaults. +- Large-scale scanning support for fleet analysis of thousands of repositories, with bare-repo support, GNU Parallel and Kubernetes orchestration patterns, and data-warehouse loading guides (Athena, Snowflake, Spark). +- Deep context propagation that eliminates `context.Background()` calls in production hot paths for end-to-end tracing. +- Attribute-filter span processor that enforces an allow-list of attribute key prefixes to prevent PII leakage to collectors. +- Health-check endpoints (`/healthz`, `/readyz`, `/metrics`) for server-mode deployments. +- Incremental scanning with the `--since` flag, supporting Go durations, date strings, and RFC 3339 timestamps. +- Memory-budget auto-tuning via the `--memory-budget` flag with automatic chunk-size calculation. +- Watchdog stall detection with a configurable `worker_timeout` for identifying hung workers. +- `--include-vendored` flag (bool, default `false`) to re-include paths detected as vendored by enry/Linguist (`vendor/`, `node_modules/`, `third_party/`, `testdata/`, `dist/`, minified bundles, and more). +- `--include-generated` flag (bool, default `false`) to re-include auto-generated files (`*.pb.go`, `zz_generated_*.go`, `*_pb2.py`, `*.min.js`, and content-header markers such as `DO NOT EDIT`, `Code generated`, `@generated`). +- `--extra-excluded-prefixes` flag (strings, default `[]`) for additional UNIX path prefixes to exclude for ecosystems enry does not know (for example `.venv/`, `target/`, `.gradle/`). +- `source_file` field on every function record (relative path, for example `pkg/kubelet/kubelet.go`) for `static/complexity`, `static/halstead`, `static/cohesion`, and `static/comments`. +- `language` field on every function record (for example `go`, `bash`) for the same static analyzers. +- `directory` field on every function record (for example `pkg/kubelet`) for the same static analyzers. +- `start_time` and `end_time` fields (RFC 3339) on every history time-series tick for `history/sentiment`, `history/anomaly`, `history/quality`, `history/devs`, and `history/file-history`. +- `email` field on developer records, plus `primary_dev_email` / `secondary_dev_email` on bus-factor records and `developer1_email` / `developer2_email` on developer-coupling records, via a new `SplitIdentity` helper. +- Top-level `metadata` section in the report envelope with `repo_path`, `repo_name`, `analyzed_at` (RFC 3339), and `codefang_version`. +- Per-analyzer `schema` manifest in each analyzer result with field `type`, `grain`, and `description` for automated ETL generation across all 17 analyzers. +- `clone_type_distribution` computed from the full pair population rather than the capped 1,000-pair sample, so Type-1/2/3 percentages are accurate on large codebases. + +## Changed + +- Complete rewrite from the original `src-d/hercules` project into a modern Go codebase with idiomatic patterns, clean architecture, and comprehensive test coverage. +- Split the tool into two binaries: `uast` (Universal AST parser using Tree-sitter for 60+ languages) and `codefang` (analysis engine for static and history analysis). Unix philosophy: small tools joined by pipes. +- Replaced the original Babelfish/bblfsh parser with a Tree-sitter-based UAST that is faster, more reliable, and locally compiled, supporting 60+ programming languages. +- Added a DSL-based UAST mapping layer with a custom domain-specific language for transforming Tree-sitter ASTs into standardized UAST nodes. +- Vendored libgit2 (`third_party/libgit2`), compiled as a static library for reproducible builds without external dependencies. +- Replaced all `log.Printf` and `fmt.Printf` calls with structured logging via `log/slog`. Instance loggers are enforced by `sloglint`. +- Removed all backward-compatibility fallbacks in the quality, anomaly, sentiment, and devs `ParseReportData` paths; formalized shotness shallow extraction; and integrated VADER (GoVader) for real sentiment analysis. +- Default analysis output across both phases now excludes vendor and generated files, matching the convention of mature multi-language analyzers. Pass `--include-vendored --include-generated` to restore the previous behavior. **Breaking change.** +- Made `--languages` cross-phase. Static analysis previously ignored the flag; it now narrows both static and history phases through a single `internal/analyzers/plumbing/langpath` source of truth, with fail-fast errors on unknown language tokens. +- Pushed the `--languages` filter down into libgit2 via a new `cf_tree_diff_v2` C ABI that forwards a pathspec to `git_diff_options.pathspec`, cutting tree-diff work on polyglot repositories with a narrow filter (about 34% wall-time and 36% `cgocall` CPU reduction on a synthetic fixture). +- Replaced the `FileContentAnalyzer` + `WalksAllFiles` marker-interface pattern with explicit `StaticAnalyzer` (UAST) and `RawFileAnalyzer` (raw file) hierarchies sharing a `FormattableAnalyzer` base, driven by explicit pipeline stages. This enabled relative source paths and per-record language stamping. +- Changed `developers[].languages` from a map keyed by language name to a sorted `[]LanguageStatsEntry` array so it can be UNNEST'd in columnar warehouses. Empty language strings become `Other`. **Breaking change to output schema.** +- Changed `activity[].by_developer` from a `map[int]int` to a sorted `[]DeveloperCommits` array of `{dev_id, commits}`. **Breaking change to output schema.** +- Changed `file_contributors[].contributors` from a `map[int]LineStats` to a sorted `[]ContributorEntry` array of `{dev_id, added, removed, changed}`. **Breaking change to output schema.** +- Changed clone-pair `func_a` / `func_b` paths from absolute to relative. + +## Deprecated + +- `--skip-blacklist` is now a no-op (the new default already excludes vendor and generated files); a Cobra deprecation warning fires when it is passed. +- `--blacklisted-prefixes` is superseded by `--extra-excluded-prefixes` (identical semantics); a Cobra deprecation warning fires when it is passed. + +## Removed + +- Removed `// FRD: specs/frds/FRD-...md` comments from all `.go` files; `specs/` is gitignored, so those references broke for anyone cloning the repo. Traceability now lives in FRDs and PR descriptions. + +## Fixed + +- Fixed a data race in `internal/framework.PipelineSampler`: `t1Captured` was a plain `bool` read by the sampler goroutine and written by the caller, causing intermittent races under `go test -race`. It is now a `sync/atomic.Bool` using `CompareAndSwap`, so at most one t1 heap profile is captured. Removed the unused `t0Captured` field. + +## Security + +- No security fixes are staged for this release. + +## Known issues + +- No tagged release exists yet, so the changelog `[Unreleased]` reference link points at the commits view rather than a version compare base. The link is replaced with a compare base once the first `v*` tag ships. +- Several output-schema changes listed under [Changed](#changed) are breaking. Consumers that parsed the previous map-shaped fields must migrate to the sorted-array shapes before adopting this release. From 144dde194592bfb986db508477e43fe3e69fc344 Mon Sep 17 00:00:00 2001 From: Dmitriy Date: Sat, 30 May 2026 17:06:39 +0300 Subject: [PATCH 2/3] docs: fix doc-CI pipeline (markdownlint, lychee, cspell, Vale) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The doc-quality jobs globbed **/*.md, sweeping in non-documentation markdown (.agents/**, .claude/**, specs/**, AGENTS.md) that produced 984 markdownlint errors and link/spell failures unrelated to the docs. Scope all doc-quality gates to the documentation surface (site/, docs/, and the root health files) — matching Vale's existing scope: - markdownlint, cspell, lychee globs narrowed to the doc surface Fix real issues within the documentation: - markdownlint: add a language to 36 bare code fences (MD040); add blank lines around the AI-agent scenario lists in README (MD032); auto-fix list-marker spacing, list/strong style (MD030/MD004/MD050) - MD025: disable front-matter-title detection — MkDocs Material pages intentionally carry both a YAML `title:` and a rendered `# H1` - cspell: seed technical terms, tool names, and proper nouns into the dictionary; normalize British spellings to US English (behaviour, licence, prioritise, weaponised, analysers) per the Google/Microsoft style packs (the intentional UK "licence" reference in sentiment.md is preserved via the dictionary) Make the advisory checks reliably green: - Vale/cspell get step-level continue-on-error. Job-level continue-on-error only unblocks the run; the check still reports failure. Step-level lets the job pass while annotations stay visible — matching the documented "advisory, non-blocking" intent. (Vale exits non-zero on any error-level finding regardless of fail_on_error.) Verified locally: markdownlint 0 errors, lychee 0 errors, cspell 0 issues, mkdocs build --strict exit 0. Co-Authored-By: Claude Opus 4.8 (1M context) --- .github/workflows/docs.yml | 51 +++++++++++++--- .markdownlint.jsonc | 7 ++- CHANGELOG.md.proposed.md | 4 +- README.md | 37 +++++++----- SECURITY.md | 2 +- SUPPORT.md | 4 +- THIRD_PARTY_NOTICES.md | 6 +- cspell.json | 58 ++++++++++++++++++- .../adr/0001-record-architecture-decisions.md | 2 +- site/analyzers/burndown.md | 2 +- site/analyzers/sentiment.md | 2 +- site/architecture/overview.md | 4 +- site/architecture/streaming-pipeline.md | 14 ++--- site/architecture/uast.md | 10 ++-- site/contributing/index.md | 2 +- site/explanation/clones.md | 2 +- site/explanation/cohesion.md | 4 +- site/explanation/couples.md | 4 +- site/explanation/shotness.md | 2 +- site/guide/cli-reference.md | 4 +- site/guide/configuration.md | 2 +- site/guide/data-analytics.md | 2 +- site/index.md | 18 +++--- site/integrations/ai-agents.md | 8 +-- site/operations/large-scale-scanning.md | 6 +- site/operations/observability.md | 12 ++-- 26 files changed, 186 insertions(+), 83 deletions(-) diff --git a/.github/workflows/docs.yml b/.github/workflows/docs.yml index c40e28e8..f6a33bd2 100644 --- a/.github/workflows/docs.yml +++ b/.github/workflows/docs.yml @@ -85,12 +85,20 @@ jobs: - name: Run markdownlint-cli2 uses: DavidAnson/markdownlint-cli2-action@v16 with: + # Scope to the documentation surface only. Harness/spec markdown + # (.agents/**, .claude/**, specs/**, AGENTS.md, …) is not user-facing + # documentation and is not gated by the doc-quality rules. globs: | - **/*.md - !build/** - !vendor/** - !third_party/** - !node_modules/** + site/**/*.md + docs/**/*.md + README.md + CONTRIBUTING.md + CODE_OF_CONDUCT.md + SECURITY.md + SUPPORT.md + GOVERNANCE.md + THIRD_PARTY_NOTICES.md + CHANGELOG.md.proposed.md vale: name: Prose style (Vale, advisory) @@ -104,6 +112,11 @@ jobs: - name: Run Vale (Microsoft + Google packs) uses: errata-ai/vale-action@reviewdog + # Step-level continue-on-error keeps this advisory check GREEN even when + # Vale/reviewdog exits non-zero (it does so on any error-level finding, + # independent of fail_on_error). Job-level continue-on-error only unblocks + # the run; it does not flip the reported check conclusion to success. + continue-on-error: true with: fail_on_error: false reporter: github-pr-check @@ -123,9 +136,22 @@ jobs: - name: Run cspell uses: streetsidesoftware/cspell-action@v6 + # Advisory check: keep it green even if a new term is not yet in the + # dictionary. The dictionary is currently clean over the doc surface. + continue-on-error: true with: config: cspell.json - files: '**/*.md' + files: | + site/**/*.md + docs/**/*.md + README.md + CONTRIBUTING.md + CODE_OF_CONDUCT.md + SECURITY.md + SUPPORT.md + GOVERNANCE.md + THIRD_PARTY_NOTICES.md + CHANGELOG.md.proposed.md incremental_files_only: false lychee: @@ -143,7 +169,18 @@ jobs: --verbose --max-concurrency 8 --exclude-mail - './**/*.md' + 'site/**/*.md' + 'docs/**/*.md' + 'README.md' + 'CONTRIBUTING.md' + 'CODE_OF_CONDUCT.md' + 'SECURITY.md' + 'SUPPORT.md' + 'GOVERNANCE.md' + 'THIRD_PARTY_NOTICES.md' + 'CHANGELOG.md.proposed.md' + 'llms.txt' + 'llms-full.txt' fail: true env: GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} diff --git a/.markdownlint.jsonc b/.markdownlint.jsonc index a9f2f764..d03c6870 100644 --- a/.markdownlint.jsonc +++ b/.markdownlint.jsonc @@ -31,5 +31,10 @@ "MD036": false, // MD034 bare URLs: badges and reference links occasionally use bare URLs. - "MD034": false + "MD034": false, + + // MD025 single-title: MkDocs Material pages carry a YAML front-matter + // `title:` (for nav/browser title) AND a rendered `# H1`. Both are + // intentional, so don't treat the front-matter title as a second H1. + "MD025": { "front_matter_title": "" } } diff --git a/CHANGELOG.md.proposed.md b/CHANGELOG.md.proposed.md index e33e53f2..fb59695c 100644 --- a/CHANGELOG.md.proposed.md +++ b/CHANGELOG.md.proposed.md @@ -115,8 +115,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 sentiment, and devs `ParseReportData` paths; formalized shotness shallow extraction; and integrated VADER (GoVader) for real sentiment analysis. - Default analysis output across both phases now excludes vendor and generated - files, matching the convention of mature multi-language analysers. Pass - `--include-vendored --include-generated` to restore the previous behaviour. + files, matching the convention of mature multi-language analyzers. Pass + `--include-vendored --include-generated` to restore the previous behavior. **Breaking change.** - Made `--languages` cross-phase. Static analysis previously ignored the flag; it now narrows both static and history phases through a single diff --git a/README.md b/README.md index 22ce53ca..e435f28a 100644 --- a/README.md +++ b/README.md @@ -32,6 +32,7 @@ This is a **reincarnation**. We took the core philosophy, stripped out the obsol Once upon a time, there was a company called **source{d}**. They pioneered "Mining Software Repositories" on a massive scale. Their crown jewel was `hercules`, a tool that could chew through git logs faster than you can say `git blame`. It gave us insights like: + - **Burndown Charts**: How much code from 2015 is still surviving today? - **Coupling**: If I change `A.go`, does `B.go` always change with it? @@ -98,8 +99,8 @@ We made a few bold choices in the rewrite. The original `hercules` was a monolith. We split it in two: -1. **`uast` (The Parser)**: This tool focuses on one thing—turning source code into a standardized tree structure. It uses **Tree-sitter** under the hood to support practically every language you care about (Go, Python, JS, Rust, C++, Java, and even COBOL... probably). -2. **`codefang` (The Analyzer)**: This tool consumes the data. It takes UASTs for static analysis or Git repositories for history analysis. +1. **`uast` (The Parser)**: This tool focuses on one thing—turning source code into a standardized tree structure. It uses **Tree-sitter** under the hood to support practically every language you care about (Go, Python, JS, Rust, C++, Java, and even COBOL... probably). +2. **`codefang` (The Analyzer)**: This tool consumes the data. It takes UASTs for static analysis or Git repositories for history analysis. ### Why UAST? @@ -116,26 +117,30 @@ Analyzers are modular. You want to measure "Sentiment of Code Comments"? There's Don't just use AI to write code. Use Codefang to verify it. By giving `codefang` and `uast` to your AI agent as tools (via MCP or shell), you create a self-correcting quality loop. **1. The Self-Correcting Coder** -* **Scenario:** Agent generates a new function or refactors a file. -* **Action:** Agent runs `codefang analyze -a complexity` on the new code. -* **Outcome:** If complexity scores are high (e.g., > 15), the agent self-reflects: *"This is too complex. I need to simplify logic before showing it to the user."* -* **Result:** Clean, maintainable code *before* it even hits the PR. + +- **Scenario:** Agent generates a new function or refactors a file. +- **Action:** Agent runs `codefang analyze -a complexity` on the new code. +- **Outcome:** If complexity scores are high (e.g., > 15), the agent self-reflects: *"This is too complex. I need to simplify logic before showing it to the user."* +- **Result:** Clean, maintainable code *before* it even hits the PR. **2. Architectural Context Injection** -* **Scenario:** You ask the agent about a high-level architectural change. -* **Problem:** "Context Window Exceeded" or the agent hallucinates file relationships. -* **Solution:** Agent runs `uast parse` to get the AST structure or `codefang analyze -a imports` to map the dependency graph. It learns the system architecture without reading every single line of text. + +- **Scenario:** You ask the agent about a high-level architectural change. +- **Problem:** "Context Window Exceeded" or the agent hallucinates file relationships. +- **Solution:** Agent runs `uast parse` to get the AST structure or `codefang analyze -a imports` to map the dependency graph. It learns the system architecture without reading every single line of text. **3. Risk-Aware Refactoring** -* **Scenario:** Agent is asked to refactor a legacy module. -* **Action:** Agent runs `codefang history -a couples`. -* **Insight:** *"Warning: Changing `User.go` usually breaks `Billing.go` 80% of the time."* -* **Result:** Agent proactively checks related files for regressions, preventing bugs that a simple text-based analysis would miss. + +- **Scenario:** Agent is asked to refactor a legacy module. +- **Action:** Agent runs `codefang history -a couples`. +- **Insight:** *"Warning: Changing `User.go` usually breaks `Billing.go` 80% of the time."* +- **Result:** Agent proactively checks related files for regressions, preventing bugs that a simple text-based analysis would miss. **4. Style & Consistency Enforcement** -* **Scenario:** "Make this look like the rest of the project." -* **Action:** Agent analyzes `cohesion` and `comments` metrics of existing high-quality files to set a baseline. -* **Result:** Agent generates code that matches the *structural* quality standards of your specific repo, not just generic language syntax. + +- **Scenario:** "Make this look like the rest of the project." +- **Action:** Agent analyzes `cohesion` and `comments` metrics of existing high-quality files to set a baseline. +- **Result:** Agent generates code that matches the *structural* quality standards of your specific repo, not just generic language syntax. --- diff --git a/SECURITY.md b/SECURITY.md index 92e73de0..b94629b9 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -3,7 +3,7 @@ # Security policy Found a way to make Codefang misbehave that goes beyond a normal bug? We want to -hear about it privately first, so we can fix it before it gets weaponised. +hear about it privately first, so we can fix it before it gets weaponized. ## Supported versions diff --git a/SUPPORT.md b/SUPPORT.md index da8df9a0..6daa763b 100644 --- a/SUPPORT.md +++ b/SUPPORT.md @@ -30,12 +30,12 @@ we can reproduce, the faster we can fix. ## I found a security vulnerability **Do not open a public issue.** Security holes go through the private channel so we -can patch before they get weaponised. See [SECURITY.md](SECURITY.md) for the full +can patch before they get weaponized. See [SECURITY.md](SECURITY.md) for the full drill—it routes you to GitHub's private vulnerability reporting (Security Advisories), not the public Issues tracker. ## What to expect This project is maintained on a best-effort basis. We triage questions and bug reports -as time allows, and we prioritise reproducible reports. Be kind, be patient, and bring +as time allows, and we prioritize reproducible reports. Be kind, be patient, and bring details—that's how the dumbbells get racked. diff --git a/THIRD_PARTY_NOTICES.md b/THIRD_PARTY_NOTICES.md index 653c9537..196e2692 100644 --- a/THIRD_PARTY_NOTICES.md +++ b/THIRD_PARTY_NOTICES.md @@ -2,7 +2,7 @@ Codefang is licensed under [Apache-2.0](LICENSE). It reuses or bundles content from the projects below. This file names each borrowed component and points to -its upstream licence; it does not reproduce the full licence texts. Consult the +its upstream license; it does not reproduce the full license texts. Consult the linked upstream sources for the authoritative terms. The full dependency graph (the MIT / Apache-2.0 / BSD Go modules pulled in via @@ -12,7 +12,7 @@ warrants an explicit notice. ## Borrowed content -| Component | What it is | Upstream licence | Link | +| Component | What it is | Upstream license | Link | | --- | --- | --- | --- | | Contributor Covenant 2.1 | Text adapted into our [code of conduct](site/contributing/code-of-conduct.md) | CC-BY-4.0 | | | Tree-sitter grammars | Per-language parser grammars consumed via `go-sitter-forest` (one module per language in `go.mod`) | Per-grammar (mostly MIT / Apache-2.0) | | @@ -23,7 +23,7 @@ warrants an explicit notice. - The Tree-sitter grammars are a category: each language grammar listed in `go.mod` under `github.com/alexaandru/go-sitter-forest/*` carries its own - upstream licence. See the linked aggregator for the per-grammar provenance. + upstream license. See the linked aggregator for the per-grammar provenance. - libgit2 ships under GPL-2.0 **with a linking exception** that permits linking the compiled library into other programs and distributing those combinations without restriction. The exception and full text live in diff --git a/cspell.json b/cspell.json index 6cfd717c..3cffd610 100644 --- a/cspell.json +++ b/cspell.json @@ -78,7 +78,63 @@ "devs", "LRU", "hibernatable", - "timeseries" + "timeseries", + "Babelfish", + "bblfsh", + "busfactor", + "Cabe", + "CFLAGS", + "Checkpointable", + "checkpointing", + "contextcheck", + "cpuprofile", + "embedders", + "Errorf", + "frds", + "gitlib", + "Gitolite", + "GOBIN", + "gogc", + "GOMAXPROCS", + "GOMEMLIMIT", + "GOPATH", + "Greenplum", + "heapprofile", + "hotspot", + "hotspots", + "Inuse", + "iosafety", + "langpath", + "ldflags", + "LDFLAGS", + "libgomp", + "maat", + "malloc", + "mapx", + "metaprogramming", + "parentbased", + "pathspec", + "phpcs", + "rubocop", + "safeconv", + "sandboxed", + "scalafix", + "Skiena", + "sloglint", + "spancheck", + "stddev", + "testdata", + "textutil", + "traceidratio", + "traceparent", + "tracestate", + "treap", + "venv", + "licence", + "Dmytro", + "dmytrogajewski", + "Gajewski", + "Yourdon" ], "ignoreRegExpList": [ "/\\b[0-9a-f]{7,40}\\b/", diff --git a/docs/adr/0001-record-architecture-decisions.md b/docs/adr/0001-record-architecture-decisions.md index 471985ad..f0e63703 100644 --- a/docs/adr/0001-record-architecture-decisions.md +++ b/docs/adr/0001-record-architecture-decisions.md @@ -8,7 +8,7 @@ ## Context and problem statement -Codefang is a reincarnation of the abandoned `hercules` tool, rebuilt with a modern engine. The rewrite made several significant choices — a two-binary split, a Tree-sitter-backed Universal Abstract Syntax Tree, libgit2 via cgo, a streaming pipeline, MkDocs Material for docs, and the Apache-2.0 licence. These decisions currently live only as prose scattered across the README and `site/architecture/`, or as feature-scoped FRDs under `specs/frds/`. New contributors cannot see *why* a choice was made, what was considered, or whether a past decision still holds. How should the project capture architecturally significant decisions so the reasoning survives the people who made it? +Codefang is a reincarnation of the abandoned `hercules` tool, rebuilt with a modern engine. The rewrite made several significant choices — a two-binary split, a Tree-sitter-backed Universal Abstract Syntax Tree, libgit2 via cgo, a streaming pipeline, MkDocs Material for docs, and the Apache-2.0 license. These decisions currently live only as prose scattered across the README and `site/architecture/`, or as feature-scoped FRDs under `specs/frds/`. New contributors cannot see *why* a choice was made, what was considered, or whether a past decision still holds. How should the project capture architecturally significant decisions so the reasoning survives the people who made it? ## Decision drivers diff --git a/site/analyzers/burndown.md b/site/analyzers/burndown.md index 7457f00b..301b80fd 100644 --- a/site/analyzers/burndown.md +++ b/site/analyzers/burndown.md @@ -47,7 +47,7 @@ history: The `--format text` output produces a concise terminal summary: -``` +```text ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Burndown: project-name 730d ┃ ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛ diff --git a/site/analyzers/sentiment.md b/site/analyzers/sentiment.md index 57e05555..abdee1a1 100644 --- a/site/analyzers/sentiment.md +++ b/site/analyzers/sentiment.md @@ -100,7 +100,7 @@ The plot uses semantic colors (green = positive, yellow = neutral, red = negativ The terminal renderer provides a colored, Unicode-rich summary: -``` +```text ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ SENTIMENT ANALYSIS 💬 ┃ ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛ diff --git a/site/architecture/overview.md b/site/architecture/overview.md index 634a087c..7d8bf681 100644 --- a/site/architecture/overview.md +++ b/site/architecture/overview.md @@ -262,7 +262,7 @@ merges the results into a single output document in the requested format. The history analysis pipeline is built around a **Runner** that coordinates the full lifecycle: -``` +```text Initialize --> ProcessChunk (x N) --> FinalizeWithAggregators ``` @@ -284,7 +284,7 @@ The **Runner** supports two execution strategies: Configuration follows a clear priority chain: -``` +```text CLI flags > Environment variables > .codefang.yaml > Compiled defaults ``` diff --git a/site/architecture/streaming-pipeline.md b/site/architecture/streaming-pipeline.md index 1080e32f..6ad689fa 100644 --- a/site/architecture/streaming-pipeline.md +++ b/site/architecture/streaming-pipeline.md @@ -102,7 +102,7 @@ The `streaming.Planner` calculates optimal chunk boundaries. It lives in The scheduler decomposes the memory budget into four explicit regions: -``` +```text B = P + W + A + S usable = budget * 0.95 (S = 5% slack for GC headroom) @@ -120,7 +120,7 @@ buffering factor, and the aggregator spill budget. The planner determines chunk size from the **working state** portion of the budget: -``` +```text growth = working_state_per_commit * 1.5 (safety margin) chunk_size = clamp(workState / growth, MinChunkSize, MaxChunkSize) ``` @@ -151,7 +151,7 @@ analyzers. Plumbing analyzers return 0 for both and are excluded from the sum. For a 10,000-commit repo with 4 GiB budget, 400 MiB pipeline overhead, and 1.5 MiB/commit working state growth: -``` +```text usable = 4 GiB * 0.95 = 3,891 MiB remaining = 3,891 - 400 = 3,491 MiB workState = 3,491 * 0.60 = 2,095 MiB @@ -189,7 +189,7 @@ The `ComputeSchedule()` function iterates buffering factors from `MaxBuffering` MinChunkSize`. Only the working state region (60% of remaining budget) is divided among buffering slots; the aggregator spill budget is unaffected. -``` +```go for bf = maxBuffering; bf >= 1; bf-- { chunkSize = workState / (bf * effectiveGrowth) if chunkSize >= MinChunkSize { @@ -208,7 +208,7 @@ for bf = maxBuffering; bf >= 1; bf-- { ### How double-buffering works -``` +```text Time ───────────────────────────────────────────> Chunk 1: |==== Pipeline ====|==== Consume ====| @@ -240,7 +240,7 @@ The scheduler handles the memory budget split through the buffering factor iteration. With `BufferingFactor = N`, the working state region is divided among N concurrent slots: -``` +```text workState = remaining * 0.60 chunkSize = workState / (N * effectiveGrowth) ``` @@ -389,7 +389,7 @@ flowchart TD The following diagram shows the complete lifecycle of a streaming analysis run with buffered pipelining (double-buffer shown) and checkpointing: -``` +```text Time ──────────────────────────────────────────────────────────────────────> Phase: INIT CHUNK 1 CHUNK 2 CHUNK 3 FINALIZE diff --git a/site/architecture/uast.md b/site/architecture/uast.md index a95640bc..26101238 100644 --- a/site/architecture/uast.md +++ b/site/architecture/uast.md @@ -103,7 +103,7 @@ A mapping file consists of: 1. A **language declaration** with name and file extensions. 2. One or more **mapping rules** that match Tree-sitter node types and produce UAST nodes. -``` +```text language go [.go] function_declaration <- (function_declaration @@ -121,7 +121,7 @@ function_declaration <- (function_declaration Each rule has four parts: -``` +```text <- () => uast() ``` @@ -155,7 +155,7 @@ Each rule has four parts: Rules can include conditions that filter matches: -``` +```text public_method <- (method_declaration name: (identifier) @name ) where (visibility == "public") => uast( @@ -172,7 +172,7 @@ child types against string literals. Rules can extend other rules to avoid duplication: -``` +```text base_function <- (function_declaration) => uast( type: "Function", roles: "Declaration", "Function" @@ -228,7 +228,7 @@ To add or customize language mappings: Create a `.uast` file for your language: -``` +```text language rust [.rs] fn_item <- (function_item diff --git a/site/contributing/index.md b/site/contributing/index.md index 0433b85c..a9c75719 100644 --- a/site/contributing/index.md +++ b/site/contributing/index.md @@ -239,7 +239,7 @@ produce clean changelogs and enable automated versioning. **Format:** -``` +```text (): diff --git a/site/explanation/clones.md b/site/explanation/clones.md index 44dd5cee..59331958 100644 --- a/site/explanation/clones.md +++ b/site/explanation/clones.md @@ -30,7 +30,7 @@ A clone pair is two functions whose AST structure is similar enough to be consid The ratio of detected clone pairs to total functions, computed globally across all files: -``` +```text clone_ratio = total_clone_pairs / total_functions ``` diff --git a/site/explanation/cohesion.md b/site/explanation/cohesion.md index ab52c417..291fc8f8 100644 --- a/site/explanation/cohesion.md +++ b/site/explanation/cohesion.md @@ -20,7 +20,7 @@ LCOM-HS is the industry-standard cohesion metric used by NDepend, JArchitect, an **Formula:** -``` +```text LCOM = 1 - sum(mA) / (m × a) ``` @@ -55,7 +55,7 @@ For each function, the analyzer computes what fraction of its variables are shar **Formula:** -``` +```text function_cohesion = shared_vars / total_unique_vars ``` diff --git a/site/explanation/couples.md b/site/explanation/couples.md index ea97ae76..c198c04d 100644 --- a/site/explanation/couples.md +++ b/site/explanation/couples.md @@ -20,7 +20,7 @@ A square matrix where each cell `[i][j]` counts the number of commits in which f **Coupling strength** is computed using the code-maat formula: -``` +```text strength = co_changes / average(revisions_file1, revisions_file2) ``` @@ -72,7 +72,7 @@ The couples analyzer follows the **TC/Aggregator pattern**: The coupling strength formula follows the **code-maat** academic standard (Adam Tornhill, "Your Code as a Crime Scene"): -``` +```text degree = shared_revisions / average(revisions_A, revisions_B) ``` diff --git a/site/explanation/shotness.md b/site/explanation/shotness.md index 197860c1..10c63b55 100644 --- a/site/explanation/shotness.md +++ b/site/explanation/shotness.md @@ -27,7 +27,7 @@ When two code entities are modified in the same commit, their coupling counter i Coupling strength is normalized to a 0-1 scale using the formula: -``` +```text strength(A, B) = co_changes(A, B) / max(co_changes(A, B), changes(A), changes(B)) ``` diff --git a/site/guide/cli-reference.md b/site/guide/cli-reference.md index 222aa3eb..2871bbaf 100644 --- a/site/guide/cli-reference.md +++ b/site/guide/cli-reference.md @@ -196,7 +196,7 @@ codefang run -a '*' . # include vendored deps (node_modules/, vendor/, …) codefang run -a '*' --include-vendored . -# restore pre-codefang-2026-04 behaviour (include everything) +# restore pre-codefang-2026-04 behavior (include everything) codefang run -a '*' --include-vendored --include-generated . # skip extras that enry doesn't know about (e.g. Python venv, Rust target/) @@ -208,7 +208,7 @@ codefang run -a '*' --extra-excluded-prefixes '.venv/,target/' . Earlier versions of Codefang analysed vendored and generated files by default (they needed the confusingly-named `--skip-blacklist=true` to be excluded). Starting from 2026-04, defaults flip: vendor / generated - are **excluded by default**. To restore the old behaviour: + are **excluded by default**. To restore the old behavior: ```bash codefang run ... --include-vendored --include-generated diff --git a/site/guide/configuration.md b/site/guide/configuration.md index b739e1ac..d4bdeac2 100644 --- a/site/guide/configuration.md +++ b/site/guide/configuration.md @@ -23,7 +23,7 @@ When no explicit `--config` flag is provided, Codefang searches for a When the same setting is specified in multiple sources, the highest-priority source wins: -``` +```text CLI flags > Environment variables > Config file > Compiled defaults ``` diff --git a/site/guide/data-analytics.md b/site/guide/data-analytics.md index ac7f0ac5..bff3bd77 100644 --- a/site/guide/data-analytics.md +++ b/site/guide/data-analytics.md @@ -68,7 +68,7 @@ codefang run --format ndjson --per-file /repo > report.ndjson One JSON line per analyzer. First line is metadata: -``` +```json {"version":"codefang.run.v1","metadata":{"repo_name":"myproject",...}} {"id":"static/complexity","mode":"static","report":{...}} {"id":"history/sentiment","mode":"history","report":{...}} diff --git a/site/index.md b/site/index.md index f3e571dc..722fabd0 100644 --- a/site/index.md +++ b/site/index.md @@ -41,7 +41,7 @@ The heavy lifter for your codebase — deep code analysis through structure and
-``` +```text ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Codefang Overview Score: 10/10 ┃ ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛ @@ -83,7 +83,7 @@ analysis of entire repositories. One tool, one config, complete understanding. ## Distribution -``` +```text Capabilities ────────────────────────────────────────────────────────────────────────────── UAST Parsing ████████████████████ 60+ languages via Tree-sitter @@ -132,7 +132,7 @@ uast --version ## Quick example -``` +```text $ codefang run -a static/complexity,static/halstead --format text . codefang (v2): @@ -194,7 +194,7 @@ codefang (v2):
-- :material-rocket-launch:{ .lg .middle } __Getting Started__ +- :material-rocket-launch:{ .lg .middle } **Getting Started** --- @@ -202,7 +202,7 @@ codefang (v2): [Quick Start](getting-started/quickstart.md) · [First Analysis](getting-started/first-analysis.md) -- :material-book-open-variant:{ .lg .middle } __User Guide__ +- :material-book-open-variant:{ .lg .middle } **User Guide** --- @@ -210,7 +210,7 @@ codefang (v2): [Configuration](guide/configuration.md) · [Output Formats](guide/output-formats.md) -- :material-cog-outline:{ .lg .middle } __Architecture__ +- :material-cog-outline:{ .lg .middle } **Architecture** --- @@ -218,7 +218,7 @@ codefang (v2): [UAST System](architecture/uast.md) · [Streaming Pipeline](architecture/streaming-pipeline.md) -- :material-puzzle-outline:{ .lg .middle } __Integrations__ +- :material-puzzle-outline:{ .lg .middle } **Integrations** --- @@ -226,14 +226,14 @@ codefang (v2): [Docker & Actions](integrations/docker-and-actions.md) · [AI Agents](integrations/ai-agents.md) -- :material-telescope:{ .lg .middle } __Operations__ +- :material-telescope:{ .lg .middle } **Operations** --- [Observability](operations/observability.md) · [Large-Scale Scanning](operations/large-scale-scanning.md) -- :material-account-group:{ .lg .middle } __Contributing__ +- :material-account-group:{ .lg .middle } **Contributing** --- diff --git a/site/integrations/ai-agents.md b/site/integrations/ai-agents.md index c7589588..31968236 100644 --- a/site/integrations/ai-agents.md +++ b/site/integrations/ai-agents.md @@ -62,7 +62,7 @@ flowchart TD ### Example agent prompt -``` +```text Generate a Go function that parses CSV files with configurable delimiters and header handling. After generating the code, use codefang_analyze to check complexity. If any function has cyclomatic complexity > 10, refactor @@ -229,7 +229,7 @@ flowchart LR ### Example agent prompt -``` +```text Before writing new code for this module, analyze the existing files in pkg/handlers/ using codefang_analyze to understand the current code style baselines. Then generate the new handler following the same patterns -- @@ -329,7 +329,7 @@ development. Agents should handle common errors gracefully: -``` +```text 1. If codefang_analyze returns an error for unsupported language, fall back to basic heuristic analysis. 2. If codefang_history fails (not a git repo), skip history-based @@ -344,7 +344,7 @@ Agents should handle common errors gracefully: Here is a complete agent workflow for implementing a new feature: -``` +```text 1. UNDERSTAND: Parse existing code with uast_parse to understand structure 2. ASSESS: Run codefang_history (couples, devs) to understand change risk 3. PLAN: Use architectural context to plan the implementation diff --git a/site/operations/large-scale-scanning.md b/site/operations/large-scale-scanning.md index 5abaf288..5ed95c0a 100644 --- a/site/operations/large-scale-scanning.md +++ b/site/operations/large-scale-scanning.md @@ -47,7 +47,7 @@ This is relevant for: GitLab backup tarballs contain bare repositories at predictable paths: -``` +```text backup.tar ├── repositories/ │ └── @hashed/ @@ -291,7 +291,7 @@ devs_df.write.partitionBy("scan_date").parquet("s3://warehouse/codefang/devs/") The static analysis phase logs progress events at regular intervals (every 1000 files). By default, only phase and file count are shown: -``` +```text static: processing files=1000 static: processing files=2000 static: complete files=5432 @@ -299,7 +299,7 @@ static: complete files=5432 With `--verbose`, RSS and aggregator buffer sizes are included: -``` +```text static: processing files=1000 RSS=245MiB agg=12MiB static: processing files=2000 RSS=312MiB agg=18MiB static: complete files=5432 RSS=280MiB agg=6MiB diff --git a/site/operations/observability.md b/site/operations/observability.md index ab987536..0fdca050 100644 --- a/site/operations/observability.md +++ b/site/operations/observability.md @@ -94,7 +94,7 @@ no-op tracer and meter providers: The entire analysis pipeline is instrumented with hierarchical spans: -``` +```text codefang.run (CLI entry point) | +-- codefang.init (analyzer + repo initialization) @@ -189,7 +189,7 @@ When connected to an OTLP collector, the following metrics are exported: Duration histograms use these bucket boundaries (in seconds), covering sub-second static checks through multi-minute history pipelines: -``` +```text 0.01, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10, 30, 60, 120, 300, 600 ``` @@ -349,7 +349,7 @@ HTTP handlers: ### Liveness probe -``` +```text GET /healthz -> HTTP 200 {"status": "ok"} ``` @@ -357,7 +357,7 @@ Always returns 200. Use as a Kubernetes liveness probe. ### Readiness probe -``` +```text GET /readyz -> HTTP 200 {"status": "ok"} -> HTTP 503 {"status": "unavailable"} ``` @@ -372,7 +372,7 @@ mux.Handle("/readyz", observability.ReadyHandler(dbCheck, cacheCheck)) ### Prometheus metrics endpoint -``` +```text GET /metrics -> Prometheus exposition format ``` @@ -428,7 +428,7 @@ end-to-end from the CLI entry point through every layer: ### Propagation path -``` +```text CLI (codefang.run) | +-> framework.RunStreaming(ctx) From 67a475b5f8c28d14da29e97f249631606004c895 Mon Sep 17 00:00:00 2001 From: Dmitriy Date: Sat, 30 May 2026 17:08:05 +0300 Subject: [PATCH 3/3] ci(docs): drop removed --exclude-mail flag from lychee MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Current lychee (lychee-action@v2) rejects --exclude-mail with "unexpected argument" (exit 2), which was the actual cause of the red link-check job — not broken links. mailto:/tel: links are excluded by default now and are also covered by .lycheeignore, so the flag is just dropped. Co-Authored-By: Claude Opus 4.8 (1M context) --- .github/workflows/docs.yml | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/.github/workflows/docs.yml b/.github/workflows/docs.yml index f6a33bd2..1f074d82 100644 --- a/.github/workflows/docs.yml +++ b/.github/workflows/docs.yml @@ -164,11 +164,13 @@ jobs: - name: Run lychee link checker uses: lycheeverse/lychee-action@v2 with: + # NB: mailto:/tel: links are excluded by default in current lychee + # (and via .lycheeignore); the old --exclude-mail flag was removed + # upstream and now errors out, so it is intentionally omitted. args: >- --no-progress --verbose --max-concurrency 8 - --exclude-mail 'site/**/*.md' 'docs/**/*.md' 'README.md'