Skip to content

METHODOLOGY.md

Latest commit

 

History

History
107 lines (68 loc) · 4.61 KB

METHODOLOGY.md

File metadata and controls

107 lines (68 loc) · 4.61 KB
Raw

Methodology (frozen)

Status: methodology-v1
Frozen before experimental agent runs.
Do not change hypotheses, rubrics, or task prompts after seeing results. Protocol deviations must be logged.

Research question

Do the same characteristics that make codebases maintainable for humans — consistency, small abstractions, explicit contracts, strong tests, predictable organization, limited degrees of freedom — also make coding agents materially more reliable?

Prior belief: yes; “agent-friendly architecture” is mostly classical software engineering reinforced by agentic development, not a new discipline.

Design goal: build an experiment capable of proving that prior wrong.

Hypotheses

H1 (architecture)

Holding task text and agent model fixed, agents achieve higher task success and narrower, more appropriate diffs on the canonical app than on the heterogeneous app.

H2 (instructions)

A minimal router AGENTS.md plus progressively disclosed skills matches or beats a giant AGENTS.md with the same information content, especially on navigation cost (files/tool calls).

Falsification

  • Reject H1 if success rate on heterogeneous ≥ canonical (within descriptive comparison), or if canonical systematically produces wider diffs / more unnecessary files / more regressions at equal success.
  • Reject H2 if giant dominates router on both success and navigation cost, or if router clearly harms success.
  • Null and mixed results are valid scientific outcomes and must be reported as such.

Experimental matrix

Cell Architecture Instructions
AH heterogeneous giant
AR heterogeneous router + skills
CH canonical giant
CR canonical router + skills

Primary agent: Claude Code (stream-json). Optional Codex adapter is non-primary.

Units under test

Two functionally equivalent TypeScript apps implementing OpsDesk (docs/PRODUCT.md):

  • apps/heterogeneous — realistic historically evolved patterns (not sabotaged)
  • apps/canonical — constrained, one-obvious-way architecture

Baseline requirement: both pass shared product acceptance tests before any task run.

Task suite

Tasks are defined under tasks/ and registered in tasks/suite.json.

Rules:

  1. Author from product behavior, not from either implementation’s file layout.
  2. Freeze task agent prompts before the first live run.
  3. Do not retune tasks after seeing results except for broken harness or impossible acceptance criteria (logged as protocol deviation).
  4. Expected-touch allowlists are for scoring only; never shown to the agent.

Initial suite: T01–T08 (see tasks/suite.json).

Run protocol

  1. Create ephemeral git worktree from tagged baseline commit for the target app.
  2. Overlay instruction condition from instructions/{giant|router}/{app}/.
  3. Confirm product acceptance + app tests green at baseline.
  4. Invoke agent with only the task prompt (no architecture coaching).
  5. On exit: capture git metrics, transcript/tool stats if available, run verification.
  6. Write results/runs/<run-id>.json.
  7. Discard or archive worktree.

Human intervention count defaults to 0 for autonomous runs.

Metrics

Defined in results/schema.json and summarized in docs/RUBRIC.md.

Automatic: success, tests pass, files/lines touched, unnecessary files, wall time, tool calls/searches, retries, regressions, token usage when available.

Subjective (0–2 each, pre-registered): diff necessity, pattern consistency, contract honesty, test quality, avoidance of parallel abstractions.

Analysis plan

  • Report per-cell success rate, median files touched, median wall time, mean rubric total.
  • Primary contrast: heterogeneous vs canonical (pooled and split by instruction style).
  • Secondary contrast: giant vs router.
  • Use descriptive statistics. Do not claim population significance from a small N.
  • Publish raw run JSON.

Stop rules

  • Pilot: T01, T03, T04 × 4 cells before expanding.
  • Full suite after pilot protocol check.
  • Abort a run only for harness failure or operator kill; record reason.

Bias controls

  • Shared product acceptance suite for behavioral parity.
  • Heterogeneous intentional inconsistencies documented in apps/heterogeneous/ARCHITECTURE_INTENT.md.
  • Instruction packs audited for matched information content (instructions/CONTENT_AUDIT.md).
  • Rubric frozen; reviewers should not be told the preferred architecture outcome.

What this methodology is not

Not a multi-model leaderboard, not a framework for agents, not a claim that one architecture is universally best outside this product and task distribution.