diff --git a/AGENTS.md b/AGENTS.md index 56216bd..3b68c7f 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -4,4 +4,4 @@ Factory is a GitHub Action (`action.yml`), a plugin (`plugin/`: skills, agents, Use an existing SDK or library before writing your own: for a protocol, a file format, a codec or a provider's API, look first for the official SDK, then for a maintained library whose licence the repository can take, because hand-written protocol code falls behind its standard and every line of it is ours to maintain. Write it yourself only when nothing fits (no library, a licence that does not fit, an unwanted install script or native binary, or output that must be byte-for-byte deterministic), and say in the PR what you looked at and why it did not fit, then add it to `docs/PLAN.md`'s Dependencies list; a PR that hand-writes such code without that is a review finding. A runtime library ships vendored in `plugin/vendor`, the same section says how. -Read `docs/PLAN.md` (what Factory is, its contracts, 0.9.x releases and the 1.0 bar), `docs/LOCAL.md` (local mode: setup, sandboxes, settings), `docs/ADAPTERS.md` (third-party adapter contracts: the cloud sandbox and the decision model) and `bench/README.md` (the scorecard and drills). Docs describe Factory as it is now; git is the history, so write no logs, dated narratives or ledgers. Code decides on structure only: JSON the harness writes, GitHub fields, exit codes, counts; never prose. Tests: `node --test plugin/test/*.test.mjs bench/test/*.test.mjs`; each test file that touches the file system imports `plugin/test/temp.mjs` first (its temp files then live in a root removed when the file ends), and CI fails when the suite leaves anything in the temp folder. +Read `docs/PLAN.md` (what Factory is, its contracts, 0.9.x releases and the 1.0 bar), `docs/LOCAL.md` (local mode: setup, sandboxes, settings), `docs/ADAPTERS.md` (third-party adapter contracts: the cloud sandbox and the decision model) and `bench/README.md` (the scorecard and drills). For adopters: `docs/INSTALL.md` (the install footprint, cost inputs, upgrade, uninstall) and `docs/USAGE.md` (writing and following an Objective). `README.md` is the overview and quick start for a first-time reader: keep it short, and put reference detail in `docs/`. Docs describe Factory as it is now; git is the history, so write no logs, dated narratives or ledgers. Code decides on structure only: JSON the harness writes, GitHub fields, exit codes, counts; never prose. Tests: `node --test plugin/test/*.test.mjs bench/test/*.test.mjs`; each test file that touches the file system imports `plugin/test/temp.mjs` first (its temp files then live in a root removed when the file ends), and CI fails when the suite leaves anything in the temp folder. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index bd3bf9a..ebe78bc 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,3 +1,21 @@ # Contributing -Factory takes no outside pull requests until its owner says otherwise, through 0.9.x and 1.0 alike; only maintainers open them. Issues are welcome: bugs, questions and ideas. Report a security problem privately instead (see [SECURITY.md](SECURITY.md)). +Issues are welcome: bugs, questions and ideas. Factory takes no outside pull requests until its owner says otherwise, through 0.9.x and 1.0 alike; only maintainers open them. + +## Reporting a bug + +[Open an issue](https://github.com/clockgrove/factory/issues/new) with: + +- the Factory tag your workflow pins (`clockgrove/factory@vX.Y.Z`), the harness (Claude Code, Codex or Copilot) and the route (Actions, local mode or the cloud sandbox); +- what you expected and what happened, with a link to the Objective's issue or PR when it is public; +- the Blocked or Question comment's text, and `/factory:doctor`'s lines when setup is involved. + +Leave out tokens, keys and private code. Report a security problem privately instead (see [SECURITY.md](SECURITY.md)). + +## Questions and ideas + +Open an issue for these too. Say what you are trying to do; a real Objective that Factory handled badly is the most useful report there is. + +## Working in this repository + +[AGENTS.md](AGENTS.md) holds the rules for maintainers and their agents, and [docs/PLAN.md](docs/PLAN.md) the design. diff --git a/README.md b/README.md index a61aa89..e33f6d3 100644 --- a/README.md +++ b/README.md @@ -1,83 +1,148 @@ -# Factory +# Clockgrove Factory -Label a GitHub issue `factory:objective`, walk away, come back to merged work. +Label a GitHub issue, walk away, come back to merged work. -A worker job checks out your default branch, reads the issue as the whole specification, plans in a comment on it, implements with pinned sub-agents, opens one PR, and waits for your required checks on it (or runs your validation commands when you have no required checks). A review job posts one comment-only review, the worker answers each comment once, validation runs again, and Factory merges exactly the reviewed head once your required checks are green on it (through your merge queue when you have one). The harness itself writes nothing to GitHub: Factory makes every push, PR, comment and merge from the files it leaves. Nothing asks a question; the only stops are **Blocked** (a draft PR, the card in the Blocked column, one comment with the reason) and **Question** (a draft PR whose body asks the one thing a human must decide), and neither stops the queue. To answer a Question, comment the decision on the issue or the PR (a review or a line comment works too) and mark the PR ready: the review job applies it, and with no answer since the question the PR goes back to a Question rather than merging. Re-applying `factory:objective` instead reruns the worker with it. A comment counts when its author has write access to the repository or is a maintainer the workflow names (`notify`, and the optional `maintainers` list), since an App cannot always see an organization member's access. +[![Release](https://img.shields.io/github/v/release/clockgrove/factory)](https://github.com/clockgrove/factory/releases) +[![Quality](https://github.com/clockgrove/factory/actions/workflows/quality.yml/badge.svg)](https://github.com/clockgrove/factory/actions/workflows/quality.yml) +[![License: MIT](https://img.shields.io/github/license/clockgrove/factory)](LICENSE) -## What you see on the issue +Factory turns an issue into a merged pull request with no one watching. You write the goal and what "done" means. A coding agent (Claude Code, Codex or GitHub Copilot) plans the work, builds it and answers one review, and Factory merges it once your own CI is green. -While a job runs, one comment on the issue (edited in place, one per run) carries a bold status line and the phases with clocks: setup, install, harness, each validation pass, reviewer, answer pass, exit. The status line comes from the harness's own event stream, not from its prose: elapsed minutes, turns, tool calls, the sub-agents running and what each was asked to do, when the last event arrived, tokens so far, the last tool it ran. A harness that prints nothing for 30 minutes is killed and retried once, and the line says so. The labels (`factory:running`, `factory:in-review`, `factory:blocked`, `factory:question`) and the board column are the state; the plan comment is the harness's own checklist, ticked as it goes; the PR carries the validation marker and the review. +> **Status:** 0.9.x, the release candidate for 1.0. Run it on real work; there is no stability promise until 1.0 ([Versioning](#versioning)). -## Siblings that touch the same files +## How it works -Two Objectives running at once that edit the same files cost a rebase run each time one merges (ten minutes, a dollar or a few) and, without a merge queue, strict checks refuse the one left behind until the worker rebases it. When you know two Objectives share an area, say so: add the GitHub issue dependency (`blocked_by`) and the dispatcher orders them; or set `max_concurrent: 1` for the stretch. Factory does not guess file overlap from prose. +1. **You label an issue `factory:objective`.** The issue is the whole specification. +2. **A worker plans and builds.** It posts its plan as a comment on the issue, implements with sub-agents and opens one PR. +3. **Your CI decides.** The worker waits for your required checks on the PR, and gets a bounded number of passes to fix a red. +4. **A reviewer comments once.** A second model call reads the diff against the issue, and the worker answers each comment once. +5. **Factory merges** exactly the reviewed commit once your required checks are green on it, through your merge queue when you have one. -## One repository per Objective +Nothing asks you a question along the way. A run that cannot finish ends in one of two states, and neither holds up your other Objectives: -Factory works in one repository per Objective. An Objective that needs a change in another repository stops after its plan with a Question on the issue, before any work: split it into one Objective per repository and re-apply the label, or answer on the issue how to do the work in this repository alone. +- **Blocked:** a draft PR and one comment with the reason. +- **Question:** a draft PR whose body asks the one thing a person must decide. -## Factory merges on its own +## Know before you start -Code lands with no human read. The gate is your CI workflow's checks on the reviewed head and the integrated commit, the comment-only reviewer pass, and the merge queue's own re-run where you have one. Set `FACTORY_AUTO_MERGE: false` for ready-for-review PRs instead. +- **Factory merges on its own by default.** Code lands with no human read. The gate is your CI, the reviewer pass and your merge queue. Set `FACTORY_AUTO_MERGE: false` to get ready-for-review PRs instead. +- **It spends money on model usage.** `max_usd` caps each job and `daily_usd` caps each day. +- **The agent never holds GitHub write access.** Factory makes every push, PR, comment and merge from the files the agent leaves. The agent's commands run in a sandbox, with no credential in their environment unless you opt one in ([docs/PLAN.md](docs/PLAN.md), Credentials and billing). +- **It works in one repository per Objective**, and acts only on an issue whose author can push to the repository or is a maintainer you name. -## Install footprint +## Quick start -| Added | What | -| --- | --- | -| `.github/workflows/factory.yml` | About 190 lines (about 350 with the cloud sandbox), written by `/factory:setup` from the plugin's `workflow-template.yml`: a `dispatch` job, a `queue-watch` job, a `wake` job, a `work` job matrixed over ready Objectives and a `review` job (on the sandbox route, launch, image and sweep jobs instead of work and review, with two short wake jobs for the sweep). Triggers: a `*/10` cron as the backstop, manual dispatch, an issue opened or labelled `factory:objective` or any issue closed, `pull_request` (the App labels a PR `factory:review`, a human marks a Question draft ready, a Factory PR merges, the merge queue drops one), and CI finishing on a `factory/` branch (`workflow_run`, `check_suite`). The worker, review and launch jobs fire the next dispatch at their end, and a `wake` job sleeps to the next deadline only time ends; on the sandbox route the sweep arms its own wakes of minutes and the dispatcher's tick runs an owed image build (docs/PLAN.md). Never from a fork. | -| A Factory GitHub App | Two secrets, `FACTORY_APP_ID` and `FACTORY_APP_PRIVATE_KEY`. Permissions, one list: contents, pull requests, issues, projects: write; checks, commit statuses: read; actions: read, optional (cloud-sandbox logs and session restore); nothing else, never workflows. Checks and commit statuses are how Factory reads required checks; actions read lets a cloud-sandbox stage, which has no job token, read a red job's log. Every push, PR, label, board move and enqueue uses its short-lived installation token, minted for this one repository. | -| A model credential | Claude: `ANTHROPIC_API_KEY` (or workload identity federation with `id-token: write`). Codex: `FACTORY_CODEX_GATEWAY_KEY`, a Vercel AI Gateway key passed as `codex_gateway_key` (Codex then runs through the gateway and bills its credits; it wins over `CODEX_API_KEY`), or `CODEX_API_KEY`; either passed only to the harness step. Copilot: `COPILOT_GITHUB_TOKEN`, a fine-grained token with only the Copilot Requests permission from a user with a Copilot seat, passed as `copilot_github_token` to the harness step only and billed to that seat; doctor checks it cannot write to the repository. `credential: plan` with `CLAUDE_CODE_OAUTH_TOKEN` is allowed for your own repository only. | -| Merge queue, branch protection, label, Project board | GitHub settings, not files. `node /plugin/scripts/setup.mjs --repo OWNER/NAME --yes --ci-checks ""`, run in your repository's clone, applies them with your `gh` login (rulesets need admin) and creates the board (five columns) and the labels. Private repositories outside Enterprise Cloud cannot have a merge queue; Factory runs them on strict required checks and sends a PR that fell behind or conflicts back to the worker once. | -| A CI workflow on `pull_request` and `merge_group` | Most repositories have one; add `merge_group:` under `on:` if yours lacks it. | +You need: + +- a GitHub repository you administer, with a CI workflow on pull requests; +- the [`gh` CLI](https://cli.github.com), logged in, and Node 22 or newer; +- a model credential: an Anthropic API key, an OpenAI key or Vercel AI Gateway key for Codex, or a GitHub Copilot seat. + +**1. Install the plugin** in Claude Code or Codex: + +```sh +# Claude Code +claude plugin marketplace add https://github.com/clockgrove/factory.git +claude plugin install factory@clockgrove + +# Codex +codex plugin marketplace add clockgrove/factory +codex plugin add factory@clockgrove +``` + +**2. Set up your repository.** Open a session in your repository's clone and run: + +``` +/factory:setup +``` + +It shows each change and waits for your yes. It writes `.github/workflows/factory.yml`, walks you through creating a GitHub App for this one repository, has you save the secrets, and applies the labels, the Project board and the merge queue. + +**3. Check it.** Each line is PASS, WARN or FAIL with the fix; it changes nothing: + +``` +/factory:doctor +``` + +**4. File your first Objective.** Commit the workflow file to your default branch. Then open an issue like this one and add the `factory:objective` label: + +```markdown +## Goal + +`mytool --version` prints the version. -The job token gets `contents: read`, `issues: write`, `actions: read` (usage artifacts and the CI workflow's triggers), and on the jobs that tick the dispatcher `actions: write` (work and review also use it to rerun flaky CI jobs); it never writes code. The worker runs on an ephemeral runner; what persists is the branch, the PR, the plan comment and a usage artifact. Cost is capped by `max_usd` per job and `daily_usd` per day; Claude (`--max-budget-usd`) and Copilot (`--max-ai-credits`) enforce the per-job cap inside the run; Codex is checked after each run. Secrets are scrubbed from every non-harness child and masked in the uploaded event log. The harness holds no GitHub write access: on Actions it runs as its own uid with a read-only token (the runner needs passwordless sudo, as GitHub's hosted runners have); Factory makes every write. Set your repository's default workflow token to read-only: doctor requires it. +## Acceptance -## The plugin and the Action are one tree +- `mytool --version` prints the version from package.json and exits 0. +- A test covers it. +``` + +One comment on the issue shows the run's progress, and the labels show its state. [docs/USAGE.md](docs/USAGE.md) covers writing Objectives, following a run and answering a Question. -`plugin/` holds the skills the worker and reviewer follow, the agent definitions (model pinned for Claude Code and Codex; Copilot's only when `subagent_model` or `reviewer_model` is set), the task card, the scripts and the workflow template. The Action (`action.yml` at the repository root) runs those scripts from beside it; `/factory:run ` in your own Claude Code or Codex session runs the same worker on your login. Install the plugin with `claude plugin marketplace add https://github.com/clockgrove/factory.git` then `claude plugin install factory@clockgrove` (Codex: `codex plugin marketplace add clockgrove/factory`, `codex plugin add factory@clockgrove`). Commands: `/factory:run`, `/factory:serve`, `/factory:review`, `/factory:score`, `/factory:queue`, `/factory:doctor`, `/factory:setup`. The marketplaces install the plugin from `main`, so a session's commands follow `main`, not the tag your Action pins. +### Without the plugin -## Check and run +The same setup runs as scripts from a clone of Factory: ```sh +git clone --depth 1 --branch "$(gh release view --repo clockgrove/factory --json tagName --jq .tagName)" \ + https://github.com/clockgrove/factory.git ~/factory + # in your repository's clone -node /plugin/scripts/doctor.mjs --repo OWNER/NAME --checkout . # PASS/WARN/FAIL per check with the fix; changes nothing -node /plugin/scripts/setup.mjs --repo OWNER/NAME --yes --ci-checks "quality" # GitHub fixes and the workflow file -gh workflow run factory.yml -f issue=811 # one Objective now; the cron and the ticks do the rest +node ~/factory/plugin/scripts/setup.mjs --repo OWNER/NAME --ci-checks "" # preview: prints each step +node ~/factory/plugin/scripts/setup.mjs --repo OWNER/NAME --ci-checks "" --yes # apply +node ~/factory/plugin/scripts/doctor.mjs --repo OWNER/NAME --checkout . ``` -Each job uploads `factory-usage-` (and `factory-usage--review`) with `events.jsonl` and `result.json` (`{harness, harnessVersion, model, usd, tokens: {input, output, cached, written}, models, basis, exit, turns, …}`); `bench/score.mjs` turns them, the progress comments and the PR into the scorecard: walk-away, quality, operator minutes, cost by component, minutes per phase, turns, runs and sessions. +The scripts do not create the GitHub App or save secrets: [docs/INSTALL.md](docs/INSTALL.md) lists both. + +### Try it on your own machine first -## Inputs that tune cost +Local mode runs the same worker and reviewer on your machine, on your own harness login, with no GitHub App: `factory serve`. It needs Linux or WSL2; on macOS it runs Claude Code and Codex only, with the sandbox verified but not run end to end. See [docs/LOCAL.md](docs/LOCAL.md). -Seven optional inputs on the Action, all off or at the harness default unless set: `effort` (the worker's effort level, for every harness, its answer pass after the review included; pinned to `medium` for Claude so a CLI default change cannot move cost silently), `reviewer_effort` (the reviewer's effort level, passed on every reviewer call; `high` for Codex as its reviewer agent file says, Claude and Copilot keep their CLI default unless you set one; the job log says `reviewer on at ` and the run's event record carries it), `compaction_window` (the context size in tokens at which the worker and reviewer sessions compact; Claude and Codex, since Copilot has no such setting; the CLI default when empty, a measured lever for large Objectives), `resume_policy` (`fresh` by default: a review fix pass, an answer pass and a conflicting rebase start a fresh session; `fix-passes` resumes the worker's saved session for them, which on Objectives of about 30 turns cost three to seven times more at equal outcomes, so it is for adopters measuring larger ones; every harness, on Actions and in local mode), `prompt_cache_ttl` (Claude only; `off`, the CLI default, until measured: `auto` asks Claude Code for a one-hour prompt cache on the main conversation when the run bills an API key, which otherwise gets five minutes and so re-writes the prefix after every CI wait; `1h` or `5m` force one; the usage record's `cache` shows the writes after each wait, see bench/README.md), `harness_version` (the npm version of the harness CLI; `latest` by default, and the version actually installed is recorded in the usage record), and `allowed_domains` (the hosts the harness's sandboxed commands (Claude Code, Codex or Copilot) may reach on Actions and in a cloud sandbox beyond GitHub and the npm and Yarn registries, comma or space separated, `*.example.com` for its subdomains: the counterpart of local mode's `FACTORY_LOCAL_DOMAINS`; docs/PLAN.md, Credentials and billing). Every Claude call also shares one cached system-prompt prefix across runs. +## What Factory adds to your repository -## Where Factory runs +- **One workflow file**, `.github/workflows/factory.yml`. +- **A GitHub App** you create and install on this one repository, with two secrets. Its permissions, one list: contents, pull requests, issues, projects: write; checks, commit statuses: read; actions: read, optional (cloud-sandbox logs and session restore); nothing else, never workflows. +- **A model credential**, saved as a secret and passed to the agent's process only. +- **GitHub settings:** the `factory:` labels, a Project board and a `factory` ruleset (merge queue and required checks). -Every stage runs behind three interfaces: trigger, sandbox and gate (`plugin/scripts/lib/backend.mjs`). GitHub is the only state on every backend: the branch, the PR, the issue's comments and labels, the board and the usage artifact. +Nothing else is installed. [docs/INSTALL.md](docs/INSTALL.md) has the full footprint, the cost inputs, and how to upgrade and uninstall. -- **Actions (the default):** every job is a fresh hosted runner. It installs the adopter's toolchain, runs the harness, pushes, waits for the required checks, and is discarded. The install is this Action plus the App's two secrets. -- **Local mode:** the same worker and reviewer on a developer's own machine, started by `factory serve` or `factory run ` (the plugin's `bin/`; `/factory:serve` and `/factory:run` in a session). The harness runs on its own login, and every command the model or Factory runs goes through a sandbox. That means no credential stores, writes only in the Objective's clone, and the network limited to GitHub and an allow-list. Its GitHub reach is one fine-grained token for this repository (plus, optionally, the Factory App's key for Blocked and Question notices). An Objective labelled `factory:local` is the local loop's, and the Action leaves it alone. Linux and WSL2 with Claude Code, Codex and Copilot; macOS with Claude Code and Codex only (no Copilot), sandbox verified, not run end to end. See [docs/LOCAL.md](docs/LOCAL.md). -- **Cloud sandbox (opt-in):** one pausable cloud sandbox per Objective on Vercel Sandbox, started from an image built by the adopter's own setup command. Short Action jobs start each stage in it on GitHub events and exit within about thirty seconds; the sandbox stops at every wait between stages, and its cost is booked beside the model's on the progress comment. Turn it on with setup's `--sandbox vercel` and the `VERCEL_TOKEN`, `VERCEL_TEAM_ID` and `VERCEL_PROJECT_ID` secrets. Limits: a stage runs in one sandbox session, which Factory asks to be `timeout_minutes` plus about 16 minutes; Vercel Pro grants up to 24 hours, and on a plan that caps sessions lower the stage is cut to the session granted and Blocks naming the cap if it outruns it; Claude Code, Copilot and Codex (through the Vercel AI Gateway) are proven there. See [docs/ADAPTERS.md](docs/ADAPTERS.md). +## Where it runs + +- **GitHub Actions (the default):** every job is a fresh hosted runner. +- **Local mode:** your own machine, started by `factory serve` or `factory run ` ([docs/LOCAL.md](docs/LOCAL.md)). +- **Cloud sandbox (opt-in):** one pausable sandbox per Objective on Vercel Sandbox ([docs/ADAPTERS.md](docs/ADAPTERS.md)). + +GitHub is the only state on every route: the branch, the PR, the issue's comments and labels, the board and the usage artifact. There is no server or database to host. + +## Documentation + +| Page | What it covers | +| --- | --- | +| [docs/USAGE.md](docs/USAGE.md) | Writing an Objective, following a run, Blocked and Question, the commands | +| [docs/INSTALL.md](docs/INSTALL.md) | The install footprint, credentials, cost inputs, upgrade and uninstall | +| [docs/LOCAL.md](docs/LOCAL.md) | Local mode: platforms, setup, sandboxes, settings | +| [docs/ADAPTERS.md](docs/ADAPTERS.md) | The cloud sandbox and the other third-party adapters | +| [docs/PLAN.md](docs/PLAN.md) | The design: rules, contracts, security, the 1.0 bar | +| [bench/README.md](bench/README.md) | The scorecard: walk-away, quality, operator minutes, cost | +| [Releases](https://github.com/clockgrove/factory/releases) | Release notes for every tag | ## Versioning Factory is at 0.9.x, the release candidate for 1.0: run it on real work. The surface below is expected to hold, and 0.9.x releases are expected to be fixes underneath it, but there is no stability promise until 1.0. - **Pin an exact tag** (`clockgrove/factory@vX.Y.Z`). Factory publishes no floating tag (no `v0`, no `v0.9`, no `v1`), and a ruleset keeps release tags from being moved or deleted. Reason: a run's behaviour is its tag's code, so nothing changes under you between runs. -- **Proposed to freeze at 1.0:** the Action's input names, the labels (`factory:objective`, `factory:running`, `factory:in-review`, `factory:question`, `factory:blocked`, `factory:review`, `factory:local`, `factory:image`, `P0` to `P2`), the markers Factory writes on issues and PRs, the App's permission set (Install footprint) and the workflow template's jobs (`dispatch`, `queue-watch`, `work`, `review`; on the sandbox route `launch`, `launch-gate`, `launch-event`, `launch-tick`, `image`, `sweep`). From 1.0, a 1.x release renames or removes none of them and asks the App for no new permission; it may add an optional input. Before 1.0, a 0.9.x release may still change one of them as the exception, and its release notes call the change out. +- **Proposed to freeze at 1.0:** the Action's input names, the labels (`factory:objective`, `factory:running`, `factory:in-review`, `factory:question`, `factory:blocked`, `factory:review`, `factory:local`, `factory:image`, `P0` to `P2`), the markers Factory writes on issues and PRs, the App's permission set (above) and the workflow template's jobs (`dispatch`, `queue-watch`, `work`, `review`; on the sandbox route `launch`, `launch-gate`, `launch-event`, `launch-tick`, `image`, `sweep`). From 1.0, a 1.x release renames or removes none of them and asks the App for no new permission; it may add an optional input. Before 1.0, a 0.9.x release may still change one of them as the exception, and its release notes call the change out. - **What makes 1.0:** five consecutive real Objectives merged with zero operator input, pooled across `clockgrove/clockgrove`, `clockgrove/assetgrove` and `clockgrove/raccoon-world` (never the scratch drill repositories), ordered by when each ended, on any harness; any miss on any of them resets it. One scorer command counts it: [docs/PLAN.md](docs/PLAN.md) (The 1.0 bar). -- **Supported:** the latest 0.9.x release gets security fixes; older tags do not. Report privately ([SECURITY.md](SECURITY.md)). - -## Upgrade - -1. Read the release notes between your tag and the new one. -2. Re-render the workflow with that release's setup: in your repository's clone, `node /plugin/scripts/setup.mjs --repo OWNER/NAME --apply workflow --action clockgrove/factory@vX.Y.Z --yes`. It keeps your settings, moves the pin and brings the template's jobs to that release. Re-render on every upgrade: the template belongs to its release, and an older file misses jobs the new Action expects (doctor names some, such as `merge ticks`). -3. Commit the file on the default branch, then run doctor. +- **Supported:** the latest 0.9.x release gets security fixes; older tags do not. -## Contributing +## Help and contributing -No outside pull requests until the owner opens the repository to them (reaching 1.0 does not); issues and private security reports are welcome ([CONTRIBUTING.md](CONTRIBUTING.md), [SECURITY.md](SECURITY.md)). +- **Bugs, questions and ideas:** [open an issue](https://github.com/clockgrove/factory/issues). [CONTRIBUTING.md](CONTRIBUTING.md) says what helps in a report. +- **Security problems:** report them privately ([SECURITY.md](SECURITY.md)), never in a public issue. +- **Pull requests:** Factory takes no outside pull requests until its owner opens the repository to them; reaching 1.0 does not. -## Uninstall +## License -Delete `.github/workflows/factory.yml`, the App and its two secrets, and any model key. Optionally delete the `factory` ruleset, the labels and the Project. For local mode, also delete `~/.config/factory` (the tokens), `~/.cache/factory`, the `.factory/` directory in each clone that ran it, and the `factory:local` label. For the cloud sandbox, also delete the three `VERCEL_*` secrets, the Vercel project's sandboxes and snapshots, the image issue and the `factory:image` label. Nothing else is installed. +[MIT](LICENSE) © Clockgrove diff --git a/docs/INSTALL.md b/docs/INSTALL.md new file mode 100644 index 0000000..9293797 --- /dev/null +++ b/docs/INSTALL.md @@ -0,0 +1,93 @@ +# Install, tune, upgrade, uninstall + +What Factory adds to a repository, how to set it up and check it, the inputs that tune cost, and how to upgrade or remove it. The README has the short version; [USAGE.md](USAGE.md) covers filing and following Objectives, and [LOCAL.md](LOCAL.md) covers local mode. + +## Requirements + +- A GitHub repository you administer (rulesets need admin), with a CI workflow that runs on `pull_request`. +- The `gh` CLI, logged in, and Node 22 or newer on the machine you run setup from. +- A model credential for the harness you choose: Claude Code, Codex or GitHub Copilot (the table below). +- A merge queue is used where GitHub offers one: public repositories and Enterprise Cloud. Elsewhere Factory runs on strict required checks (the table below). + +## Where Factory runs + +Every stage runs behind three interfaces: trigger, sandbox and gate (`plugin/scripts/lib/backend.mjs`). GitHub is the only state on every backend: the branch, the PR, the issue's comments and labels, the board and the usage artifact. + +- **Actions (the default):** every job is a fresh hosted runner. It installs the adopter's toolchain, runs the harness, pushes, waits for the required checks, and is discarded. The install is this Action plus the App's two secrets. +- **Local mode:** the same worker and reviewer on a developer's own machine, started by `factory serve` or `factory run ` (the plugin's `bin/`; `/factory:serve` and `/factory:run` in a session). The harness runs on its own login, and every command the model or Factory runs goes through a sandbox. That means no credential stores, writes only in the Objective's clone, and the network limited to GitHub and an allow-list. Its GitHub reach is one fine-grained token for this repository (plus, optionally, the Factory App's key for Blocked and Question notices). An Objective labelled `factory:local` is the local loop's, and the Action leaves it alone. Linux and WSL2 with Claude Code, Codex and Copilot; macOS with Claude Code and Codex only (no Copilot), sandbox verified, not run end to end. See [LOCAL.md](LOCAL.md). +- **Cloud sandbox (opt-in):** one pausable cloud sandbox per Objective on Vercel Sandbox, started from an image built by the adopter's own setup command. Short Action jobs start each stage in it on GitHub events and exit within about thirty seconds; the sandbox stops at every wait between stages, and its cost is booked beside the model's on the progress comment. Turn it on with setup's `--sandbox vercel` and the `VERCEL_TOKEN`, `VERCEL_TEAM_ID` and `VERCEL_PROJECT_ID` secrets. Limits: a stage runs in one sandbox session, which Factory asks to be `timeout_minutes` plus about 16 minutes; Vercel Pro grants up to 24 hours, and on a plan that caps sessions lower the stage is cut to the session granted and Blocks naming the cap if it outruns it; Claude Code, Copilot and Codex (through the Vercel AI Gateway) are proven there. See [ADAPTERS.md](ADAPTERS.md). + +The rest of this page is the Actions route; the other two have their own pages. + +## Install footprint + +| Added | What | +| --- | --- | +| `.github/workflows/factory.yml` | About 190 lines (about 350 with the cloud sandbox), written by `/factory:setup` from the plugin's `workflow-template.yml`: a `dispatch` job, a `queue-watch` job, a `wake` job, a `work` job matrixed over ready Objectives and a `review` job (on the sandbox route, launch, image and sweep jobs instead of work and review, with two short wake jobs for the sweep). Triggers: a `*/10` cron as the backstop, manual dispatch, an issue opened or labelled `factory:objective` or any issue closed, `pull_request` (the App labels a PR `factory:review`, a human marks a Question draft ready, a Factory PR merges, the merge queue drops one), and CI finishing on a `factory/` branch (`workflow_run`, `check_suite`). The worker, review and launch jobs fire the next dispatch at their end, and a `wake` job sleeps to the next deadline only time ends; on the sandbox route the sweep arms its own wakes of minutes and the dispatcher's tick runs an owed image build (docs/PLAN.md). Never from a fork. | +| A Factory GitHub App | Two secrets, `FACTORY_APP_ID` and `FACTORY_APP_PRIVATE_KEY`. Permissions, one list: contents, pull requests, issues, projects: write; checks, commit statuses: read; actions: read, optional (cloud-sandbox logs and session restore); nothing else, never workflows. Checks and commit statuses are how Factory reads required checks; actions read lets a cloud-sandbox stage, which has no job token, read a red job's log. Every push, PR, label, board move and enqueue uses its short-lived installation token, minted for this one repository. | +| A model credential | Claude: `ANTHROPIC_API_KEY` (or workload identity federation with `id-token: write`). Codex: `FACTORY_CODEX_GATEWAY_KEY`, a Vercel AI Gateway key passed as `codex_gateway_key` (Codex then runs through the gateway and bills its credits; it wins over `CODEX_API_KEY`), or `CODEX_API_KEY`; either passed only to the harness step. Copilot: `COPILOT_GITHUB_TOKEN`, a fine-grained token with only the Copilot Requests permission from a user with a Copilot seat, passed as `copilot_github_token` to the harness step only and billed to that seat; doctor checks it cannot write to the repository. `credential: plan` with `CLAUDE_CODE_OAUTH_TOKEN` is allowed for your own repository only. | +| Merge queue, branch protection, label, Project board | GitHub settings, not files. `node /plugin/scripts/setup.mjs --repo OWNER/NAME --yes --ci-checks ""`, run in your repository's clone, applies them with your `gh` login (rulesets need admin) and creates the board (five columns) and the labels. Private repositories outside Enterprise Cloud cannot have a merge queue; Factory runs them on strict required checks and sends a PR that fell behind or conflicts back to the worker once. | +| A CI workflow on `pull_request` and `merge_group` | Most repositories have one; add `merge_group:` under `on:` if yours lacks it. | + +What each credential can do: + +- **The job token:** `contents: read`, `issues: write`, `actions: read` (usage artifacts and the CI workflow's triggers), and on the jobs that tick the dispatcher `actions: write` (work and review also use it to rerun flaky CI jobs); it never writes code. Set your repository's default workflow token to read-only: doctor requires it. +- **The harness:** it holds no GitHub write access: on Actions it runs as its own uid with a read-only token (the runner needs passwordless sudo, as GitHub's hosted runners have); Factory makes every write. +- **Secrets:** they are scrubbed from every non-harness child and masked in the uploaded event log. +- **Cost:** it is capped by `max_usd` per job and `daily_usd` per day; Claude (`--max-budget-usd`) and Copilot (`--max-ai-credits`) enforce the per-job cap inside the run; Codex is checked after each run. +- **What persists:** the worker runs on an ephemeral runner; what persists is the branch, the PR, the plan comment and a usage artifact. + +## Set it up + +With the plugin, `/factory:setup` in a session in your repository's clone walks every step below and waits for a yes before each change. Without it, the steps are: + +1. **The workflow file and the GitHub settings.** In your repository's clone, with a clone of Factory beside it: + + ```sh + node /plugin/scripts/setup.mjs --repo OWNER/NAME --ci-checks "" # preview: prints each step, changes nothing + node /plugin/scripts/setup.mjs --repo OWNER/NAME --ci-checks "" --yes # apply + ``` + + It writes `.github/workflows/factory.yml` pinned to the latest release, and creates the labels, the Project board and the `factory` ruleset with your `gh` login. `--apply labels|project|protection|workflow` runs one step. If your CI workflow lacks `merge_group:` under `on:`, add it. +2. **The GitHub App.** Open `https://github.com/settings/apps/new` (or your organization's `/organizations//settings/apps/new`). Uncheck webhooks and grant exactly the repository permissions in the table above. Create it, generate a private key, note its App ID, and install it on this repository only. +3. **The secrets.** `gh secret set FACTORY_APP_ID`, `gh secret set FACTORY_APP_PRIVATE_KEY < path.pem`, and your model credential from the table above. +4. **Commit the workflow file** on the default branch; its cron starts then. + +## The plugin and the Action are one tree + +`plugin/` holds the skills the worker and reviewer follow, the agent definitions (model pinned for Claude Code and Codex; Copilot's only when `subagent_model` or `reviewer_model` is set), the task card, the scripts and the workflow template. The Action (`action.yml` at the repository root) runs those scripts from beside it; `/factory:run ` in your own Claude Code or Codex session runs the same worker on your login. Install the plugin with `claude plugin marketplace add https://github.com/clockgrove/factory.git` then `claude plugin install factory@clockgrove` (Codex: `codex plugin marketplace add clockgrove/factory`, `codex plugin add factory@clockgrove`). Commands: `/factory:run`, `/factory:serve`, `/factory:review`, `/factory:score`, `/factory:queue`, `/factory:doctor`, `/factory:setup`. The marketplaces install the plugin from `main`, so a session's commands follow `main`, not the tag your Action pins. + +## Check and run + +```sh +# in your repository's clone +node /plugin/scripts/doctor.mjs --repo OWNER/NAME --checkout . # PASS/WARN/FAIL per check with the fix; changes nothing +node /plugin/scripts/setup.mjs --repo OWNER/NAME --yes --ci-checks "quality" # GitHub fixes and the workflow file +gh workflow run factory.yml -f issue=811 # one Objective now; the cron and the ticks do the rest +``` + +Each job uploads `factory-usage-` (and `factory-usage--review`) with `events.jsonl` and `result.json` (`{harness, harnessVersion, model, usd, tokens: {input, output, cached, written}, models, basis, exit, turns, …}`); `bench/score.mjs` turns them, the progress comments and the PR into the scorecard: walk-away, quality, operator minutes, cost by component, minutes per phase, turns, runs and sessions. + +## Inputs that tune cost + +Seven optional inputs on the Action, all off or at the harness default unless set. Every Claude call also shares one cached system-prompt prefix across runs. + +| Input | What it does | +| --- | --- | +| `effort` | The worker's effort level, for every harness, its answer pass after the review included; pinned to `medium` for Claude so a CLI default change cannot move cost silently. | +| `reviewer_effort` | The reviewer's effort level, passed on every reviewer call; `high` for Codex as its reviewer agent file says, Claude and Copilot keep their CLI default unless you set one; the job log says `reviewer on at ` and the run's event record carries it. | +| `compaction_window` | The context size in tokens at which the worker and reviewer sessions compact; Claude and Codex, since Copilot has no such setting; the CLI default when empty, a measured lever for large Objectives. | +| `resume_policy` | `fresh` by default: a review fix pass, an answer pass and a conflicting rebase start a fresh session; `fix-passes` resumes the worker's saved session for them, which on Objectives of about 30 turns cost three to seven times more at equal outcomes, so it is for adopters measuring larger ones; every harness, on Actions and in local mode. | +| `prompt_cache_ttl` | Claude only; `off`, the CLI default, until measured: `auto` asks Claude Code for a one-hour prompt cache on the main conversation when the run bills an API key, which otherwise gets five minutes and so re-writes the prefix after every CI wait; `1h` or `5m` force one; the usage record's `cache` shows the writes after each wait, see bench/README.md. | +| `harness_version` | The npm version of the harness CLI; `latest` by default, and the version actually installed is recorded in the usage record. | +| `allowed_domains` | The hosts the harness's sandboxed commands (Claude Code, Codex or Copilot) may reach on Actions and in a cloud sandbox beyond GitHub and the npm and Yarn registries, comma or space separated, `*.example.com` for its subdomains: the counterpart of local mode's `FACTORY_LOCAL_DOMAINS`; PLAN.md, Credentials and billing. | + +## Upgrade + +1. Read the release notes between your tag and the new one. +2. Re-render the workflow with that release's setup: in your repository's clone, `node /plugin/scripts/setup.mjs --repo OWNER/NAME --apply workflow --action clockgrove/factory@vX.Y.Z --yes`. It keeps your settings, moves the pin and brings the template's jobs to that release. Re-render on every upgrade: the template belongs to its release, and an older file misses jobs the new Action expects (doctor names some, such as `merge ticks`). +3. Commit the file on the default branch, then run doctor. + +## Uninstall + +Delete `.github/workflows/factory.yml`, the App and its two secrets, and any model key. Optionally delete the `factory` ruleset, the labels and the Project. For local mode, also delete `~/.config/factory` (the tokens), `~/.cache/factory`, the `.factory/` directory in each clone that ran it, and the `factory:local` label. For the cloud sandbox, also delete the three `VERCEL_*` secrets, the Vercel project's sandboxes and snapshots, the image issue and the `factory:image` label. Nothing else is installed. diff --git a/docs/USAGE.md b/docs/USAGE.md new file mode 100644 index 0000000..755ad86 --- /dev/null +++ b/docs/USAGE.md @@ -0,0 +1,66 @@ +# Using Factory + +How to write an Objective, what you see while it runs, what the two stops mean and how to answer them. [INSTALL.md](INSTALL.md) covers setup; [PLAN.md](PLAN.md) has the full contracts. + +## Write an Objective + +An Objective is a GitHub issue with the `factory:objective` label. The issue is the whole specification: nobody answers questions during the run. + +- **State the goal and what must be true when it is done.** Put the acceptance entries in a list under a heading such as `Acceptance` (`Done when`, `Definition of done`, `Must be true` and `Criteria` are read the same way). The reviewer judges each entry, met or not, with the line that shows it. Without such a section, the reviewer names the entries from the issue text itself. +- **Say what you have decided.** Where the issue admits more than one reading, the worker takes the simplest one that satisfies every acceptance entry and records it as an assumption in its plan and in the PR. +- **Label it yourself, or have a maintainer do it.** Factory acts on an issue only when its author and, once the body was edited, its last editor can push to the repository or are maintainers the workflow names. Otherwise the run Blocks before any model sees the text. +- **Keep it to one repository** (below). + +```markdown +## Goal + +`mytool --version` prints the version. + +## Acceptance + +- `mytool --version` prints the version from package.json and exits 0. +- A test covers it. +``` + +Ready Objectives start in priority order (`P0` to `P2`, then the lowest issue number), up to `max_concurrent` at once. An Objective too large for one session is delivered as up to six pull requests in sequence, one open at a time. + +## What you see on the issue + +While a job runs, one comment on the issue (edited in place, one per run) carries a bold status line and the phases with clocks: setup, install, harness, each validation pass, reviewer, answer pass, exit. The status line comes from the harness's own event stream, not from its prose: elapsed minutes, turns, tool calls, the sub-agents running and what each was asked to do, when the last event arrived, tokens so far, the last tool it ran. A harness that prints nothing for 30 minutes is killed and retried once, and the line says so. The labels (`factory:running`, `factory:in-review`, `factory:blocked`, `factory:question`) and the board column are the state; the plan comment is the harness's own checklist, ticked as it goes; the PR carries the validation marker and the review. + +## Blocked and Question + +Nothing asks a question during a run. The only stops are **Blocked** (a draft PR, the card in the Blocked column, one comment with the reason) and **Question** (a draft PR whose body asks the one thing a human must decide), and neither stops the queue. + +- **Blocked:** fix what the comment names, then re-apply `factory:objective` to run the Objective again. +- **Question:** comment the decision on the issue or the PR (a review or a line comment works too) and mark the PR ready: the review job applies it, and with no answer since the question the PR goes back to a Question rather than merging. Re-applying `factory:objective` instead reruns the worker with it. + +A comment counts when its author has write access to the repository or is a maintainer the workflow names (`notify`, and the optional `maintainers` list), since an App cannot always see an organization member's access. + +## Factory merges on its own + +Code lands with no human read. The gate is your CI workflow's checks on the reviewed head and the integrated commit, the comment-only reviewer pass, and the merge queue's own re-run where you have one. Set `FACTORY_AUTO_MERGE: false` for ready-for-review PRs instead. + +## Siblings that touch the same files + +Two Objectives running at once that edit the same files cost a rebase run each time one merges (ten minutes, a dollar or a few) and, without a merge queue, strict checks refuse the one left behind until the worker rebases it. When you know two Objectives share an area, say so: add the GitHub issue dependency (`blocked_by`) and the dispatcher orders them; or set `max_concurrent: 1` for the stretch. Factory does not guess file overlap from prose. + +## One repository per Objective + +Factory works in one repository per Objective. An Objective that needs a change in another repository stops after its plan with a Question on the issue, before any work: split it into one Objective per repository and re-apply the label, or answer on the issue how to do the work in this repository alone. + +## Commands + +In a Claude Code or Codex session with the plugin installed: + +| Command | What it does | +| --- | --- | +| `/factory:setup` | Installs Factory in the current repository, one consented step at a time | +| `/factory:doctor` | Checks the repository, its settings, the credential and the runner; changes nothing | +| `/factory:run ` | Runs the worker on one Objective in your own session, on your login | +| `/factory:review ` | Runs the reviewer on one Factory PR in your session | +| `/factory:serve` | Starts the local loop ([LOCAL.md](LOCAL.md)) | +| `/factory:queue` | Starts the unattended scheduler for the repository | +| `/factory:score` | Prints the scorecard: walk-away, quality, operator minutes, cost | + +On the command line, `gh workflow run factory.yml -f issue=` starts one Objective now, and local mode has `factory serve`, `factory run ` and `factory review ` ([LOCAL.md](LOCAL.md), Run it).