diff --git a/README.md b/README.md index 4cbe075..e563cc7 100644 --- a/README.md +++ b/README.md @@ -1,32 +1,65 @@ -# Mintlify Starter Kit +# Amplify Security docs -Click on `Use this template` to copy the Mintlify starter kit. The starter kit contains examples including +Documentation site for [Amplify Security](https://github.com/amplify-security), built with +[Mintlify](https://mintlify.com/). Configuration lives in `docs.json`. -- Guide pages -- Navigation -- Customizations -- API Reference pages -- Use of popular components +## Structure -### Development +The site has two tabs. **Amplify Console** is the current product, organized into three ideas that build +on each other: -Install the [Mintlify CLI](https://www.npmjs.com/package/mintlify) to preview the documentation changes locally. To install, use the following command +| Group | Covers | +| --- | --- | +| `Get Started` | Introduction, quickstart, CLI installation | +| `The Harness` (`harness/`) | The authorable primitives — agents, skills, detections — and the tool surface | +| `Context` (`context/`) | Projects, connections, vendor data, findings | +| `Working interactively` (`interactive/`) | Chat and the CLI | +| `Workflows` (`workflows/`) | Creating, triggering, running, and delivering workflow results | -``` -npm i -g mintlify +**Amplify Dashboard (Legacy)** (`legacy/`) documents the previous product and is not actively developed. + +## Development + +Install the Mintlify CLI: + +```bash +npm i -g mint ``` -Run the following command at the root of your documentation (where mint.json is) +Run the dev server from the repository root (the directory containing `docs.json`): +```bash +mint dev ``` -mintlify dev + +It serves on port 3000 by default. Use `--port` if that's taken: + +```bash +mint dev --port 3333 ``` -### Publishing Changes +## Adding a page + +1. Create the `.mdx` file in the appropriate directory, with `title` and `description` frontmatter. +2. Register its path (without the `.mdx`) in the correct group in `docs.json`. **A page not listed in + `docs.json` will not appear in the navigation.** +3. If you move or rename a page, add a `redirects` entry in `docs.json` so existing links keep working. + +## Conventions + +- Frontmatter requires `title` and `description`. +- Mintlify components in use: ``, ``, ``, ``, ``. +- Internal links are root-relative and omit the extension — `/harness/writing-an-agent`. +- Document what ships today. Where a capability is partial, say so explicitly in a `` or `` + rather than describing the intended end state. + +## Publishing -Install our Github App to auto propagate changes from your repo to your deployment. Changes will be deployed to production automatically after pushing to the default branch. Find the link to install on your dashboard. +The Mintlify GitHub App deploys automatically on push to the default branch. -#### Troubleshooting +## Troubleshooting -- Mintlify dev isn't running - Run `mintlify install` it'll re-install dependencies. -- Page loads as a 404 - Make sure you are running in a folder with `mint.json` +- **404 on every page** — you're not in the directory containing `docs.json`. +- **A new page isn't in the sidebar** — it isn't registered in `docs.json`. +- **`mint` not found** — install with `npm i -g mint`. Note the CLI is `mint`, not the older `mintlify` + package, and configuration is `docs.json`, not the older `mint.json`. diff --git a/context/connections.mdx b/context/connections.mdx new file mode 100644 index 0000000..68d5297 --- /dev/null +++ b/context/connections.mdx @@ -0,0 +1,109 @@ +--- +title: Connections +description: 'Connect your source control provider and the security and IT vendors you already run.' +--- + +## Two kinds of connection + +The **Connections** page handles two different jobs behind one interface: + +| Kind | What it enables | +| --- | --- | +| **Source control** | Cloning repositories, pull request triggers, review comments, merge gates | +| **Vendor connectors** | Reading signal from the security and IT tools you already run | + +Source control is what makes Console able to *act* on your code. Vendor connectors are what let it reason +about findings your existing stack has already produced. + +## Source control + +### GitHub + +Install the Console GitHub App for your organization. You can grant access to every repository or pick a +subset; the repositories you grant become [projects](/context/projects). + +The App is what powers: + +- Cloning repositories into sandboxes +- [Pull request triggers](/workflows/triggers#pull-request-triggers) firing on open, reopen, and new commits +- [Review comments](/workflows/outputs#comment-on-triggering-pull-request) +- [Merge gates](/workflows/outputs#gate-merging-on-security-review) as required checks +- Opening a pull request from a [finding's suggested fix](/context/findings#accepting-a-fix) + +See [Installation](/install-console#install-the-github-app) for the install flow. + +### GitLab + +Connect a GitLab instance by providing its host URL, the project, and a personal access token. + + + GitLab supports cloning and merge request comments. **Merge blocking is not supported yet** — a + [merge gate](/workflows/outputs#gitlab) on a GitLab run is recorded as skipped. + + +## Vendor connectors + +Console connects to the security and IT tools you already run, so agents can reason about the findings +already sitting in them. + +Open **Connections**, add a connection, and search the connector catalog. Connectors are grouped by +category: + +| Category | Covers | +| --- | --- | +| **EDR** | Endpoint detection and response | +| **AppSec** | Application security scanners — SAST, SCA, and similar | +| **VMS** | Vulnerability management | +| **CSPM** | Cloud security posture management | +| **IDP** | Identity providers | +| **GRC** | Governance, risk, and compliance | +| **TPRM** | Third-party risk management | +| **ITSM** | IT service management and ticketing | + +The catalog is searchable by vendor name or slug, and a connector spanning more than one category appears +under each — AWS Inspector shows up under both AppSec and VMS, for instance. + +Completing a connection walks you through that vendor's own authorization flow. Console stores the resulting +connection and never handles the underlying vendor API key directly — requests are brokered server-side. + + + Connecting a vendor and *reading from it* are two different things. A connection can be established for any + connector in the catalog, but what an agent can query today is scoped to **vulnerability findings**. An + agent can read your Snyk or Semgrep backlog; it cannot read Jira issues or Cloudflare configuration. + + Read [what agents can read](/context/vendor-data) before designing a workflow around a connection. + + +## Which connections do you need? + +| If you want to | You need | +| --- | --- | +| Ask an agent about your code | A source-control connection | +| Run workflows on pull requests | GitHub | +| Block merges on security review | GitHub | +| Triage an existing scanner backlog against real code | An AppSec or VMS connector | +| Just use the CLI on local code | Nothing — the CLI reads your working directory | + +## Troubleshooting + +**A repository isn't available as a project.** The GitHub App installation probably doesn't include it. +Adjust the installation's repository access. + +**An output shows Failed on a run.** Confirm the App is still installed for that repository and has +permission to write checks and pull request comments. See +[checking delivery](/workflows/outputs#checking-delivery). + +**The connector catalog is empty or won't load.** The catalog is fetched when you open the dialog, so a +transient upstream failure shows there without affecting the rest of the page. Retry. + +## Next steps + + + + The exact vendor-data surface today. + + + + Turn a connection into analyzable code. + + diff --git a/context/findings.mdx b/context/findings.mdx new file mode 100644 index 0000000..c6b5005 --- /dev/null +++ b/context/findings.mdx @@ -0,0 +1,99 @@ +--- +title: Findings +description: 'Confirmed issues with the agent’s reasoning, the affected code, and often a ready-to-merge fix.' +--- + +## What a finding is + +A finding is a vulnerability an agent has confirmed and recorded. It's the durable output of agent work — +what survives after the conversation or run is over. + +Every finding carries: + +- **Where** — the affected file, and the symbol it's anchored to +- **What** — the vulnerability, its classification, and severity +- **Why** — the agent's reasoning for believing it's genuinely exploitable +- **The fix** — a suggested patch, when one was generated +- **Provenance** — the run that produced it, and the [detection](/harness/detections) behind it if there was one + +The reasoning is the part worth reading. A pattern match tells you a shape appeared; a finding tells you why +the agent believes an attacker can actually reach it. That's what makes it triageable. + +## Where findings come from + +| Source | How | +| --- | --- | +| **Workflow runs** | A scanner or detections step confirms an issue and calls `report_finding` | +| **Chat** | An agent investigating on your behalf records what it confirms | +| **Detections** | A stored rule matches, and the finding links back to the rule | +| **The CLI** | Local scans record findings, and sync to your organization when authenticated | + +## Deduplication + +Findings are deduplicated per project by identity. The same issue rediscovered on a later run updates the +existing finding rather than creating a second one, so a long-lived issue doesn't inflate your counts and run +history stays readable. + +Findings are anchored to a **symbol** rather than only a line number, so a finding survives edits that shift +line numbers around it. + +## Closing the loop + +Two actions on a finding detail page turn a report into progress. + +### Accepting a fix + +When a finding has a generated patch, the suggested fix section can open a pull request with that patch +applied. You review it like any other pull request. + + + Accepting a fix opens pull requests through the GitHub API, so it requires a GitHub-connected project. + Other providers can't do this yet, and the action is unavailable when a project has no code host connected. + + +Patches aren't guesses. The agent that generated one edited the file, re-ran the detection to confirm the +match was gone, and captured the resulting diff — for policy detections, an independent read-only verifier +judged the result. You're reviewing a change that has already been checked, not a proposal. + +### Starting a chat + +**Start a chat** opens a session with the finding's context already loaded — the affected code, the agent's +reasoning, and the generated patch if there is one. You type your question in the same step. + +Use it to ask the things a report can't anticipate: *is this actually reachable in production?*, *what else +in this codebase has the same problem?*, *why is this the right fix?* + +The session is titled after the finding's file and line, so it's easy to find later. + + + This also requires a project connected to a code host — the agent needs to load the code to answer. + + +## From finding to detection + +The highest-leverage move on a finding is turning it into a rule. A finding is one instance; a +[detection](/harness/detections) catches the whole class from then on, in every repository, without an agent +re-deriving it. + +Ask in chat, or add [`detection-author`](/harness/agent-library#detections) as a workflow step to do it +automatically for every finding a scan produces. + +## Delivering findings automatically + +Findings don't have to be pulled — a workflow can push them where your team already works: + +- [Comment on the triggering pull request](/workflows/outputs#comment-on-triggering-pull-request), with links + back to the full finding. +- [Gate merging](/workflows/outputs#gate-merging-on-security-review) until review passes or a human approves. + +## Next steps + + + + Catch the class, not the instance. + + + + Comments and merge gates. + + diff --git a/context/overview.mdx b/context/overview.mdx new file mode 100644 index 0000000..24b5b48 --- /dev/null +++ b/context/overview.mdx @@ -0,0 +1,71 @@ +--- +title: Context +description: 'The harness is only as good as what it can see — repositories, and the security tools you already run.' +--- + +## Why context is its own layer + +An agent that can't see your environment is guessing. Console's agents don't answer from a static index +or a model's memory of open-source code — they read your actual files, run commands against your actual +repository, and query the findings your existing tools have already produced. + +That means the quality of everything downstream depends on what you've connected. + +## Two kinds of context + +| Kind | What it gives the agent | Set up in | +| --- | --- | --- | +| **[Projects](/context/projects)** | Your code. The repositories Console clones and reads. | Projects | +| **[Connections](/context/connections)** | Your source control provider, and signal from the security and IT vendors you already run. | Connections | + +Projects are what an agent reads *from disk*. Connections are what it queries *over the network* — +including the pull request events that let a workflow fire on its own. + + + These are prerequisites, not optional extras. Chat can't analyze a repository that isn't connected, and + a workflow has nothing to run against without at least one project. + + +## Two ways to direct it + +Having context isn't the same as using it. Console gives you an interactive path and a declarative one: + +- **[Chat](/interactive/chat)** — point an agent at something and ask. Best when you don't yet know what + you're looking for, or when the question is a one-off. +- **[Workflows](/workflows/overview)** — declare the repositories, the agent chain, and where results go, + then let it run unattended. Best when the task is worth repeating. + +The usual progression is to explore a question in chat, and once the approach works, promote it to a +workflow. + +## What accumulates + +As agents work, results collect in your organization rather than evaporating with the conversation: + +- **[Findings](/context/findings)** — confirmed issues, each with the agent's reasoning, the affected + code, and often a suggested fix. +- **[Detections](/harness/detections)** — the rules you've built up, which later runs apply + automatically. + +Both become context in their own right. A finding can seed a new chat; detections turn a one-time +discovery into a permanent check. + +## Next steps + + + + Give agents code to read. + + + + Source control, plus the security stack you already run. + + + + Exactly what a connection exposes today. + + + + Where confirmed results land. + + diff --git a/context/projects.mdx b/context/projects.mdx new file mode 100644 index 0000000..3e3f0dd --- /dev/null +++ b/context/projects.mdx @@ -0,0 +1,73 @@ +--- +title: Projects +description: 'Connect the repositories Console clones, reads, and analyzes.' +--- + +## What a project is + +A project is a repository Console can reach. It carries a name in `owner/repo` form, a clone URL, and the +source-control connection that authorizes access. + +Projects are the unit of targeting everywhere else in the product: + +- **Chat** clones a project on demand so an agent can read its code. +- **Manual workflow runs** ask which projects to run against. +- **Pull request triggers** watch a set of projects. +- **Findings** are recorded against a project, and deduplicated within it. + +## Connecting a repository + +Projects come from a source-control [connection](/context/connections). Install the Console GitHub App for +your organization — granting access to all repositories or a chosen subset — or connect a GitLab instance, +and the repositories become available as projects. + +See [Connections](/context/connections#source-control) for both flows. + + + Access is inherited from the connection. If a repository isn't showing up, the usual cause is that the + GitHub App installation doesn't include it — adjust the installation's repository access rather than + looking for a per-project setting. + + +## How code reaches an agent + +Console never analyzes code in place. Each run gets an isolated sandbox and clones the repository into it, +which is why an agent can run builds and execute commands without touching anything of yours. + +What gets cloned depends on how the run started: + +| Run type | Cloned | +| --- | --- | +| Chat | The project you're asking about | +| Manual workflow run | Each selected project, at its default branch — or at a [git ref](/workflows/running#run-a-workflow-manually) you pin | +| Pull request run | The pull request's **head** — the proposed code, not the base branch | + +Agents can also discover and clone projects themselves through `list_projects` and `clone_project`, which is +how a chat session pulls in a repository mid-conversation. + +## Local code and the CLI + +The CLI is the exception: it runs against **the directory you started it in**, with no project or clone +involved. That makes it the right tool for code you're actively editing, including uncommitted changes that +don't exist in any repository yet. + +Projects still matter to the CLI when it's authenticated, since detections and findings sync to your +organization. See [The CLI](/interactive/cli). + +## Findings are scoped per project + +Findings are deduplicated within a project by identity, so the same issue rediscovered on a later run +updates the existing finding instead of creating a second one. Run history stays readable and a recurring +issue doesn't inflate your counts. + +## Next steps + + + + Set up source control and vendor connectors. + + + + Start a chat against a connected project. + + diff --git a/context/vendor-data.mdx b/context/vendor-data.mdx new file mode 100644 index 0000000..b37eea8 --- /dev/null +++ b/context/vendor-data.mdx @@ -0,0 +1,126 @@ +--- +title: What agents can read +description: 'Exactly what a vendor connection exposes to an agent today — and what it does not.' +--- + +## The short version + +Once a vendor is connected, an agent can do two things: **discover your connections**, and **query +normalized vulnerability findings** from them. + +That's the whole surface today. It's genuinely useful — it's what lets an agent triage an existing scanner +backlog against your real code — but it's narrower than the connector catalog, so it's worth being precise +before you design a workflow around it. + +## What an agent can do + +| Capability | Tool | +| --- | --- | +| List the connections configured for your organization | `list_leen_connections` | +| Query vulnerability findings from a connection, with filters and pagination | `get_leen_vulnerability_findings` | +| Fetch one vendor finding in full | `get_leen_vulnerability_finding` | + +An agent discovers connections first, then queries the one it wants — the same order you'd use by hand. + +### What a vendor finding carries + +Findings are **normalized**, so the shape is the same whether they came from Snyk, Semgrep, or an EDR: + +| Field | Notes | +| --- | --- | +| `title`, `description` | What the vendor reported | +| `severity` | `CRITICAL`, `HIGH`, `MEDIUM`, `LOW`, `INFO` | +| `state` | `OPEN`, `CLOSED`, `REOPENED`, `IGNORED` | +| `type` | The finding's classification | +| `has_fix` | Whether a fix is available | +| `remediation` | The vendor's suggested remediation | +| `first_seen`, `last_seen` | Timestamps | +| `product` | The product the finding belongs to | +| `vulnerabilities` | Associated CVE knowledge-base entries | + +Filter by severity and state, restrict to findings that do or don't have a fix, and page through large +result sets with a cursor. Heavier vendor-specific payloads are deliberately dropped — they'd crowd an +agent's context without improving its reasoning. + +## What an agent cannot do + + + The vendor surface is scoped to **vulnerability findings**. Connecting a vendor from another category + makes the connection available, but does not give agents a way to read that vendor's other data. + + +Concretely, today an agent **cannot**: + +- Read Jira or other ITSM issues, or create tickets in them +- Read cloud or CSPM configuration state +- Read identity provider users, groups, or policies +- Read GRC controls or evidence +- Write anything back to a vendor — update a finding's state, comment, or close it + +There is also no general-purpose outbound write tool. `web_fetch` performs reads only, with no request method +or body, so an agent cannot use it to POST to a vendor API. Console's outbound writes go through +[workflow outputs](/workflows/outputs), which today deliver to pull requests and GitHub checks. + +## What this is good for + +The one thing this surface does well is worth building on, because nothing else in your stack does it: +**deciding which of your existing findings actually matter.** + +A scanner tells you a vulnerable function exists in a dependency. It usually can't tell you whether your code +ever calls it. An agent with both your vendor findings *and* your source can: + +``` +Pull the open CRITICAL and HIGH findings from our Snyk connection. For each one, +trace whether the vulnerable code path is actually reachable from an entrypoint in +this repository. Report the reachable ones as findings and tell me which are noise. +``` + +That turns a backlog of hundreds into a list of the few that are real — using the vendor for breadth and the +harness for judgment. + +Other things this supports well: + +- **Cross-referencing tools.** Ask whether two scanners agree, and where they disagree and why. +- **Explaining a finding in context.** Take a terse vendor finding and have an agent explain what it means in + your codebase specifically. +- **Prioritizing by real exposure** rather than by CVSS alone, by checking what's actually deployed and + reachable. + +## Using it in a workflow + +Because these are ordinary tools, an agent you write can use them in any workflow step. Grant the tools in +[`allowed-tools`](/harness/writing-an-agent#allowed-tools-restricts-it-doesnt-grant) and describe the job: + +```markdown +--- +name: backlog-triage +description: Triages open vendor vulnerability findings against this repository's code and reports which are genuinely reachable. +allowed-tools: + - shell + - ripgrep_search + - code_lineage + - list_leen_connections + - get_leen_vulnerability_findings + - report_finding +--- + +You triage an existing scanner backlog against real code. + +1. Call `list_leen_connections` to find the available connections. +2. Query open findings at CRITICAL and HIGH severity. +3. For each, locate the affected package or symbol in this repository and use + `call_graph` to determine whether it is reachable from an entrypoint. +4. Report only the reachable ones with `report_finding`, citing the call path. +``` + +## Next steps + + + + Browse the connector catalog. + + + + Grant these tools and put them to work. + + diff --git a/docs.json b/docs.json index f2cd43d..01982c9 100644 --- a/docs.json +++ b/docs.json @@ -20,7 +20,7 @@ "anchors": [ { "anchor": "GitHub App (Preview)", - "href": "https://github.com/apps/console-lab", + "href": "https://github.com/apps/console-staging", "icon": "code-simple" } ] @@ -30,8 +30,50 @@ "group": "Get Started", "pages": [ "introduction", + "quickstart", "install-console" ] + }, + { + "group": "The Harness", + "pages": [ + "harness/overview", + "harness/agents", + "harness/writing-an-agent", + "harness/agent-library", + "harness/skills", + "harness/detections", + "harness/tool-reference" + ] + }, + { + "group": "Context", + "pages": [ + "context/overview", + "context/projects", + "context/connections", + "context/vendor-data", + "context/findings" + ] + }, + { + "group": "Working interactively", + "pages": [ + "interactive/chat", + "interactive/cli" + ] + }, + { + "group": "Workflows", + "pages": [ + "workflows/overview", + "workflows/create-a-workflow", + "workflows/agent-chain", + "workflows/triggers", + "workflows/outputs", + "workflows/running", + "workflows/artifacts" + ] } ] }, @@ -108,7 +150,7 @@ "primary": { "type": "button", "label": "Sign In", - "href": "https://app.console.lab.amplify.security/" + "href": "https://app.console.stg.amplify.security/" } }, "footer": { @@ -119,8 +161,8 @@ }, "redirects": [ { - "source": "/quickstart", - "destination": "/legacy/quickstart" + "source": "/workflows/agents", + "destination": "/workflows/agent-chain" }, { "source": "/sample-project", diff --git a/harness/agent-library.mdx b/harness/agent-library.mdx new file mode 100644 index 0000000..880634b --- /dev/null +++ b/harness/agent-library.mdx @@ -0,0 +1,87 @@ +--- +title: The agent library +description: 'The agents Console ships, what each is for, and which ones to put in a workflow.' +--- + +## Using the library + +Console ships the agents below. They appear in the agent list and the +[workflow agent picker](/workflows/create-a-workflow#agents) alongside anything your organization writes. + +They're also the best available examples of the format — if you're about to +[write an agent](/harness/writing-an-agent), open one first and read how its frontmatter and instructions +are put together. + +## Scanning + +The vulnerability scanners share one approach: map the repository's security conventions, hunt for places +the implementation diverges from that intent, then confirm candidates through analysis. They differ only in +how broadly they hunt and how deeply they confirm. + +| Agent | Profile | Use when | +| --- | --- | --- | +| `vulnerability-scanner-basic` | Pursues the top 6 candidates, caps at 6 findings, generic analysis only. | You want a fast signal — a pull request check where latency matters. | +| `vulnerability-scanner-standard` | Pursues the top 10 candidates, caps at 10 findings, uses the class-specific skill library plus confirmation. | The default choice for most workflows. | +| `vulnerability-scanner-comprehensive` | Pursues the top 40 candidates, caps at 40 findings, uses the full skill library and a more capable model. | Auditing a repository in depth, where thoroughness matters more than speed. | + +The caps are deliberate. A scanner that returns everything it half-suspects is noise; these stop at a +defined budget so the results stay reviewable. + +## Detections + +| Agent | What it does | +| --- | --- | +| `detections-runner` | Runs every [detection](/harness/detections) stored in your organization against the repository, dispatching by type — OpenGrep rules directly, policy detections via one child evaluator each. Findings are linked to the detection that produced them. | +| `detection-author` | Reads findings from an earlier step and authors a reusable detection for each, choosing between an OpenGrep rule and a natural-language policy. | + +These two are the ends of a loop worth understanding: `detection-author` turns a one-time discovery into a +permanent rule, and `detections-runner` is what applies every rule you've accumulated from then on. + +## Review + +| Agent | What it does | +| --- | --- | +| `code-reviewer` | Reviews code for security vulnerabilities, performance issues, and best practices. | +| `security-analyzer` | Deep security analysis with taint tracking and vulnerability assessment. | + +## Agents spawned by other agents + +These exist in the library but are normally driven by another agent rather than added directly as workflow +steps: + +| Agent | Spawned by | Role | +| --- | --- | --- | +| `patch-generator` | `detections-runner`, when patch generation is enabled | Generates a verified patch for one detection's findings in one file: edits the file, re-runs the detection to confirm the fix, captures the diff, and records it. | +| `policy-evaluator` | `detections-runner` | Evaluates one policy detection against the repository, bound to that detection so every finding links back to it. | +| `policy-fix-verifier` | `patch-generator` | Independently judges whether a patched file still violates a policy. Deliberately read-only — it returns a verdict and cannot report findings or edit code. | + +You *can* name one as a workflow step, but they expect a briefing their usual parent provides, so they work +best left to it. + +## The orchestrator + +`workflow-runner` is the agent that executes workflows: it reads a workflow's description and steps, spawns +each step's agent in order, and composes each briefing. You don't add it to a chain — it *is* the chain. See +[the agent chain](/workflows/agent-chain) for how it composes briefings. + +## Common chains + +| Goal | Chain | +| --- | --- | +| Fast pull request check | `vulnerability-scanner-basic` | +| Standard pull request review | `vulnerability-scanner-standard` | +| Turn findings into permanent rules | `vulnerability-scanner-standard` → `detection-author` | +| Apply everything you've already vetted | `detections-runner` | +| Deep audit | `vulnerability-scanner-comprehensive` → `security-analyzer` | + +## Next steps + + + + Extend or shadow any of these. + + + + Put a chain to work. + + diff --git a/harness/agents.mdx b/harness/agents.mdx new file mode 100644 index 0000000..54de310 --- /dev/null +++ b/harness/agents.mdx @@ -0,0 +1,91 @@ +--- +title: Agents +description: 'What an agent is, how it runs, and how agents delegate to each other.' +--- + +## What an agent is + +An agent is a participant in the harness: a model, a set of tools it may call, a budget, and a body of +instructions. Give it a task and it works until the task is done or the budget runs out. + +Unlike a scanner, an agent decides what to do next based on what it just learned. It reads a file, notices a +suspicious call, traces the caller, runs a command to check a hypothesis, and either confirms or discards it. +That loop — reason, act, observe, reason again — is what makes agents suited to security work, where the +interesting questions are about *whether* something is reachable rather than *whether* a pattern appears. + +## How a run proceeds + +1. **The agent receives a task.** In chat that's your message; in a workflow it's a briefing composed by the + orchestrator. +2. **It reasons and calls tools.** Each call returns a result it reads before deciding the next step. +3. **It may load a skill** with `activate_skill` when it hits a task a documented procedure covers. +4. **It may delegate** with `spawn_agent`, handing focused work to a child and waiting for the summary. +5. **It records durable results** — findings, patches, detections — rather than only replying in prose. +6. **It stops** when the task is done, the budget is exhausted, or it's cancelled. + +Everything an agent does in step 2 is bounded by the [tool surface](/harness/tool-reference). + +## Budgets + +Two ceilings keep a run from going forever, both settable per agent: + +- **`maxIterations`** — how many reasoning↔tool cycles it may take. +- **`timeout`** — wall-clock milliseconds for the whole execution. + +For an orchestrator, `timeout` covers every child it spawns, so it must exceed the worst-case sum of their +durations. Leave both unset unless the agent is genuinely an outlier. + +## Delegation and agent trees + +An agent can spawn sub-agents, and those can spawn their own, forming a tree. This exists for two reasons: + +- **Focus.** A child starts with a clean context scoped to one job, so a broad scan doesn't drown in detail + from the first file it opened. +- **Parallelism.** Independent work runs concurrently — one evaluator per detection, one patch generator per + file. + +A child returns a **summary**, not its full transcript. The parent sees the conclusion and quotes it forward. +This is why an agent's `description` and its final summary both matter so much: they're the interface +between agents. + +You can watch the tree live — as a nested view in the CLI, and in the web console's chat while a turn runs. + +### Forcing sequential delegation + +Set [`sequential-spawns: true`](/harness/writing-an-agent#sequential-spawns) when steps depend on each other +and the agent must see one result before starting the next. Console's `workflow-runner` uses exactly this to +guarantee workflow steps run in order. + +## Where agents come from + +| Source | Description | +| --- | --- | +| **Platform agents** | The library Console ships. See [the agent library](/harness/agent-library). | +| **Organization agents** | Agents your team writes, in the web console or the CLI. | + +Both appear together wherever agents are listed, and the workflow orchestrator treats them identically. An +organization agent whose `name` matches a platform agent **shadows** it — the supported way to customize +built-in behavior. + +## When to write your own + +Reach for a new agent when: + +- The task is a distinct job with its own output — "audit dependencies", "review IaC for public exposure". +- You want different tool permissions, like a read-only reviewer that can't modify code. +- You want a different model for cost or depth reasons. + +Prefer a [skill](/harness/skills) instead when you're capturing *how to do one thing well* and an existing +agent could follow it. Skills are cheaper: no separate budget, no separate model, loaded only when relevant. + +## Next steps + + + + The frontmatter reference. + + + + Sequence agents into a workflow. + + diff --git a/harness/detections.mdx b/harness/detections.mdx new file mode 100644 index 0000000..19abbee --- /dev/null +++ b/harness/detections.mdx @@ -0,0 +1,126 @@ +--- +title: Detections +description: 'Reusable rules that outlive any one conversation — OpenGrep patterns and natural-language policies.' +--- + +## Why detections exist + +A finding describes one moment: this file, this commit, this vulnerability. A **detection** is the rule +behind it, and it keeps checking forever. + +This is the difference between an agent that's useful once and a platform that compounds. When an agent +confirms a vulnerability, the valuable output isn't only the fix — it's the rule that catches the same +mistake in every repository from then on, cheaply, without an agent having to re-derive it. + +## Detection types + +| Type | Format | What it's good at | +| --- | --- | --- | +| **OpenGrep rule** | YAML | Structural patterns with a known shape — a dangerous API, a missing flag, a taint path from source to sink. Fast and deterministic. | +| **Policy** | Natural language | Intent that resists pattern matching — "every endpoint that mutates data must check authorization", "no service may log request bodies". Evaluated by an agent. | +| **CodeQL query** | QL | Deep dataflow queries. See the limitation below. | + + + **CodeQL detections can be authored but are not yet executed.** Only OpenGrep and policy detections have + runtimes today; `detections-runner` stores other types and skips them with a note. The + `codeql-rule-creator` skill and the editor's CodeQL support exist so the rules are ready when execution + lands. + + +### OpenGrep rules + +OpenGrep rules support two modes: + +- **`search`** — match a pattern. Use `pattern`, `patterns`, `pattern-either`, or `pattern-regex`. +- **`taint`** — track data flow. Declare `pattern-sources`, `pattern-sinks`, and optionally + `pattern-sanitizers`; a match is a source reaching a sink with nothing neutralizing it in between. + +Taint mode is the one that earns its keep for security work, because it encodes *reachability* rather than +mere presence. + +Rules carry a severity of `INFO`, `WARNING`, or `ERROR`, and OpenGrep filters by the rule's own `languages` +field at run time — so an irrelevant rule exits cheaply rather than wasting a pass. + +### Policy detections + +A policy is a security requirement written in plain language. At run time, `detections-runner` spawns a +`policy-evaluator` per policy, bound to that detection so every finding links back to it. + +Policies are the right tool when the rule is about intent — business logic, authorization, data handling — +where no pattern captures the requirement and a human reviewer would need to reason about the code. + +## Fields that matter + +| Field | Values | What it does | +| --- | --- | --- | +| **Status** | `TEST`, `PRODUCTION` | Whether the detection is still being evaluated or is trusted. Promote once its results hold up. | +| **Severity** | `critical`, `high`, `medium`, `low` | How serious a match is. | +| **Disposition** | `flag`, `escalate` | What should happen on a match — record it, or escalate it. | +| **Tags** | free-form | Grouping and filtering. | +| **Project** | optional | Scopes the detection to one repository. Unset means it applies organization-wide. | + +### The test-to-production lifecycle + +New detections start at `TEST`. Run them, review what they catch, tune the rule, and promote to +`PRODUCTION` when the signal is trustworthy. + +This exists because a noisy detection is worse than no detection — it trains your team to ignore results. +Keeping unproven rules visibly in `TEST` lets you build the library without eroding trust in it. + +## Where detections come from + +| Source | Meaning | +| --- | --- | +| `manual` | Authored by a person, or by an agent on request. | +| `risk-register` | Compiled from your organization's risk register. | +| `threat-model` | Compiled from a threat model. | + +Compiled detections keep provenance back to the upstream document, so a rule can be traced to the +requirement that motivated it. + +## Authoring a detection + +**In the web console.** Open **Detections** and create one. The editor syntax-highlights by type — YAML for +policies, Markdown with YAML frontmatter for rule types — and labels the language in the header. Customer +types the UI doesn't recognize still render with a generic label rather than breaking. + +**With an agent.** Often the better path, because agents can validate as they go: + +- `opengrep-rule-creator` writes and checks an OpenGrep rule. +- `policy-detection-creator` turns a requirement into a stored policy. +- `detection-author` reads a scan's findings and authors a detection for each — the automated version of + the same loop. + +Ask in [chat](/interactive/chat): *"Write an OpenGrep rule that catches this pattern and store it as a test +detection."* + +## Running detections + +Add [`detections-runner`](/harness/agent-library#detections) as a workflow step. It lists every stored +detection, triages which apply, and dispatches by type — OpenGrep rules directly, policies via one child +evaluator each. Findings link back to the detection that produced them, so you can see which rules are +earning their place. + +Its bias is deliberate: it dispatches when in doubt, because a detection that never ran is worse than a +wasted pass. + +## The compounding loop + +1. An agent confirms a vulnerability in [chat](/interactive/chat) or a workflow run. +2. `detection-author` — or you — turns it into a detection, at `TEST`. +3. You review what it catches and promote it to `PRODUCTION`. +4. A `detections-runner` workflow applies it on every pull request from then on. + +Step 4 is cheap and repeatable. That's the payoff for the reasoning spent in step 1. + +## Next steps + + + + Add `detections-runner` to a chain. + + + + What a detection produces when it matches. + + diff --git a/harness/overview.mdx b/harness/overview.mdx new file mode 100644 index 0000000..e659f57 --- /dev/null +++ b/harness/overview.mdx @@ -0,0 +1,79 @@ +--- +title: The Harness +description: 'Console supplies the loop; you supply the domain logic as agents, skills, and detections.' +--- + +## What "harness" means + +Console runs an agentic loop: a model reasons, calls tools, reads the results, and reasons again, inside +an isolated environment with access to your code. Building that loop — the orchestration, the tool +implementations, the sandboxing, the sub-agent delegation, the event streaming — is Console's job. + +Deciding *what the loop should do* is yours. Console exposes three authorable primitives for that, and +everything else in the product is built on top of them. + +| Primitive | What it is | Format | Authored in | +| --- | --- | --- | --- | +| **[Agent](/harness/agents)** | A participant that reasons, calls tools, and can delegate | Markdown + YAML frontmatter | Web console, CLI | +| **[Skill](/harness/skills)** | A procedure an agent loads on demand | Markdown + YAML frontmatter | CLI, API | +| **[Detection](/harness/detections)** | A rule that can be run repeatedly against code | OpenGrep YAML, or a natural-language policy | Web console, CLI | + +The distinction between them is worth getting right, because it's the most common early mistake: + +- An **agent** is *who* is working. It has its own model, tool permissions, and budget. +- A **skill** is *how* to do one thing well. It has no budget of its own — an agent loads it when the + task calls for it. +- A **detection** is *what* to look for. It's the durable artifact: a finding describes one moment, a + detection keeps checking forever. + +## Yours are first-class + +Console ships a library of agents, skills, and detections. Anything you write sits alongside them with no +second-class status — the orchestrator that runs workflows treats platform and organization agents +identically, briefing both from their stated `description`. + +An agent you write with the same name as a built-in one **shadows** it. That's the supported way to +customize built-in behavior without breaking workflows that already reference that name. + +## The tool surface is the ceiling + +An agent can only do what some tool lets it do. However you word its instructions, its real capabilities +are the union of the tools it's allowed to call — reading and writing files, running commands, querying +code structure, scanning, fetching a URL, reading your vendors' findings, recording results, and spawning +sub-agents. + +This is the single most useful thing to internalize before writing an agent, because it tells you which +tasks are achievable and which aren't. The full inventory is in the +[tool reference](/harness/tool-reference). + +## How the pieces run together + +A realistic chain uses all three primitives at once: + +1. A **workflow** step spawns your **agent**. +2. The agent **activates a skill** for the vulnerability class it's investigating. +3. It runs stored **detections** against the repository. +4. It records **findings**, and optionally a patch. +5. A workflow **output** posts the result to the pull request. + +Each layer is replaceable. That's the point of a harness. + +## Next steps + + + + The frontmatter reference and the editor. + + + + Everything an agent can actually do. + + + + What ships with Console. + + + + Connect repositories and vendors. + + diff --git a/harness/skills.mdx b/harness/skills.mdx new file mode 100644 index 0000000..f74164b --- /dev/null +++ b/harness/skills.mdx @@ -0,0 +1,124 @@ +--- +title: Skills +description: 'Package a procedure an agent loads on demand — and when to write one instead of an agent.' +--- + +## What a skill is + +A skill is a documented procedure an agent loads when it becomes relevant. It's a Markdown file with YAML +frontmatter, same as an agent, but it isn't a participant: it has no model, no budget, and it never runs on +its own. An agent calls `activate_skill` and the skill's instructions enter its context. + +The reason skills exist is context economy. A comprehensive scanner might know how to analyze twenty +vulnerability classes, but loading all twenty procedures up front would crowd out the code it's supposed to +be reading. Instead it loads the SQL-injection procedure when it's looking at a query, and the SSRF +procedure when it's looking at an outbound request. + +## Skill or agent? + +| Write a **skill** when | Write an **agent** when | +| --- | --- | +| You're capturing *how to do one thing well* | You're defining *a job with its own output* | +| An existing agent could follow your instructions | You need different tool permissions or a different model | +| The procedure is only relevant sometimes | The work deserves its own budget and can be delegated to | +| You want it available to many agents | You want to name it as a workflow step | + +When in doubt, start with a skill. It's cheaper — nothing to budget, nothing to orchestrate — and you can +promote it to an agent later if it grows its own output. + +## Frontmatter reference + +| Key | Required | Type | What it does | +| --- | --- | --- | --- | +| `name` | Yes | string | How the skill is activated and listed. | +| `description` | Yes | string, ≤500 chars | When to use this skill. Agents read it to decide whether to activate. | +| `allowed-tools` | No | string[] | Tools the procedure expects. See the [tool reference](/harness/tool-reference). | +| `license` | No | string | Optional license string. | +| `compatibility` | No | object | `min-version` / `max-version` constraints. | +| `metadata` | No | object | Free-form key/value data. | + +```markdown +--- +name: ssrf-analysis +description: Confirms or rules out server-side request forgery where user input reaches an outbound HTTP call. Use when a request URL, host, or path is influenced by request data. +allowed-tools: + - shell + - code_lineage + - ripgrep_search +--- + +# SSRF analysis + +## Confirming + +1. Identify the outbound call and the client library in use. +2. Trace the URL argument back to its source with `call_graph`. +3. Establish whether an attacker controls the scheme, host, or path — not merely the query string. +4. Check for an allow-list, a resolved-IP check, or a proxy that constrains the destination. + +## Ruling out + +Discard the candidate when the host is a compile-time constant, or when the only +attacker-controlled portion is a path segment appended to a fixed host with no +traversal possible. +``` + +Like an agent's, the `description` is load-bearing: it's what an agent reads when deciding whether this skill +applies. + +## Built-in skills + +| Skill | What it does | +| --- | --- | +| `vulnerability-scan` | Uses natural-language policy descriptions to find vulnerabilities in application logic. | +| `opengrep-rule-creator` | Authors OpenGrep rules for detecting a vulnerability or code pattern. | +| `codeql-rule-creator` | Authors CodeQL queries for vulnerabilities, bug patterns, and code-quality issues. | +| `policy-detection-creator` | Turns a security requirement into a stored natural-language [policy detection](/harness/detections). | + +The scanner agents also draw on an internal library of class-specific analysis skills, which is what +distinguishes the `standard` and `comprehensive` [profiles](/harness/agent-library#scanning) from `basic`. + +## Where to author a skill + + + Skills are authored in the **CLI** (as files) or through the API. Unlike agents and detections, there is no + skill editor in the web console today. + + +In the CLI, a skill is a file on disk: + +``` +~/.amplify/skills//SKILL.md # available in every session +./skills//SKILL.md # project-local +``` + +Override those locations with `AMPLIFY_SKILLS_DIR`. Skills load at startup, so restart the CLI after adding +one. + +A skill directory can hold supporting files — reference documents, example rules, helper scripts — next to +`SKILL.md`, and the instructions can point the agent at them. + +## Using skills in the CLI + +| Command | What it does | +| --- | --- | +| `/skills` | Lists available skills and marks which are active. | +| `/skill ` | Toggles a skill on or off for the session. | + +Toggling a skill on makes it available immediately, which is the fastest way to test one you're writing: +edit the file, restart, activate, and give the agent a task that should trigger it. + +Agents also activate skills on their own via `activate_skill` — manual toggling is for pinning a procedure +you specifically want followed. + +## Next steps + + + + Turn a procedure's results into a permanent rule. + + + + Where skills are authored and toggled. + + diff --git a/harness/tool-reference.mdx b/harness/tool-reference.mdx new file mode 100644 index 0000000..947bbf7 --- /dev/null +++ b/harness/tool-reference.mdx @@ -0,0 +1,138 @@ +--- +title: Tool reference +description: 'Every tool an agent can call — the real ceiling on what any agent you write can do.' +--- + +## Why this page matters + +An agent's capabilities are exactly the tools it can call. Instructions describe intent; tools determine +what's possible. Before writing an agent, check that the work you have in mind maps onto something here — +otherwise the agent will try, fail, and narrate the failure. + +Use these names in an agent's [`allowed-tools`](/harness/writing-an-agent#frontmatter-reference). Omitting +`allowed-tools` inherits the default set rather than granting everything. + + + Availability differs slightly by surface. Tools that operate on a cloned target repository or on your + organization's connections are available to cloud agents; the CLI runs against your working directory + instead. + + +## Reading and searching code + +| Tool | What it does | +| --- | --- | +| `shell` | Runs a shell command in the agent's environment. The general-purpose escape hatch — reading files, running builds, invoking any CLI. Output is truncated past a limit, and a blocked-pattern list rejects dangerous commands. | +| `ripgrep_search` | Fast regex or literal search across the repository, returning file, line number, and matching text. `.gitignore`-aware. The right choice for plain search, and the fallback for languages without a tree-sitter grammar. | +| `read_pdf` | Extracts text from a PDF — useful for threat models, policy documents, and vendor reports. | + +## Understanding code structure + +`code_lineage` provides three structural operations backed by tree-sitter. Coverage is limited to +languages with a grammar; on anything else, use `ripgrep_search`. + +| Operation | What it does | +| --- | --- | +| `find_symbol` | Finds where a function, method, or class is defined and every place it's referenced. | +| `call_graph` | Traces callers or callees of a symbol to a bounded depth (1–3). `direction='callers'` finds what invokes it; `callees` finds what it invokes. Computed on demand — no persistent index. | +| `ast_query` | Runs a tree-sitter S-expression query against one file and returns captured nodes with line ranges. For precise structural search, like every call expression in a file. | + +This is how an agent traces taint: find the sink, walk callers back toward a source, and confirm nothing on +the path neutralizes the input. + +## Scanning + +| Tool | What it does | +| --- | --- | +| `opengrep_scan` | Runs OpenGrep static rules over the repository for a fast first pass, returning matches for the agent to triage. Defaults to the broad `auto` ruleset. **Does not** record findings — it produces candidates. | +| `run_opengrep_detection` | Runs one *stored* OpenGrep [detection](/harness/detections) and emits matches as findings linked to that detection. Refuses non-OpenGrep detections. | +| `check_opengrep_rule` | Re-runs a stored detection's rule against a single file, read-only. Zero matches means a fix worked. This is the verification half of patch generation. | + +## Confirming exploitability + +| Tool | What it does | +| --- | --- | +| `generate_poc` | Produces a proof-of-concept for a candidate vulnerability. | +| `execute_poc` | Runs a generated PoC against a configured running target to confirm the vulnerability. Passes a safety gate first, and only executes when a target URL is configured — otherwise it returns `executed=false` with a reason and the agent falls back to reasoning-only confirmation. Supports bash, Python, and JavaScript. | +| `vuln_poc_evaluation` | Judges the PoC result to decide whether the vulnerability is confirmed. | + + + PoC execution requires an explicitly configured target and clears a safety screen. Without a running + target, agents confirm by reasoning — tracing the data flow end to end — rather than by executing an + exploit. + + +## Reading the outside world + +| Tool | What it does | +| --- | --- | +| `web_fetch` | Fetches a URL and extracts its text. Accepts custom headers and an extraction mode. | + + + `web_fetch` performs **reads only** — it takes a URL, headers, and an extraction mode, with no request + method or body. There is no built-in tool for writing to a third-party API, so agents cannot create + tickets or update external records through it. Console's outbound writes go through + [workflow outputs](/workflows/outputs). + + +## Your vendors' data + +| Tool | What it does | +| --- | --- | +| `list_leen_connections` | Lists the vendor connections configured for your organization, returning each connection's id and vendor. The discovery step before either query below. | +| `get_leen_vulnerability_findings` | Queries normalized vulnerability findings from a connected vendor. Filters by severity, state, and whether a fix is available; paginates with a cursor. | +| `get_leen_vulnerability_finding` | Fetches one vendor finding in full. | + +These three are the entire vendor-data surface today, and they are scoped to **vulnerability findings**. +See [what agents can read](/context/vendor-data) for what that covers and what it doesn't. + +## Projects + +| Tool | What it does | +| --- | --- | +| `list_projects` | Lists the repositories connected to your organization, with ids, `owner/repo` names, and clone URLs. | +| `clone_project` | Clones a repository into the workspace so the agent can read and search it. | + +## Recording results + +| Tool | What it does | +| --- | --- | +| `report_finding` | Records a confirmed vulnerability, with evidence. This is what makes a result durable rather than conversational. | +| `list_findings` | Lists findings already recorded. | +| `report_patch` | Attaches a verified remediation patch (a unified diff) to the finding(s) it fixes. Call it only after editing the file **and** confirming the fix. It stores the patch; it does not verify. | + +## Detections + +| Tool | What it does | +| --- | --- | +| `create_detection` | Stores a new detection. | +| `list_detections` | Lists stored detections with id, name, description, and type. | +| `get_detection` | Fetches one detection, including its rule body. | +| `delete_detection` | Removes a detection. | + +## Coordination + +| Tool | What it does | +| --- | --- | +| `spawn_agent` | Spawns a sub-agent on a focused task and returns its summary. How a broad job fans out without one agent losing the thread. | +| `spawn_multiple_agents` | Spawns several sub-agents. Used to run one child per item — a policy evaluator per detection, for instance. | +| `activate_skill` | Loads a [skill](/harness/skills) into the agent's context on demand. | +| `add_todo`, `start_todo`, `complete_todo`, `clear_todos` | Maintains the agent's plan as a visible todo list you can watch while it works. | + +## Managing agents + +| Tool | What it does | +| --- | --- | +| `create_agent`, `get_agent`, `list_agents`, `update_agent`, `delete_agent` | Lets an agent read and write agent definitions — so an agent can help you author another one. | + +## Next steps + + + + Put these names in `allowed-tools`. + + + + Package a procedure instead of a participant. + + diff --git a/harness/writing-an-agent.mdx b/harness/writing-an-agent.mdx new file mode 100644 index 0000000..cbef563 --- /dev/null +++ b/harness/writing-an-agent.mdx @@ -0,0 +1,144 @@ +--- +title: Writing an agent +description: 'The AGENT.md format — YAML frontmatter plus a Markdown body — and every field it accepts.' +--- + +## The format + +An agent is a Markdown document with two parts: + +1. **YAML frontmatter** — the machine-readable declaration: name, model, tool permissions, budgets. +2. **A Markdown body** — the agent's instructions. This becomes its system prompt. + +```markdown +--- +name: dependency-auditor +description: Audits third-party dependencies for known-vulnerable versions and unmaintained packages, and reports each one as a finding. +model: anthropic/claude-sonnet-4-6 +allowed-tools: + - shell + - ripgrep_search + - web_fetch + - report_finding +--- + +You audit third-party dependencies. + +## Workflow + +1. Locate every manifest and lockfile in the repository. +2. For each direct dependency, determine the resolved version. +3. Flag versions with known advisories, and packages with no release in over two years. +4. Report each one with `report_finding`, citing the manifest path and the resolved version. + +## Rules + +- Report the resolved version from the lockfile, never the range from the manifest. +- Do not report transitive dependencies unless the advisory is critical. +``` + +That's the entire contract. No build step, no registration. + +## Frontmatter reference + +| Key | Required | Type | What it does | +| --- | --- | --- | --- | +| `name` | Yes | string | How the agent is referenced — by workflow steps and by `spawn_agent`. Must be unique. | +| `description` | Yes | string, ≤500 chars | What this agent does. **Load-bearing** — see below. | +| `model` | No | string | Which model to run. Omit to inherit the default. | +| `allowed-tools` | No | string[] | Restricts the agent to these tools. Omit to inherit. | +| `maxIterations` | No | positive int | Budget of reasoning↔tool cycles before the agent is stopped. | +| `timeout` | No | positive int (ms) | Wall-clock limit for one execution. | +| `sequential-spawns` | No | boolean | Rejects concurrent `spawn_agent` calls, forcing one child at a time. | + +### `description` is not a comment + +The description does real work at runtime. When a workflow runs your agent, the orchestrator reads the +description to compose that step's briefing, and reflects the agent's stated role back at it. Other agents +deciding whether to delegate also see only the name and description. + +A vague description produces a vague briefing. Write it as a precise statement of what the agent does and +what it produces: + +```yaml +# Good — states the job and the output +description: Audits third-party dependencies for known-vulnerable versions and unmaintained packages, and reports each one as a finding. + +# Too vague to brief against +description: Dependency helper. +``` + +### `allowed-tools` restricts, it doesn't grant + +Listing a tool doesn't create capability that doesn't exist — it narrows the agent to a subset of what the +harness already offers. See the [tool reference](/harness/tool-reference) for valid names. + +Restricting tools is a real design technique, not just hygiene. An agent that shouldn't modify code should +not be given `shell`, and a verifier that must stay honest should not be given the ability to report +findings. Console's own `policy-fix-verifier` works this way: it is deliberately read-only so its verdict +can't be self-serving. + +### Budgets: `maxIterations` and `timeout` + +Both are ceilings, not targets. Leave them unset unless the agent is an outlier. + +- Raise `maxIterations` for agents that legitimately need many tool calls — a broad scan across a large + repository. +- Raise `timeout` for **orchestrators**, whose wall clock includes every child they spawn. Set it above the + worst-case sum of the children's durations. + +### `sequential-spawns` + +By default an agent may spawn several sub-agents at once. Setting `sequential-spawns: true` makes a second +concurrent spawn return an error instead of queuing, which forces the agent to observe each child's result +before starting the next. + +Use it for orchestrators whose steps depend on each other. It applies only to the agent that declares it — +children are free to fan out. + +## Writing one in the web console + +Open **Agents** and create an agent. The editor is a Markdown editor with: + +- **Frontmatter linting** — malformed YAML is flagged as you type. +- **A model picker** — selecting a model rewrites the `model:` line in place, so what you see in the + frontmatter is always what will run. +- **Folders** — organize agents as the list grows. + +Your organization's agents appear in the [workflow agent picker](/workflows/create-a-workflow#agents) +next to the built-in ones. + +## Writing one in the CLI + +The CLI loads agent definitions from the filesystem, so an agent is just a file: + +``` +~/.amplify/agents//AGENT.md # available in every session +./agents//AGENT.md # project-local +``` + +Override those locations with `AMPLIFY_AGENTS_DIR`. Definitions load at startup, so restart the CLI after +adding one. + +## Shadowing a built-in agent + +Give your agent the same `name` as one Console ships and yours takes precedence. This is the supported way +to change built-in behavior — a workflow step referencing that name keeps working and picks up your +version. + + + Start by copying the built-in agent you want to change, editing the body, and keeping the name. You + inherit a working structure and only change what you meant to. + + +## Next steps + + + + Valid `allowed-tools` values and what each does. + + + + Built-in agents worth reading as examples. + + diff --git a/interactive/chat.mdx b/interactive/chat.mdx new file mode 100644 index 0000000..f455cd8 --- /dev/null +++ b/interactive/chat.mdx @@ -0,0 +1,99 @@ +--- +title: Chat +description: 'Direct an agent by hand — and promote what works into a workflow.' +--- + +## What chat is for + +Chat is the interactive way to use the harness. You give an agent a task, watch it work, and redirect it as +it goes. + +It's the right mode when you don't yet know what you're looking for. The agent doesn't answer from an index — +it investigates: reads files, runs commands, traces call paths, and follows the code until it can answer. + +In a session an agent can: + +- **Explore your codebase** — read, search, and trace how data flows through the application +- **Run commands** in its sandbox to test its own hypotheses +- **Load [skills](/harness/skills)** for specialized procedures +- **Delegate to sub-agents**, so a broad question can fan out without losing the thread +- **Track its plan** as a running todo list you can watch +- **Record [findings](/context/findings)** and write patches for what it confirms + +Ask open-ended questions — *"are there any exposed endpoints in this service that shouldn't be?"* — or point +it at something specific and ask it to dig in. + +## Watching it work + +A session shows you what the agent is doing rather than just its conclusion: + +- **Tool calls** appear as they run, with a status dot and the command or arguments, so you can see the + reasoning path. +- **The todo list** shows the plan and what's done, and disappears once everything completes. +- **The sub-agent tree** appears when the agent delegates, showing which children are running. +- **The Tools panel** on the right lists every tool call in the session; click one to jump to it in the + transcript. + +This visibility is the point. When an agent reaches a wrong conclusion, the tool trail usually shows exactly +where it went sideways. + +## Sessions + +Chats persist to your organization, so work doesn't evaporate when you close the tab. + +| Action | Notes | +| --- | --- | +| **New chat** | Starts a fresh session | +| **Rename** | Sessions title themselves from your first message | +| **Archive** | Keeps the session read-only; unarchive to continue | +| **Delete** | Permanent, with a confirmation | + +A session survives a page reload mid-turn — reconnecting picks the running turn back up rather than losing it. + +## Choosing a model + +The model picker in the header sets the model for the session, and you can switch **mid-conversation**. + +Switching carries your context forward: Console summarizes the conversation so far and hands that summary to +the new model, marking the switch point in the transcript. If a turn is in flight, you'll be asked to confirm, +since switching cancels it. + +Use a faster model to explore, then switch to a more capable one for the hard part. + +## Cancelling a turn + +Press Escape while a turn is running. The agent stops and the transcript notes the interruption, so +you can redirect without starting over. Anything already recorded — findings, patches — stays. + +## Starting from a finding + +The fastest way into a productive session is from a [finding](/context/findings). **Start a chat** on any +finding opens a session with the affected code, the agent's reasoning, and any generated patch already loaded, +and you type your question in the same step. + +## Promote it to a workflow + +Chat is exploratory by design. Once you've asked the same question a third time, it belongs in a +[workflow](/workflows/overview): + +| In chat | As a workflow | +| --- | --- | +| You pick the repository each time | The trigger names the repositories | +| You type the task | The workflow's description and agent chain encode it | +| You read the answer | [Outputs](/workflows/outputs) deliver it to the pull request | +| Runs when you remember | Runs on every pull request | + +The translation is usually direct: the prompt you refined becomes the workflow description, and the agents you +found useful become the chain. + +## Next steps + + + + Same agents, against your local working directory. + + + + Automate what worked in chat. + + diff --git a/interactive/cli.mdx b/interactive/cli.mdx new file mode 100644 index 0000000..75f7081 --- /dev/null +++ b/interactive/cli.mdx @@ -0,0 +1,132 @@ +--- +title: The CLI +description: 'Run the same agents in your terminal, against the code you have checked out right now.' +--- + +## Why use the CLI + +The CLI runs the same agents as the web console, with one important difference: it works on **the directory +you started it in**, on your machine. + +That makes it the right tool for code you're actively editing. An agent can look at uncommitted changes, +work-in-progress branches, and files that exist nowhere but your laptop — none of which a cloud sandbox can +reach. + +See [Installation](/install-console) to get set up. + +## Web console vs. CLI + +The agents' capabilities are the same. What differs is where they run and what they can reach. + +| | Web console | CLI | +| --- | --- | --- | +| Runs in | An isolated cloud sandbox | Your own machine | +| Code it analyzes | Clones connected projects on demand | The directory you started it in | +| Uncommitted changes | Not visible | Visible | +| Chat history | Saved to your organization; resume, rename, archive | Not persisted between sessions | +| Choosing a model | Model picker | `/model` | +| Slash commands | Not available | `/help`, `/model`, `/skills`, `/skill`, `/context`, `/clear`, `/quit` | +| One-shot prompts | Not available | `console --prompt "..."` | +| Authoring skills | Not available | Files on disk | +| Starting from a finding | Yes | No | +| Works offline | No | Yes | + +## Starting a session + +```bash +cd ~/src/hello-world +console +``` + +Ask it something: + +- *Find and resolve any vectors for XSRF/XSS attacks within this project.* +- *Are there any exposed service endpoints in this project that shouldn't be?* + +The footer shows whether you're connected to Amplify Cloud (and which organization) or running offline, plus +the active model. + +## One-shot prompts + +Run a single prompt and exit — useful in scripts, git hooks, and CI: + +```bash +console --prompt "Scan this project for vulnerabilities" +``` + +It prints the agent's final answer and exits. Also `--version`. + +## Slash commands + +| Command | Aliases | What it does | +| --- | --- | --- | +| `/help` | `/?` | Lists all commands | +| `/model` | | Opens the model picker | +| `/skills` | | Lists available skills and which are active | +| `/skill ` | | Toggles a skill on or off | +| `/context ` | | Switches organization | +| `/clear` | | Clears the conversation | +| `/quit` | `/q`, `/exit` | Exits | + +Type `/` to get an autocomplete dropdown rather than memorizing them. + +### The model picker + +`/model` opens a tabbed picker: + +- **Popular** — a curated shortlist. Works even if the model catalog is unreachable. +- **Browse** — grouped by provider; drill into one to see its models. +- **Search** — substring match across the full catalog, with pricing and context length per model. + +Switching models mid-session carries your context forward via a summary, the same as the web console, and the +transcript notes the switch. + +## Authoring skills and agents + +The CLI is where [skills](/harness/skills) are authored, because it loads both skills and agents from the +filesystem: + +``` +~/.amplify/skills//SKILL.md # available in every session +~/.amplify/agents//AGENT.md +./skills//SKILL.md # project-local +./agents//AGENT.md +``` + +Override with `AMPLIFY_SKILLS_DIR` and `AMPLIFY_AGENTS_DIR`. Both load at startup, so restart after adding +one. + +The loop is quick: write the file, restart, `/skill ` to activate, then give the agent a task that +should trigger it. + +## Offline mode + +Without an Amplify API key the CLI runs fully local: detections and findings go to a SQLite database under +`~/.amplify` instead of your organization, and the footer shows a red **offline mode** indicator. + +Once authenticated, detections write through to your organization instead, so work done locally shows up for +your team. + +## Cancelling and exiting + +Ctrl+C cancels a running turn and leaves the session open so you can redirect. Pressing it again +when nothing is running exits. + +## What the CLI doesn't do + +- **No persistent chat history.** Sessions aren't saved; `/clear` is final and there's no resuming a past + conversation. Findings and detections do persist. +- **No input history recall.** Arrow keys don't walk previous messages. +- **No workflows.** Workflows are a cloud feature — the CLI runs agents interactively, one session at a time. + +## Next steps + + + + The CLI is where skills are authored. + + + + Setup and API keys. + + diff --git a/introduction.mdx b/introduction.mdx index b6dcad0..2565fd5 100644 --- a/introduction.mdx +++ b/introduction.mdx @@ -1,13 +1,17 @@ --- title: Introduction -description: 'Amplify Console is an agentic AI platform for security analysis.' +description: 'Amplify Console is an AI harness for security — you define the agents, give them context, and automate them as workflows.' --- ## What is Console? -Console is Amplify Security's agentic AI platform for security analysis. It runs as an interactive -terminal application that uses AI agents to reason about your codebase, surface security findings, -and help your team understand and remediate vulnerabilities. +Console is Amplify Security's AI harness for security work. It supplies the machinery — LLM +orchestration, tool execution, isolated sandboxes, sub-agent delegation — and you supply the domain +logic: the agents that reason, the skills they follow, and the detections they enforce. + +That distinction matters. Console is not a scanner with a fixed set of checks. It's a harness you point +at your own problems, and you get the most out of it by defining your own agents rather than only using +the ones we ship. Console is currently in alpha, undergoing rapid development. Contact @@ -15,16 +19,71 @@ and help your team understand and remediate vulnerabilities. work with us to help our project come to fruition! -## How it works +## Three ideas + +Everything in Console fits into one of three layers, and they build on each other in this order. + + + + The agents, skills, and detections that do the work. All of them are yours to write. + + + + Repositories and vendor connections, so the harness reasons about your actual environment. + + + + Chain agents into automation that runs on every pull request and produces real artifacts. + + + +### 1. The harness + +An **agent** is a Markdown file with YAML frontmatter: the frontmatter declares its name, model, and +which tools it may use, and the body is its instructions. That's the whole format, which is why writing +one is a text-editing task rather than a programming one. + +Agents delegate to sub-agents, load **skills** for specialized procedures, and run **detections** — +reusable rules that outlive any single conversation. Console ships a library of each, and yours sit +alongside them as equals. -Console operates as an interactive chat session in your terminal. The AI agent has access to your -connected repositories and Amplify platform data, allowing it to answer questions about your -security posture, investigate specific findings, and guide remediation. +### 2. Context + +A harness with no context is just a chatbot. Console gets context from two places: + +- **Projects** — the repositories it clones and reads. +- **Connections** — your source control provider, plus the security and IT vendors you already run. + Connect Semgrep, Snyk, Cloudflare, Jira and others through the connector catalog. + +You direct that context two ways: interactively in **chat**, or declaratively in a **workflow**. + +### 3. Workflows + +Workflows are where it comes together. A workflow chains agents into an ordered sequence, fires on a +schedule you choose — manually, or automatically on every pull request — and delivers results where your +team already works: as review comments, or as a required check that blocks merging until security review +passes. + +## Two interfaces + +The same agents run in both places. What differs is where they run and what code they can reach. + +| | Web console | CLI | +| --- | --- | --- | +| Runs in | An isolated cloud sandbox | Your own machine | +| Code it analyzes | Clones your connected projects on demand | The directory you started it in | +| Best for | Team work, automation, anything shared | Code you're editing right now, including uncommitted changes | + +See [Chat](/interactive/chat) and [The CLI](/interactive/cli) for the differences in detail. ## Get started - - Install Console on Linux or macOS. + + Connect a repository, ask an agent a question, and build your first workflow. + + + + Run Console in your terminal on Linux or macOS. diff --git a/quickstart.mdx b/quickstart.mdx new file mode 100644 index 0000000..aee8aaa --- /dev/null +++ b/quickstart.mdx @@ -0,0 +1,127 @@ +--- +title: Quickstart +description: 'Connect a repository, ask an agent a question, and turn the answer into automation.' +--- + +This walks through all three layers of Console in one pass: connect **context**, use the **harness** +interactively, then automate it as a **workflow**. Budget about twenty minutes. + +## 1. Connect a repository + +Sign in to the web console and open **Connections**. Install the Console GitHub App for your organization, +granting access to at least one repository you're comfortable experimenting on. + +The repositories you grant appear under **Projects**. That's your context — the code agents can read. + + + Pick a real repository rather than an empty one. Agents reason about actual code, so a repository with real + application logic gives you a far better sense of what Console does. + + +## 2. Ask an agent something + +Open **Console** in the sidebar and start a new chat. Ask a question about the repository you just connected: + +> Look at this repository and tell me where user input reaches a database query without parameterization. + +Watch what happens. The agent clones the repository, reads files, searches, and traces call paths — and you +see each tool call as it runs. This is the difference between an agent and a scanner: it's deciding what to +look at next based on what it just found. + +Try following up. *Is that actually reachable from an HTTP handler?* The agent investigates rather than +re-answering from memory. + +If it confirms something, it records a [finding](/context/findings) — durable, with its reasoning attached. + +## 3. Write an agent + +The built-in library covers a lot, but the harness is most useful when the agents are yours. Open **Agents** +and create one: + +```markdown +--- +name: secrets-auditor +description: Finds credentials, API keys, and tokens committed to the repository, and reports each one as a finding with the file and line. +model: anthropic/claude-sonnet-4-6 +allowed-tools: + - shell + - ripgrep_search + - report_finding +--- + +You audit a repository for committed secrets. + +1. Search for common credential patterns — API keys, private keys, connection + strings, bearer tokens. +2. For each candidate, read the surrounding code to judge whether it's a real + credential or a placeholder, test fixture, or example. +3. Report only real credentials with `report_finding`, citing file and line. + +Do not report obvious placeholders (`xxx`, `changeme`, `example.com`), values +clearly loaded from the environment, or anything under a fixtures directory. +``` + +Save it. That's a working agent — YAML frontmatter declaring its name, model, and tools, plus instructions as +the body. See [writing an agent](/harness/writing-an-agent) for every field. + +## 4. Automate it + +Open **Workflows** and click **New workflow**. + +| Field | What to enter | +| --- | --- | +| **Name** | `Secrets check` | +| **Description** | `Audit pull requests for committed credentials and comment on anything found.` | +| **Triggers** | Add **On pull requests**, and select your repository | +| **Agents** | Add `secrets-auditor` | +| **Output** | Add **Comment on triggering pull request** | + +Click **Save**. + +The description isn't decoration — Console's orchestrator reads it as the goal and briefs your agent against +it. Write it as an instruction, not a label. See [create a workflow](/workflows/create-a-workflow#description). + +## 5. Run it + +Click **Run workflow**, select your repository, and leave the git ref empty to use the default branch. + +Console provisions a sandbox, clones the repository, and runs your agent. Click **View runs** to watch the +chain: each step shows status and duration as it completes, and the run page lists findings and output +deliveries when it's done. + +Your pull request trigger is live too — open a pull request in that repository and the workflow fires on its +own, against the pull request's head. + + + The **Comment on triggering pull request** output only acts on pull-request runs, so it's skipped on the + manual run you just did. Open a pull request to see it deliver. + + +## What you just built + +- **Context** — a connected repository agents can read +- **Harness** — your own agent, defined in YAML frontmatter and Markdown +- **Workflow** — that agent running automatically on every pull request, reporting to your team + +Each layer is replaceable independently. Swap the agent, add a step, change where results go — the rest keeps +working. + +## Where to go next + + + + Skills, detections, and the full tool surface. + + + + Connect a scanner and find what's actually reachable. + + + + Block merging until security review passes. + + + + Run the same agents on local, uncommitted code. + + diff --git a/workflows/agent-chain.mdx b/workflows/agent-chain.mdx new file mode 100644 index 0000000..ea37fe4 --- /dev/null +++ b/workflows/agent-chain.mdx @@ -0,0 +1,102 @@ +--- +title: The agent chain +description: 'How workflow steps run in sequence, pass results forward, and how to order them.' +--- + +## How the chain runs + +The agents you add to a workflow form an ordered chain. When a run starts, an orchestrating agent takes your +workflow definition and drives it: + +1. It reads your **description** as the goal of the whole workflow. +2. It walks the chain **in declared order**, spawning one agent at a time. +3. For each step it composes a briefing from the workflow's intent, that agent's role, what triggered the run + (for a pull-request run: the PR number, head and base commits, the diff), and the relevant parts of earlier + steps' results. +4. It **waits** for each step to finish before starting the next. +5. When every step succeeds, it writes a short summary of what each step did and how the results chained + together. + +Steps never run in parallel. That's the point of a chain — each step gets to see what the previous one +produced. + + + If a step fails, the run **stops immediately** and later steps do not run. Workflow steps do not retry by + default. Design chains so the expensive, broad step comes first and the steps that depend on it come after. + + +## Passing results between steps + +You don't wire inputs and outputs together by hand. Each agent returns a summary of what it did, and the +orchestrator quotes the relevant parts of that summary into later steps' briefings verbatim. + +In practice this means a two-step chain works because the second agent is *told* what the first one found — +for example, a scanner reports findings to a manifest, and the next agent is briefed on where that manifest is +and what's in it. + +What this does **not** do is let you transform or filter results between steps. If you need different +handling, that belongs inside an agent, not between them. + +## The briefing is why descriptions matter + +Two descriptions shape every step's briefing: + +- **The workflow's description** — quoted as the overall intent. +- **Each agent's `description`** — reflected back at the agent to confirm its role. + +A vague description on either produces a vague briefing. This is the most common reason a chain +underperforms, and it's usually fixed by editing prose rather than by changing agents. See +[writing an agent](/harness/writing-an-agent#description-is-not-a-comment). + +## Choosing agents + +Any agent can be a step — Console's or your own. Your organization's agents appear in the picker alongside the +built-in ones, and the orchestrator treats them identically. + +- For what ships with Console and what each is for, see [the agent library](/harness/agent-library). +- To write your own, see [writing an agent](/harness/writing-an-agent). + +A few agents in the library are designed to be spawned *by* other agents rather than used as steps directly — +[noted here](/harness/agent-library#agents-spawned-by-other-agents). + +## Ordering + +The rules of thumb: + +- **Broad before narrow.** Discover first, then act on what was discovered. +- **Expensive before dependent.** If a step fails, everything after it is skipped — so put the step most + likely to fail where it costs least. +- **One step is a valid chain.** A single scanner plus an [output](/workflows/outputs) is a complete, useful + workflow. Don't add steps for symmetry. + +## Common chains + +| Goal | Chain | +| --- | --- | +| Fast pull request check | `vulnerability-scanner-basic` | +| Standard pull request review | `vulnerability-scanner-standard` | +| Turn findings into permanent rules | `vulnerability-scanner-standard` → `detection-author` | +| Apply everything you've already vetted | `detections-runner` | +| Deep audit | `vulnerability-scanner-comprehensive` → `security-analyzer` | + +## Editing a chain + +In the workflow editor the chain is drawn left to right as pills with arrows: + +- **Add** with the **+** button, which searches the agent library. +- **Reorder** by dragging a pill. +- **Remove** via the grip icon on a pill. + +Between 1 and 20 steps. + +## Next steps + + + + What's available to chain. + + + + Deliver what the chain produces. + + diff --git a/workflows/artifacts.mdx b/workflows/artifacts.mdx new file mode 100644 index 0000000..f055a0f --- /dev/null +++ b/workflows/artifacts.mdx @@ -0,0 +1,90 @@ +--- +title: What a run produces +description: 'Findings, patches, and the artifact model behind them.' +--- + +## Two kinds of result + +A workflow run produces results in two places, and it's worth keeping them straight: + +| Result | Where it lives | Durable? | +| --- | --- | --- | +| **Findings and patches** | Your organization, attached to the project and the run | Yes — they outlive the run | +| **Output deliveries** | The pull request, as comments or a check | Yes, but they're a *copy* delivered elsewhere | + +The run itself is a record — status, per-step timings, which agents ran. The *results* are what the agents +recorded while running. + +## Findings + +The primary output. When an agent confirms a vulnerability it records a +[finding](/context/findings) with the affected file and symbol, its reasoning, and provenance back to the run +and the detection that produced it. + +Findings are deduplicated per project, so a recurring issue updates in place rather than accumulating +duplicates across runs. + +## Patches + +When patch generation is part of the chain, an agent that fixes a finding records the resulting diff against +that finding. + +Patches are verified before they're recorded, not proposed speculatively: the agent edits the file, re-runs +the detection to confirm the match is gone — for policy detections, an independent read-only verifier judges +the result — and only then captures the diff. That's why a finding's suggested fix can +[open a pull request](/context/findings#accepting-a-fix) directly. + +## The artifact model + +Underneath findings and patches, Console stores results as **artifacts**. An artifact has: + +- A **kind** — what type of thing it is +- A **summary** — a short description +- **Files** — one or more, each with a path, a content type, and its contents +- **Subjects** — optionally, the findings the artifact addresses + +This model is deliberately open-ended: an artifact is *a set of files an agent produced, with provenance and +optional linkage to the findings it relates to*. Nothing about it is specific to security. A risk assessment, +a compliance report, a threat model, a generated test suite are all describable in the same shape. + + + Today the only artifact kind produced in practice is the **patch**. The model is general, but patch + generation is the one producer that's wired up. Treat the broader shape as where this is heading, not as a + set of features to build against right now. + + +## Getting results out + +Recording a result and delivering it are different steps. A run's results sit in Console until an +[output](/workflows/outputs) pushes them somewhere: + +| Output | Delivers | +| --- | --- | +| Comment on triggering pull request | Findings as review comments, with links back to the full finding | +| Gate merging on security review | A required check reflecting the run's verdict | + +Both act on the pull request that triggered the run, so they're skipped on manual runs. + + + These two destinations are the whole set today. There's no built-in delivery to ticketing systems, chat + tools, or arbitrary webhooks, and no general outbound-write tool — `web_fetch` performs reads only. If you + need results elsewhere, read them from Console rather than expecting a workflow to push them. + + +## Inspecting a run's results + +Open the run from **Runs** or from **View runs** on the workflow. You'll see the step chain with per-step +status and duration, the findings the run produced, and each output with its delivery status. See +[running a workflow](/workflows/running#follow-a-run). + +## Next steps + + + + Review results and accept fixes. + + + + Deliver results to pull requests. + + diff --git a/workflows/create-a-workflow.mdx b/workflows/create-a-workflow.mdx new file mode 100644 index 0000000..a3b52f7 --- /dev/null +++ b/workflows/create-a-workflow.mdx @@ -0,0 +1,147 @@ +--- +title: Create a workflow +description: 'Build a workflow section by section, and what each part of the editor is for.' +--- + +## Before you start + +You need at least one **connected repository**. Workflows run against the projects Console has access +to through your source control provider — if the Projects page is empty, connect a repository first. + +You do not need to create any agents. Console ships a library of agents that covers scanning, +detection authoring, patching, and review. See [the agent library](/harness/agent-library) for the catalog. + +## Create it + +From **Workflows**, click **New workflow**. You land in the editor with an empty draft and four +sections to fill in, top to bottom. Nothing is saved until you click **Save**. + + + Click **Save** at any point to see what's still missing — incomplete fields highlight in red with a + message explaining what's required. Nothing is submitted until the draft is valid. + + +## Name + +The field at the top of the editor. It identifies the workflow in lists, run history, and — if you add +a merge gate — in the GitHub check that appears on pull requests. + +- Required, and must be unique within your organization. +- Choose something durable. Once a workflow has a **merge gate** output, its name is locked, because + the check name customers pin in branch protection is derived from it. Renaming would silently stop + the check from reporting and leave pull requests waiting forever. To rename, remove the merge gate + output, save, then rename. + +## Description + +A short statement of what the whole workflow is for. + +This is the most important field in the editor, and the easiest to underestimate. Console runs your +chain through an orchestrating agent, and that orchestrator: + +1. Reads the description as the **intent** of the entire workflow. +2. Composes a briefing for each step, quoting the description so the agent knows the larger goal. +3. Passes the relevant parts of earlier steps' results into later steps. + +So the description isn't documentation for your teammates — it's the instruction that shapes every +step's task. Write it as a goal, in plain language. + +**Good:** + +> Scan the pull request for injection and access-control vulnerabilities, then generate a patch for +> anything confirmed exploitable and post it as a review comment. + +**Too vague:** + +> Security workflow. + + + If your description and your agent chain disagree, the chain wins — the orchestrator runs the steps + as declared and notes the disagreement in its summary. Keep the two aligned. + + +## Triggers + +*Optional.* When the workflow should fire on its own. + +You can save a workflow with no triggers at all and still run it by hand whenever you like — every +workflow supports manual runs. Add a trigger only when you want it to fire automatically. + +Click **Add trigger** and pick a type. Today that means **on pull requests**, where you choose which +repositories to watch and, optionally, restrict it to specific base branches. + +If you add a trigger, it must name at least one repository, or the workflow won't save. + +See [Triggers](/workflows/triggers) for the full details. + +## Agents + +*Required.* The ordered chain of agents that does the actual work. + +Click the **+** button to search the agent library and add an agent. Each one becomes a step, drawn as +a pill with an arrow to the next — the chain reads left to right, and that's the order it runs in. + +- **At least one** agent, **at most twenty**. +- **Reorder** by dragging a pill. +- **Remove** a step by clicking the grip icon on its pill and choosing *Remove from chain*. + +Steps run strictly one at a time. Each waits for the previous one to finish, and if any step fails the +run stops there — later steps don't run. + +See [The agent chain](/workflows/agent-chain) for how sequencing works, and [the agent library](/harness/agent-library) for which agents to use. + +## Output + +*Optional.* Where the results go when a run completes. + +Click **Add output** and choose a destination: + +- **Comment on triggering pull request** — posts review comments on the pull or merge request that + fired the run. +- **Gate merging on security review** — adds a required check that blocks merging until the agent's + verdict passes or a human approves the pull request. + +You can add one of each, but not two of the same type — an already-added destination shows as +*Already added* in the picker. + +Both destinations describe a *triggering pull request*, so they only do something on runs that a pull +request started. On a manual run they're skipped. See [Outputs](/workflows/outputs). + +## Save + +Click **Save**. Console creates the workflow, then applies your triggers and outputs. + +If a trigger or output can't be applied, the workflow is still created — you'll land on its detail page +with a banner explaining what didn't stick, and you can finish wiring it up there. + +Clicking **Cancel** with unsaved changes asks you to confirm before discarding. + +## Edit or delete later + +Open a workflow from the **Workflows** list to see it in read-only form, then click **Edit**. The same +four sections become editable, with **Save**, **Cancel**, and **Delete** in the header. + +Editing a workflow does not affect runs already in flight — each run uses the definition as it was +when it fired. + +## Requirements at a glance + +| Field | Rule | +| --- | --- | +| Name | Required, unique, non-blank. Locked while a merge gate output exists. | +| Description | Required, non-blank. | +| Agents | At least 1, at most 20. | +| Triggers | Optional. Any trigger you add must name at least one repository. | +| Outputs | Optional. One per destination type. | + +## Next steps + + + + Fire it manually and watch the run. + + + + The agent catalog and common chains. + + diff --git a/workflows/outputs.mdx b/workflows/outputs.mdx new file mode 100644 index 0000000..2b5dd6f --- /dev/null +++ b/workflows/outputs.mdx @@ -0,0 +1,126 @@ +--- +title: Outputs +description: 'Post workflow results back to a pull request, and block merging until security review passes.' +--- + +## What an output does + +An output is a destination for a run's results. When a run completes, Console dispatches every output +you've configured. Outputs are optional — without one, a run still records its findings in Console; +it just doesn't push them anywhere. + +Add them from the **Output** section of the workflow editor. You can add one of each destination type, +but not two of the same kind. + +Both destinations act on *the pull request that triggered the run*, so they only do something for runs a +pull request started. On a manual run they're recorded as skipped. + +## Comment on triggering pull request + +Posts the run's findings as review comments on the pull or merge request that fired the workflow, with +links back to the full finding in Console. + +This works for both GitHub pull requests and GitLab merge requests. + +Use it when you want the agent's results to show up where reviewers already are, without changing +whether the pull request can merge. + +## Gate merging on security review + +Adds a **required check** to the pull request that blocks merging until the security review passes or a +human signs off. This is the output to use when you want the workflow to have teeth. + +### The check + +The check is named: + +``` +Console / +``` + +Its details link points at the run in Console, so anyone looking at a blocked pull request can click +through to what the agent actually found. + + + Because the check name is derived from the workflow name, Console **locks the workflow name** while a + merge gate output exists. If you rename it, the check reports under a new name, and any branch + protection rule pinned to the old name waits forever for a check that will never arrive. To rename: + remove the merge gate output, save, rename, then add the gate back. + + +### How the gate decides + +| Outcome | Check result | What it means | +| --- | --- | --- | +| Agent's verdict is **pass** | Passing | Security review passed. Merging is unblocked. | +| Agent's verdict is **fail** | Failing | Review failed. Push a commit addressing the findings. | +| Agent asks for **human review** | Action required | The agent isn't confident enough to decide. A human approval releases it. | +| **No verdict recorded** | Action required | The gate fails safe. A human approval releases it. | +| The run **errored** | Failing | Merging stays blocked — a broken run is never treated as a pass. | + +The gate fails closed by design. A workflow whose agents don't report a gate verdict will block every +pull request pending human review, rather than waving them through. + + + For the gate to pass on its own, your chain needs an agent that records a gate verdict. If none does, + the gate still works — it just always routes to human approval instead of ever passing automatically. + + +### Human approval + +Approving the pull request releases a gate that's waiting on human review, and the check flips to +passing. This also works retroactively: if someone approved the pull request before the gate finished +posting, Console resolves the gate as approved rather than leaving an already-approved pull request +stuck. + +By default, later requesting changes on a pull request whose gate was already approved does not +re-block it. + +### Requiring the check in GitHub + +Adding the output makes the check *report*. It doesn't make it *required* — that's a GitHub branch +protection setting you control: + +1. In GitHub, go to **Settings → Branches** (or **Rules → Rulesets**) for the repository. +2. Edit the rule protecting your target branch, e.g. `main`. +3. Enable **Require status checks to pass before merging**. +4. Search for `Console / ` and add it. + + + Let the workflow run on one pull request first. GitHub only offers a check in that search box once it + has seen it report at least once. + + +Until you complete this step the check appears on pull requests as information only, and merging isn't +actually blocked. + +### GitLab + +Merge blocking isn't supported for GitLab merge requests yet. A merge gate on a GitLab run is recorded +as skipped. Use **Comment on triggering pull request** for GitLab in the meantime. + +## Checking delivery + +Every run's detail page lists the outputs it dispatched and whether each was delivered: + +| Status | Meaning | +| --- | --- | +| **Sent** | Delivered successfully. | +| **Failed** | Delivery failed — for example, Console lacks permission on the repository. | +| **Skipped** | Not applicable to this run, such as a pull request comment on a manually-fired run. | +| **Pending** | Delivery hasn't completed yet. | + +If an output shows **Failed**, check that the Console GitHub App is still installed for the repository +and has permission to write checks and pull request comments. + +## Next steps + + + + Fire a run and confirm your outputs delivered. + + + + Outputs need a pull-request run to act on. + + diff --git a/workflows/overview.mdx b/workflows/overview.mdx new file mode 100644 index 0000000..65649bb --- /dev/null +++ b/workflows/overview.mdx @@ -0,0 +1,85 @@ +--- +title: Workflows +description: 'Chain security agents into a repeatable, automated review you can trigger on demand or on every pull request.' +--- + +## What is a workflow? + +A workflow is a saved sequence of agents that Console runs against your repositories. Where chat is a +conversation you drive turn by turn, a workflow is the same agents running unattended: you define the +sequence once, and Console executes it whenever you fire it manually or when a pull request opens. + +Workflows are where the other two layers pay off. The [harness](/harness/overview) gives you agents, +skills, and detections; [context](/context/overview) gives them your code and your vendors' signal. A +workflow puts those to work on a schedule, with no one in the loop, and delivers the result to where your +team already works. + +A typical workflow scans a repository for vulnerabilities, then hands what it found to a second agent +that authors reusable detections or generates patches — and finally posts the result back to the pull +request that started it. + +## Anatomy of a workflow + +Every workflow is made of four parts. Only the first two are required. + +| Part | Required | What it does | +| --- | --- | --- | +| **Name** | Yes | Identifies the workflow. Must be unique in your organization. | +| **Description** | Yes | States the workflow's *intent*. This is not a comment — the orchestrating agent reads it and briefs every step against it. | +| **Agents** | Yes | The ordered chain of agents to run. At least one, at most twenty. | +| **Triggers** | No | When the workflow fires automatically. Manual runs are always available without a trigger. | +| **Outputs** | No | Where the results go when the run finishes — a pull request comment, a merge gate, or both. | + + + The description does real work. Console's orchestrator reads it as the goal of the whole workflow and + quotes it into each agent's briefing, so a vague description produces vague step instructions. See + [Create a workflow](/workflows/create-a-workflow#description) for how to write one. + + +## What happens during a run + +When a workflow fires, Console does the following for **each** repository you targeted: + +1. **Creates a run.** Every run gets its own record, visible under [Runs](/workflows/running). +2. **Snapshots the definition.** The run captures the workflow's name, description, and steps as they + are at that moment. Editing the workflow later never changes a run that is already in flight. +3. **Provisions an isolated sandbox** and clones the repository into it. For pull-request runs, it + clones the pull request's head — not the default branch. +4. **Runs the chain in order.** An orchestrating agent spawns each step's agent one at a time, waits + for it to finish, and passes the relevant parts of earlier steps' results into the next briefing. +5. **Stops on failure.** If a step fails, the run stops there and later steps do not run. +6. **Records findings** from the run, and **dispatches your outputs** once the run completes. + +Targeting three repositories produces three independent runs — one per repository — not one run that +loops. Each gets its own sandbox, and one failing does not stop the others. + +## Where workflows live + +- **Workflows** — the list of workflows in your organization, with a run count for the last 7, 30, or + 90 days. This is where you create, edit, and manually fire them. +- **Runs** — the execution history for every workflow, including per-step status, findings, and + whether each output was delivered. + +## Next steps + + + + Walk through the editor section by section. + + + + How steps run in sequence and how to order them. + + + + Fire on pull requests, or run on demand. + + + + Post comments and gate merges on review. + + + + Fire a workflow and read its run history. + + diff --git a/workflows/running.mdx b/workflows/running.mdx new file mode 100644 index 0000000..e7b3cc7 --- /dev/null +++ b/workflows/running.mdx @@ -0,0 +1,99 @@ +--- +title: Running a workflow +description: 'Fire a workflow manually, then follow the run through to its results.' +--- + +## Run a workflow manually + +Open the workflow from the **Workflows** list and click **Run workflow**. + +In the dialog: + +1. **Repositories** — select one or more connected repositories to run against. The button stays + disabled until you pick at least one. +2. **Git ref** *(optional)* — a branch, tag, or commit. It's applied to **every** selected repository, so + leave it empty unless they share the ref you want. Empty means each repository's own default branch. +3. Click **Run workflow**. + +Console fires immediately and starts one run per repository. + + + Firing a workflow manually cancels any earlier **manual** runs of that workflow still pending or + running. It does not touch runs started by a pull request. See + [superseding in-flight runs](/workflows/triggers#superseding-in-flight-runs). + + +## Run it automatically + +Add a [pull request trigger](/workflows/triggers#pull-request-triggers) and Console fires the workflow +whenever a pull request in the selected repositories is opened, reopened, or updated with new commits. +Automatic runs always use the pull request's head commit. + +## Follow a run + +From a workflow, click **View runs**. From anywhere, the **Runs** page lists every run in your +organization; filter it by **status** or by **workflow name**. + +Open a run to see: + +- **The chain** — each step drawn in order with its agent name, how long it took, and its status icon. + Steps light up as they complete, so you can watch progress on a running workflow. +- **Findings** — what the run confirmed, linked through to the full finding. +- **Outputs** — each configured destination and whether it was delivered. See + [checking delivery](/workflows/outputs#checking-delivery). +- **Go to workflow** — jump back to the definition that produced this run. + +A run page refreshes itself while the run is active, so you can leave it open. + +## Run statuses + +| Status | Meaning | +| --- | --- | +| **Pending** | Queued. Console is provisioning a sandbox and cloning the repository. | +| **Running** | The agent chain is executing. | +| **Completed** | Every step finished successfully and outputs were dispatched. | +| **Error** | A step failed. The run stopped there and later steps did not run. | +| **Cancelled** | Cancelled by you, or superseded by a newer run from the same trigger. | + +## Cancel a run + +Open the run and click **Cancel**. Console stops the agent chain and tears down the sandbox. + +Cancelling doesn't undo anything already done — findings recorded before the cancellation stay, and any +output already delivered stays delivered. If a merge gate was posted, a cancelled run leaves it blocking +rather than passing. + +## Reading a failed run + +A run ending in **Error** stops at the failing step. Open it and look at the chain: the step with the +error icon is where it stopped, and everything to its right never ran. + +Common causes: + +- **The repository couldn't be cloned.** Usually a permissions problem — confirm the Console GitHub App + is still installed for that repository, or that a `Git ref` you pinned actually exists. +- **A step exceeded its time budget.** Long chains against large repositories can run out of budget. + Splitting a broad scan into a narrower profile (see + [scanning agents](/harness/agent-library#scanning)) usually resolves it. +- **A step named an agent that no longer exists.** If you deleted an agent your workflow references, + edit the workflow and repair the chain. + +Because Console snapshots the workflow definition when a run fires, fixing the workflow doesn't repair +an existing failed run — edit the workflow, then fire it again. + +## Run history at a glance + +The **Workflows** list shows a run count per workflow, and you can switch the window between **7d**, +**30d**, and **90d** to see how often each one has been firing. + +## Next steps + + + + Push results to pull requests automatically. + + + + Adjust scan depth and sequencing. + + diff --git a/workflows/triggers.mdx b/workflows/triggers.mdx new file mode 100644 index 0000000..758a1bd --- /dev/null +++ b/workflows/triggers.mdx @@ -0,0 +1,102 @@ +--- +title: Triggers +description: 'Fire a workflow on demand, or automatically when a pull request opens.' +--- + +## What a trigger does + +A trigger decides **when** a workflow fires and **which repositories** it fires against. Triggers are +optional: every workflow can always be run by hand, so add a trigger only when you want Console to fire +it without you. + +Manage triggers in the **Triggers** section of the workflow editor. Click **Add trigger**, pick a type, +and choose its repositories. + +## Manual runs + +Every workflow supports manual runs. There's nothing to configure and nothing appears in the Triggers +section — it's always available. + +With a manual run you choose the repositories at the moment you fire it, and you can optionally pin a +specific branch, tag, or commit. See [Running a workflow](/workflows/running). + +## Pull request triggers + +Select **On pull requests** to fire the workflow automatically against pull and merge requests. + +### When it fires + +Despite the name, this trigger fires on more than just opening a pull request. It fires when a pull +request is: + +- **Opened** +- **Reopened** +- **Updated with new commits** + +An update supersedes the earlier run — Console cancels the in-flight run and starts a fresh one at the +new head commit, so a merge gate re-evaluates against the code that's actually there now rather than +staying stuck on a verdict for a commit nobody is merging. + +### Repositories + +Choose one or more connected repositories to watch. A trigger with no repositories won't save. + +Pull-request runs always clone **the pull request's head** — the proposed code, not the base branch. You +cannot pin a branch or commit on a pull request trigger, because the head is what's under review. + +### Base branch filter + +By default the workflow fires on pull requests targeting **any** branch. Add one or more base branches +to narrow it to pull requests aimed at those branches specifically — a common choice is to gate only +what's merging into `main`. + +Matching is exact. A pull request whose base branch isn't in your list is skipped before Console +provisions anything, so filtered-out pull requests cost you nothing. + +### Draft pull requests + +Draft pull requests are always skipped. Work in progress doesn't warrant a run, and the workflow will +fire once the pull request is marked ready. + +### Requirements + +Pull request triggers depend on Console receiving events from your source control provider, which means +the repository must be connected and the Console GitHub App installed for it. See +[Installation](/install-console#install-the-github-app). + +## One run per repository + +A trigger that names three repositories produces **three runs** when it fires — one per repository, each +in its own sandbox. Runs from the same firing are grouped, so you can see them together in the run +history, but they succeed or fail independently. + +## Superseding in-flight runs + +When a workflow fires again from the same trigger, Console cancels any of that trigger's runs that are +still pending or running before starting the new ones. + +This keeps a rapidly-updated pull request from stacking up parallel sandboxes for commits that are +already stale, and it means the verdict on a pull request always reflects its latest commit. It applies +to manual runs too: firing a workflow manually while an earlier manual run is still going replaces it. + + + Cancellation is scoped to the trigger. A manual run does not cancel a pull-request run of the same + workflow, and vice versa. + + +## Scheduled triggers + +Scheduled triggers aren't available to create yet. If your organization has any from an earlier +configuration, the editor displays them read-only. + +## Next steps + + + + Decide what a triggered run posts back. + + + + Fire a run and follow it through. + +