Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/ISSUE_TEMPLATE/bug_report.yml
Original file line number Diff line number Diff line change
Expand Up @@ -16,9 +16,9 @@ body:
options:
- CLI (`opfor run` / `opfor setup`)
- CLI — Autonomous mode (`opfor hunt`)
- MCP Server (`@agent-opfor/mcp`)
- MCP Server (`@keyvaluesystems/agent-opfor-mcp`)
- Browser Extension
- SDK (`@agent-opfor/sdk`)
- SDK (`@keyvaluesystems/agent-opfor-sdk`)
- Evaluators / Suites
- Judge (LLM-as-judge)
- Report (HTML / JSON output)
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -89,7 +89,7 @@ jobs:
- run: npm run build

- name: Core tests
run: npm test --workspace=@agent-opfor/core
run: npm test --workspace=@keyvaluesystems/agent-opfor-core

- name: SDK tests
run: npm test --workspace=@agent-opfor/sdk
run: npm test --workspace=@keyvaluesystems/agent-opfor-sdk
2 changes: 2 additions & 0 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -18,3 +18,5 @@ jobs:
- uses: googleapis/release-please-action@v4
with:
token: ${{ secrets.GITHUB_TOKEN }}
config-file: release-please-config.json
manifest-file: .release-please-manifest.json
8 changes: 8 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,14 @@ cli/node_modules/
.opfor
tracedata.json

# Vendored into core/ at pack time by its prepack script (see core/package.json).
# Cleaned up by postpack; ignored here so an interrupted pack can't commit copies.
core/evaluators/
core/suites/
core/skills/
core/data/
core/atlas-data/

# Pre-refactor local trees / accidental rebuilds (canonical code is cli/, core/, mcp/)
/src/
src/agent/**/dist/
Expand Down
24 changes: 12 additions & 12 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,15 +18,15 @@ Opfor is an open-source red-teaming toolkit for AI agents and MCP servers. It ge
| Browser extension | Click the toolbar icon on any chat UI | Non-developers — QA, PMs, security analysts |
| MCP server | `opfor_setup`, `opfor_run` tools | MCP-compatible host (Cursor, Claude Desktop) |
| Skills | `/opfor-setup`, `/opfor-run` slash commands | AI coding agent reads markdown skill files |
| SDK | `import { run, hunt } from "@agent-opfor/sdk"` | Developers embedding opfor in their own code |
| SDK | `import { run, hunt } from "@keyvaluesystems/agent-opfor-sdk"` | Developers embedding opfor in their own code |

---

## Monorepo structure

```
```text
opfor/
├── core/ # @agent-opfor/core — shared engine (npm workspace, compiled to core/dist/)
├── core/ # @keyvaluesystems/agent-opfor-core — shared engine (npm workspace, compiled to core/dist/)
│ └── src/
│ ├── autonomous/ # Autonomous red-teaming orchestration (orchestrator, prompts, tools, state, report, knowledge)
│ ├── catalog/ # discoverEvaluators.ts, loadCatalog.ts — YAML evaluator/suite discovery
Expand All @@ -46,7 +46,7 @@ opfor/
│ ├── prompts/ # Inlined system prompts (attacker, judge) used by core
│ └── util/ # yamlFrontmatter.ts and other utility helpers
├── runners/
│ ├── cli/ # @agent-opfor/cli — `opfor` CLI binary (npm workspace)
│ ├── cli/ # @keyvaluesystems/agent-opfor-cli — `opfor` CLI binary (npm workspace)
│ │ └── src/
│ │ ├── index.ts # CLI entrypoint (commander) — registers setup, run, and hunt
│ │ ├── commands/
Expand All @@ -55,18 +55,18 @@ opfor/
│ │ │ └── hunt.ts # `opfor hunt` — autonomous red-teaming with agentic orchestration
│ │ └── lib/
│ │ └── artifacts.ts # .opfor/configs/ + .opfor/reports/ path helpers
│ ├── mcp/ # @agent-opfor/mcp — MCP server runner (npm workspace)
│ ├── mcp/ # @keyvaluesystems/agent-opfor-mcp — MCP server runner (npm workspace)
│ │ └── src/
│ │ └── index.ts # MCP server entrypoint — registers tools, stdio transport
│ ├── sdk/ # @agent-opfor/sdk — programmatic SDK (npm workspace)
│ ├── sdk/ # @keyvaluesystems/agent-opfor-sdk — programmatic SDK (npm workspace)
│ │ └── src/
│ │ └── index.ts # SDK entrypoint
│ └── extension/ # @agent-opfor/extension — Chrome MV3 browser extension (npm workspace)
│ └── extension/ # @keyvaluesystems/agent-opfor-extension — Chrome MV3 browser extension (npm workspace)
│ ├── service_worker.js # Entry point — message routing only; imports modules below
│ ├── orchestrator.js # Main run loop: locate → attack → extract → reset → judge (calls runAllBrowser from bundled core)
│ ├── llmUiActions.js # DOM-specific LLM helpers (input picker, UI planner, message shortener)
│ ├── domTarget.js # Adapter exposing the DOM send/extract path as a core AgentTarget
│ ├── dist/core.bundle.js # esbuild bundle of @agent-opfor/core/browser (attack + judge engine)
│ ├── dist/core.bundle.js # esbuild bundle of @keyvaluesystems/agent-opfor-core/browser (attack + judge engine)
│ ├── frameDiscovery.js # Frame collection, scoring, chat-frame selection
│ ├── domActions.js # chrome.scripting wrappers (send, click, verify, vendor APIs)
│ ├── responseExtractor.js # Three-phase polling extractor for bot responses
Expand Down Expand Up @@ -135,7 +135,7 @@ opfor/
│ ├── mcp.md # MCP server (runner) setup + tools reference
│ ├── browser-extension.md # Browser extension guide
│ ├── skills.md # Skill bundle usage
│ ├── sdk.md # SDK (@agent-opfor/sdk) reference
│ ├── sdk.md # SDK (@keyvaluesystems/agent-opfor-sdk) reference
│ ├── evaluators.md # Evaluator + suite reference
│ ├── evaluator-schema.md # Evaluator YAML schema
│ └── telemetry.md # Trace-aware testing (Langfuse / Netra)
Expand All @@ -158,7 +158,7 @@ npm run format:check # prettier --check
npm test # vitest in core/
```

`core` must compile before any runner — `runners/{cli,mcp}` import from `core/dist/`, and `runners/extension` esbuild-bundles `@agent-opfor/core/browser` at build time. Always run `npm run build` from the repo root, never per-package.
`core` must compile before any runner — `runners/{cli,mcp}` import from `core/dist/`, and `runners/extension` esbuild-bundles `@keyvaluesystems/agent-opfor-core/browser` at build time. Always run `npm run build` from the repo root, never per-package.

---

Expand Down Expand Up @@ -197,7 +197,7 @@ npm test # vitest in core/
| `runners/extension/service_worker.js` | Extension entry point — message routing; imports from focused ES modules |
| `runners/extension/orchestrator.js` | Full adaptive run loop — drives `runAllBrowser` against `DomTarget` |
| `runners/extension/domTarget.js` | Implements the core `AgentTarget` interface against the live chat DOM |
| `runners/extension/dist/core.bundle.js` | esbuild bundle of `@agent-opfor/core/browser`; supplies `runAllBrowser` + `generateNextTurn` + judge |
| `runners/extension/dist/core.bundle.js` | esbuild bundle of `@keyvaluesystems/agent-opfor-core/browser`; supplies `runAllBrowser` + `generateNextTurn` + judge |

---

Expand Down Expand Up @@ -342,7 +342,7 @@ For a new MCP _transport_ (beyond stdio/url):
- **No barrel re-exports** — import directly from the file that owns the symbol
- **Error messages are actionable** — tell the user what to fix, not just what went wrong
- **Evaluator files are data** — no business logic in `.md` files; logic lives in `core/src/evaluators/`
- **Never invoke the CLI as a subprocess from the MCP server** — call `@agent-opfor/core` directly
- **Never invoke the CLI as a subprocess from the MCP server** — call `@keyvaluesystems/agent-opfor-core` directly

---

Expand Down
8 changes: 4 additions & 4 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -456,7 +456,7 @@ Only submit findings for systems you are authorized to test, or where you have c
### Project structure

```
core/ ← @agent-opfor/core — shared engine (npm workspace)
core/ ← @keyvaluesystems/agent-opfor-core — shared engine (npm workspace)
src/
autonomous/ ← autonomous red-teaming orchestration (opfor hunt)
catalog/ ← evaluator/suite discovery + loading
Expand All @@ -473,9 +473,9 @@ core/ ← @agent-opfor/core — shared engine (npm workspa
targets/ ← agent + MCP target adapters
telemetry/ ← Langfuse + Netra adapters
runners/
cli/ ← @agent-opfor/cli — `opfor setup`, `opfor run`, `opfor hunt`
mcp/ ← @agent-opfor/mcp — opfor as an MCP server
sdk/ ← @agent-opfor/sdk — programmatic SDK for embedding
cli/ ← @keyvaluesystems/agent-opfor-cli — `opfor setup`, `opfor run`, `opfor hunt`
mcp/ ← @keyvaluesystems/agent-opfor-mcp — opfor as an MCP server
sdk/ ← @keyvaluesystems/agent-opfor-sdk — programmatic SDK for embedding
extension/ ← Chrome MV3 browser extension
evaluators/
agent/ ← agent evaluators by category
Expand Down
20 changes: 10 additions & 10 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ Apache 2.0. Built from India.
## Quick Start

```bash
npm install -g @agent-opfor/cli
npm install -g @keyvaluesystems/agent-opfor-cli
export OPENAI_API_KEY=your-key # or GEMINI_API_KEY, ANTHROPIC_API_KEY, etc.
```

Expand Down Expand Up @@ -75,13 +75,13 @@ Most red-team tooling in this space is excellent at one thing — a probe librar

Different people on your team need different entry points. Opfor ships five.

| Mode | How | Best for |
| ------------------------ | ----------------------------------------------------------------------- | ----------------------------------------------------------------------------------------- |
| 🖥️ **CLI** | `opfor setup` → `opfor run` | Engineers, CI/CD, terminal-first workflows |
| 🌐 **Browser extension** | Install the extension, click the icon on any chat interface | Product managers, designers, QA, security analysts — anyone who can't or won't write code |
| 🤖 **MCP server** | Register opfor in Cursor or Claude Desktop, then ask in chat | AI coding agents that test your other agents |
| ⚡ **Skills** | `/opfor-setup` · `/opfor-run` · `/opfor-mcp-setup` · `/opfor-mcp-run` | Developers who want one-command testing inside their IDE |
| 📦 **SDK** | `npm install @agent-opfor/sdk`, then call `run` / `hunt` from your code | Programmatic red-teaming and custom workflows |
| Mode | How | Best for |
| ------------------------ | --------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------- |
| 🖥️ **CLI** | `opfor setup` → `opfor run` | Engineers, CI/CD, terminal-first workflows |
| 🌐 **Browser extension** | Install the extension, click the icon on any chat interface | Product managers, designers, QA, security analysts — anyone who can't or won't write code |
| 🤖 **MCP server** | Register opfor in Cursor or Claude Desktop, then ask in chat | AI coding agents that test your other agents |
| ⚡ **Skills** | `/opfor-setup` · `/opfor-run` · `/opfor-mcp-setup` · `/opfor-mcp-run` | Developers who want one-command testing inside their IDE |
| 📦 **SDK** | `npm install @keyvaluesystems/agent-opfor-sdk`, then call `run` / `hunt` from your code | Programmatic red-teaming and custom workflows |

All five share the same evaluators, attack templates, and judge logic.

Expand Down Expand Up @@ -160,10 +160,10 @@ This is the path for the half of every product team that doesn't open a terminal

## SDK — embed red-teaming in your code

The SDK is opfor's programmatic path. Install `@agent-opfor/sdk`, call `run` or `hunt`, and get structured results back — no CLI, no config files, no subprocess.
The SDK is opfor's programmatic path. Install `@keyvaluesystems/agent-opfor-sdk`, call `run` or `hunt`, and get structured results back — no CLI, no config files, no subprocess.

```typescript
import { Opfor } from "@agent-opfor/sdk";
import { Opfor } from "@keyvaluesystems/agent-opfor-sdk";

const opfor = new Opfor({ apiKey: process.env.ANTHROPIC_API_KEY });

Expand Down
2 changes: 1 addition & 1 deletion core/package.json
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
{
"name": "@agent-opfor/core",
"name": "@keyvaluesystems/agent-opfor-core",
"version": "0.9.0",
"description": "Opfor core engine — attacker prompt generation, judge, and execution shared by all runners",
"license": "Apache-2.0",
Expand Down
4 changes: 2 additions & 2 deletions core/src/autonomous/index.ts
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
/**
* Autonomous red-team engine — public API.
*
* Consumers import from "@agent-opfor/core/autonomous/index.js" (or sub-paths
* like "@agent-opfor/core/autonomous/report/types.js" for deeper access).
* Consumers import from "@keyvaluesystems/agent-opfor-core/autonomous/index.js" (or sub-paths
* like "@keyvaluesystems/agent-opfor-core/autonomous/report/types.js" for deeper access).
*/

export { runAutonomous } from "./orchestrator/run.js";
Expand Down
16 changes: 11 additions & 5 deletions core/src/autonomous/knowledge/load.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@
// Uses core's shared frontmatter parser. Resolves the bundled `data/`
// directory relative to this module so it works regardless of the caller's cwd.

import { existsSync } from "node:fs";
import { readFile, readdir, writeFile, mkdir } from "node:fs/promises";
import path from "node:path";
import { fileURLToPath } from "node:url";
Expand All @@ -12,13 +13,18 @@ import type { KnowledgeBase, VulnClass, Persona, Strategy, KnowledgeKind } from
const __dirname = path.dirname(fileURLToPath(import.meta.url));

/**
* Resolve the seed data directory. At runtime this module lives in
* `core/dist/autonomous/knowledge/load.js`, so the bundled seeds are at
* `../../../../runners/cli/data` (the CLI's `data/` dir shipped via
* package.json `files`). Callers should always pass `seedDir`
* explicitly; this fallback is a best-effort for in-repo dev use.
* Resolve the seed data directory. Two layouts are supported:
* - Published package: `<core>/data` (vendored into the tarball at pack time). This module
* lives at `core/dist/autonomous/knowledge/load.js`, so the package root is 3 levels up.
* - Monorepo dev: `<repo>/runners/cli/data` (the source of truth, 4 levels up).
*
* Callers may still pass `seedDir` explicitly to override.
*/
function defaultSeedDir(): string {
const packageLocal = path.resolve(__dirname, "../../../data");
if (existsSync(packageLocal)) {
return packageLocal;
}
return path.resolve(__dirname, "../../../../runners/cli/data");
}

Expand Down
2 changes: 1 addition & 1 deletion core/src/autonomous/lib/types.ts
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
// Shared option/config types for the autonomous red-team runner.
// This package is fully standalone — it does NOT import from @agent-opfor/core.
// This package is fully standalone — it does NOT import from @keyvaluesystems/agent-opfor-core.

/** How the target HTTP agent maintains conversation state. */
export type TargetMode = "stateless" | "stateful";
Expand Down
2 changes: 1 addition & 1 deletion core/src/browser.ts
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
// Browser-safe entry for @agent-opfor/core. Re-exports only the parts of the engine
// Browser-safe entry for @keyvaluesystems/agent-opfor-core. Re-exports only the parts of the engine
// that are reachable from a Chrome MV3 extension or any other non-Node runtime.
//
// What's IN: attacker prompt generation, multi-turn escalation, judge logic,
Expand Down
14 changes: 13 additions & 1 deletion core/src/catalog/loadEvaluatorCatalog.ts
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ import { type EvaluatorCategory } from "../config/evaluatorsLayout.js";
import { discoverEvaluatorFiles, discoverSuiteFiles } from "./discoverEvaluators.js";
import { resolveStandardsFromFrontmatter } from "../evaluators/standards.js";
import { loadAtlasTechniqueIdSet } from "../standards/atlas.js";
import { log } from "../lib/logger.js";
import type { StandardsMap } from "../evaluators/schema.js";

export interface EvaluatorMeta {
Expand Down Expand Up @@ -111,7 +112,18 @@ export async function loadEvaluatorCatalog(category: EvaluatorCategory): Promise
suites: SuiteMeta[];
}> {
const validateAtlas = process.env.OPFOR_VALIDATE_ATLAS !== "0"; // On by default now
const atlasTechniqueIds = validateAtlas ? await loadAtlasTechniqueIdSet() : null;
let atlasTechniqueIds: Set<string> | null = null;
if (validateAtlas) {
try {
atlasTechniqueIds = await loadAtlasTechniqueIdSet();
} catch (e: unknown) {
// ATLAS data is an authoring-time validation aid. If it's unavailable at runtime
// (e.g. a published install where the data wasn't bundled), skip validation rather
// than crash a scan. Set OPFOR_VALIDATE_ATLAS=0 to silence this.
const msg = e instanceof Error ? e.message : String(e);
log.warn(`Skipping ATLAS technique-id validation: ${msg.split("\n")[0]}`);
}
}

// Discover and load evaluators
const discoveredEvaluators = await discoverEvaluatorFiles(category);
Expand Down
11 changes: 10 additions & 1 deletion core/src/config/evaluatorsLayout.ts
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
import { realpathSync } from "node:fs";
import { existsSync, realpathSync } from "node:fs";
import path from "node:path";
import { fileURLToPath } from "node:url";

Expand All @@ -7,6 +7,15 @@ const __dirname = path.dirname(realpathSync(fileURLToPath(import.meta.url)));
export type EvaluatorCategory = "agent" | "mcp";

export function getRepoRoot(): string {
// Bundled runner: data dirs at package root, one level up from dist/
// Check for evaluators/agent/ (not just evaluators/) to avoid false-matching
// core/src/evaluators/ which is a TypeScript source directory, not data.
const oneUp = path.resolve(__dirname, "..");
if (existsSync(path.join(oneUp, "evaluators", "agent"))) return oneUp;
// Compiled core (npm installed): data dirs colocated at core package root, 2 levels up from dist/config/
const twoUp = path.resolve(__dirname, "../..");
if (existsSync(path.join(twoUp, "evaluators", "agent"))) return twoUp;
// Monorepo: data dirs at repo root, 3 levels up from core/dist/config/
return path.resolve(__dirname, "../../..");
}

Expand Down
2 changes: 1 addition & 1 deletion core/src/report/types.ts
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@

import type { JudgeResult } from "../lib/judgeTypes.js";

/** @deprecated Use JudgeResult from @agent-opfor/core/lib/judgeTypes.js directly. */
/** @deprecated Use JudgeResult from @keyvaluesystems/agent-opfor-core/lib/judgeTypes.js directly. */
export type ReportJudge = JudgeResult;

export type DetailCard =
Expand Down
Loading
Loading