Skip to content

Gate benchmark runs on Playwright readiness #58

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
giaphutran12 wants to merge 1 commit into from
Closed

Gate benchmark runs on Playwright readiness #58

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 18 additions & 0 deletions benchmarks/dataset-agent/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -146,6 +146,24 @@ without committing raw run folders:
Agent canary actually emitted browser actions before starting a Playwright
compiler.

For browser-action canaries, add `--require-playwright-ready` to make the
benchmark fail with `failureCategory: "capability_gate"` unless the
`playwright-candidate-readiness` artifact is `ready`. This gate uses the
readiness artifact, not raw browser step counts, so it still requires
actionable browser steps, source anchors, and no Agent-disabled diagnostic.

```bash
COLLECTION_AGENT_ENABLE_AGENT=true \
COLLECTION_AGENT_POLL_TIMEOUT_MS=480000 \
COLLECTION_AGENT_PIPELINE_MODULE=./backend/BigSet_Data_Collection_Agent/src/orchestrator/pipeline.ts \
BIGSET_COLLECTION_BENCHMARK_RUNNER_MODULE=./backend/src/pipeline/collection-agent-runner.ts \
node benchmarks/dataset-agent/run-benchmark.mjs \
--require-playwright-ready \
--prompt-ids mcp-docs-pages \
--timeout-ms 900000 \
--system collection-self-heal='node --import ./backend/node_modules/tsx/dist/esm/index.mjs benchmarks/dataset-agent/adapters/collection-self-healing-adapter.mjs'
```

## Verify Self-Healing Stack

Use this before asking someone else to migrate a new collection agent into the
Expand Down
160 changes: 143 additions & 17 deletions benchmarks/dataset-agent/run-benchmark.mjs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -567,11 +567,19 @@ async function runSystemPrompt(input) {
parsedPayload,
normalized,
});
const status = infraBlockerReason
? "blocked"
: execution.exitCode === 0 && parsedPayload && answerKeyScore.passed
? "ok"
: "failed";
const capabilityGateReason = infraBlockerReason
? null
: playwrightReadinessGateReason({
diagnostics: normalized.diagnostics,
requirePlaywrightReady: input.config.requirePlaywrightReady,
});
const status = benchmarkStatusForOutcome({
execution,
parsedPayload,
answerKeyScore,
infraBlockerReason,
capabilityGateReason,
});

const promptRunDirectory = join(
input.runDirectory,
Expand All @@ -597,9 +605,12 @@ async function runSystemPrompt(input) {
expectedStress: input.promptDefinition.expectedStress,
answerKey: answerKeyForPrompt(input.promptDefinition),
status,
failureCategory: status === "ok" ? undefined : (
infraBlockerReason ? "infra" : answerKeyScore.failureCategory
),
failureCategory: failureCategoryForOutcome({
status,
infraBlockerReason,
capabilityGateReason,
answerKeyScore,
}),
factualAccuracyScore: answerKeyScore.factualAccuracyScore,
entityCoverageRatio: answerKeyScore.entityCoverageRatio,
domainAccuracyRatio: answerKeyScore.domainAccuracyRatio,
Expand Down Expand Up @@ -657,6 +668,7 @@ async function runSystemPrompt(input) {
validation,
answerKeyScore,
infraBlockerReason,
capabilityGateReason,
minRequiredCompleteness: input.config.minRequiredCompleteness,
validationIssues: normalized.validationIssues,
}),
Expand Down Expand Up @@ -758,6 +770,7 @@ function parseArgs(args) {
tinyFishAgentStepUsd: 0.015,
minRequiredCompleteness: 0.75,
minFactualAccuracy: defaultMinimumFactualAccuracy,
requirePlaywrightReady: false,
};

for (let index = 0; index < args.length; index += 1) {
Expand Down Expand Up @@ -797,6 +810,8 @@ function parseArgs(args) {
} else if (arg === "--min-factual-accuracy") {
config.minFactualAccuracy = nonNegativeNumber(value, config.minFactualAccuracy);
index += 1;
} else if (arg === "--require-playwright-ready") {
config.requirePlaywrightReady = true;
} else if (arg === "--help" || arg === "-h") {
printHelpAndExit();
} else {
Expand Down Expand Up @@ -972,6 +987,71 @@ export function normalizePayload(payload) {
};
}

export function playwrightReadinessGateReason({
diagnostics,
requirePlaywrightReady,
}) {
if (!requirePlaywrightReady) {
return null;
}
const readiness = diagnostics?.playwrightCandidateReadiness;
if (!readiness || typeof readiness !== "object") {
return "Playwright readiness gate failed: missing playwrightCandidateReadiness diagnostics.";
}
const reasons = stringArrayValue(readiness.reasons);
if (readiness.status !== "ready") {
return [
"Playwright readiness gate failed:",
reasons.length > 0
? reasons.join("; ")
: `status is ${String(readiness.status ?? "missing")}.`,
].join(" ");
}
if (numberValue(readiness.browserStepCount) <= 0) {
return "Playwright readiness gate failed: no actionable browser steps.";
}
if (numberValue(readiness.sourceUrlCount) <= 0) {
return "Playwright readiness gate failed: no source URLs to anchor replay.";
}
return null;
}

export function benchmarkStatusForOutcome({
execution,
parsedPayload,
answerKeyScore,
infraBlockerReason,
capabilityGateReason,
}) {
if (infraBlockerReason) {
return "blocked";
}
if (capabilityGateReason) {
return "failed";
}
return execution.exitCode === 0 && parsedPayload && answerKeyScore.passed
? "ok"
: "failed";
}

export function failureCategoryForOutcome({
status,
infraBlockerReason,
capabilityGateReason,
answerKeyScore,
}) {
if (status === "ok") {
return undefined;
}
if (infraBlockerReason) {
return "infra";
}
if (capabilityGateReason) {
return "capability_gate";
}
return answerKeyScore.failureCategory;
}

function normalizeUsage(value) {
return {
promptTokens: numberValue(value?.promptTokens ?? value?.inputTokens ?? value?.prompt_tokens),
Expand Down Expand Up @@ -1041,7 +1121,7 @@ function evaluateRows({ rows, promptDefinition }) {
};
}

async function rescoreBenchmarkRun({ runDirectory, prompts, config }) {
export async function rescoreBenchmarkRun({ runDirectory, prompts, config }) {
const previousSummary = JSON.parse(await readFile(join(runDirectory, "summary.json"), "utf8"));
const promptsById = new Map(prompts.map((promptDefinition) => [
promptDefinition.id,
Expand Down Expand Up @@ -1089,11 +1169,19 @@ async function rescoreBenchmarkRun({ runDirectory, prompts, config }) {
parsedPayload: usablePayload,
normalized,
});
const status = infraBlockerReason
? "blocked"
: execution.exitCode === 0 && usablePayload && answerKeyScore.passed
? "ok"
: "failed";
const capabilityGateReason = infraBlockerReason
? null
: playwrightReadinessGateReason({
diagnostics: normalized.diagnostics,
requirePlaywrightReady: config.requirePlaywrightReady,
});
const status = benchmarkStatusForOutcome({
execution,
parsedPayload: usablePayload,
answerKeyScore,
infraBlockerReason,
capabilityGateReason,
});

rescoredLaneResults.push({
...laneResult,
Expand All @@ -1103,9 +1191,12 @@ async function rescoreBenchmarkRun({ runDirectory, prompts, config }) {
expectedStress: promptDefinition.expectedStress,
answerKey: answerKeyForPrompt(promptDefinition),
status,
failureCategory: status === "ok" ? undefined : (
infraBlockerReason ? "infra" : answerKeyScore.failureCategory
),
failureCategory: failureCategoryForOutcome({
status,
infraBlockerReason,
capabilityGateReason,
answerKeyScore,
}),
factualAccuracyScore: answerKeyScore.factualAccuracyScore,
entityCoverageRatio: answerKeyScore.entityCoverageRatio,
domainAccuracyRatio: answerKeyScore.domainAccuracyRatio,
Expand All @@ -1130,6 +1221,32 @@ async function rescoreBenchmarkRun({ runDirectory, prompts, config }) {
needsReviewCount: validation.needsReviewCount,
validationIssueCount: normalized.validationIssues.length,
validationIssues: normalized.validationIssues,
selfHealingAction: normalized.diagnostics.selfHealingAction,
selfHealingArtifactKinds: normalized.diagnostics.artifactKinds,
processTraceStepCount: normalized.diagnostics.processTrace?.stepCount,
processTraceBrowserStepCount:
normalized.diagnostics.processTrace?.browserStepCount,
playwrightCandidateStatus:
normalized.diagnostics.playwrightCandidateReadiness?.status,
playwrightCandidateBrowserStepCount:
normalized.diagnostics.playwrightCandidateReadiness?.browserStepCount,
playwrightCandidateSourceUrlCount:
normalized.diagnostics.playwrightCandidateReadiness?.sourceUrlCount,
diagnostics: normalized.diagnostics,
usage: normalized.usage,
searchCallCount: normalized.metrics.searchCallCount,
fetchCallCount: normalized.metrics.fetchCallCount,
browserCallCount: normalized.metrics.browserCallCount,
agentRunCount: normalized.metrics.agentRunCount,
agentStepCount: normalized.metrics.agentStepCount,
estimatedModelCostUsd: estimateModelCostUsd(normalized.usage, config),
estimatedTinyFishAgentCostUsd: roundUsd(
normalized.metrics.agentStepCount * config.tinyFishAgentStepUsd
),
estimatedTotalCostUsd: roundUsd(
estimateModelCostUsd(normalized.usage, config) +
normalized.metrics.agentStepCount * config.tinyFishAgentStepUsd
),
errorMessage: status === "ok"
? undefined
: failureReason({
Expand All @@ -1138,6 +1255,7 @@ async function rescoreBenchmarkRun({ runDirectory, prompts, config }) {
validation,
answerKeyScore,
infraBlockerReason,
capabilityGateReason,
minRequiredCompleteness: config.minRequiredCompleteness,
validationIssues: normalized.validationIssues,
}),
Expand Down Expand Up @@ -1652,13 +1770,15 @@ export function failureReason({
validation,
answerKeyScore,
infraBlockerReason,
capabilityGateReason,
minRequiredCompleteness,
validationIssues = [],
}) {
if (infraBlockerReason) return infraBlockerReason;
if (execution.timedOut) return "Command timed out.";
if (execution.exitCode !== 0) return `Command exited ${execution.exitCode}.`;
if (!parsedPayload) return "No parseable JSON object found in stdout.";
if (capabilityGateReason) return capabilityGateReason;
const capabilityDiagnostic = capabilityDiagnosticReason(validationIssues);
if (capabilityDiagnostic) return capabilityDiagnostic;
if (answerKeyScore?.failureCategory === "clarification") {
Expand Down Expand Up @@ -1807,6 +1927,12 @@ node benchmarks/dataset-agent/run-benchmark.mjs \\
Rescore existing artifacts without spending credits:
node benchmarks/dataset-agent/run-benchmark.mjs --rescore-dir benchmark-results/<run>

Require self-healing Playwright readiness for browser-action canaries:
node benchmarks/dataset-agent/run-benchmark.mjs \\
--require-playwright-ready \\
--prompt-ids mcp-docs-pages \\
--system collection-self-heal='node --import ./backend/node_modules/tsx/dist/esm/index.mjs benchmarks/dataset-agent/adapters/collection-self-healing-adapter.mjs'

Agent command contract:
- stdout should contain a JSON object.
- Preferred shape: { "rows": [], "validationIssues": [], "usage": {}, "metrics": {} }
Expand Down
Loading