Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -287,6 +287,19 @@ This runs the photo-studio multi-turn suite and exports a customer-ready report.

## Turn A Real Failure Into A Regression Test

When a pilot replies with one sanitized transcript, start with the intake wrapper. It writes a private suite draft, merchant draft, and triage summary without quoting the raw transcript in the summary:

```bash
pbpaste | npx voice-agent-testops transcript-intake \
--stdin \
--suite .voice-testops/transcript-intake/suite.json \
--merchant-out .voice-testops/transcript-intake/merchant.json \
--summary .voice-testops/transcript-intake/summary.md \
--merchant-name "Pilot demo agent"
```

Use this before a live endpoint is ready. The summary highlights inferred industry, turn counts, assertion mix, risk signals, privacy warnings, generated artifacts, and the next `validate` / `doctor` / `run` commands. It does not print raw transcript text.

Paste a failed call and generate a starter suite plus an editable merchant draft:

```bash
Expand Down
13 changes: 13 additions & 0 deletions README.zh-CN.md
Original file line number Diff line number Diff line change
Expand Up @@ -217,6 +217,19 @@ npm run voice-test -- \

## 把真实失败对话变成回归测试

如果试点对象先回复了一条脱敏 transcript,还没有 endpoint,先跑 intake 包装命令。它会写出私有 suite 草稿、商家草稿和 triage 摘要;summary 不会引用原始 transcript 文本:

```bash
pbpaste | npx voice-agent-testops transcript-intake \
--stdin \
--suite .voice-testops/transcript-intake/suite.json \
--merchant-out .voice-testops/transcript-intake/merchant.json \
--summary .voice-testops/transcript-intake/summary.md \
--merchant-name "Pilot demo agent"
```

这个 summary 会列出推断行业、turn 数、断言分布、风险信号、隐私警告、生成产物和下一步 `validate` / `doctor` / `run` 命令,但不输出原始 transcript。

如果你已经遇到过一次真实失败,可以直接复制 transcript,生成一个可编辑的 suite 和商家资料草稿:

```bash
Expand Down
9 changes: 9 additions & 0 deletions docs/growth/2026-05-07-outreach-followup.md
Original file line number Diff line number Diff line change
Expand Up @@ -66,3 +66,12 @@ Issue:[kev-hu/vapi-voice-agent#1](https://github.com/kev-hu/vapi-voice-agent/i
为降低对方提供样本的成本,已补充可直接复制填写的 intake 包:[Insurance transcript intake pack](../ops/insurance-transcript-intake.md)。下一次 follow-up 优先贴这个模板,而不是继续泛泛请求“提供 transcript”。

已将 intake 包回贴到 `kev-hu/vapi-voice-agent#1`:[issuecomment-4403442303](https://github.com/kev-hu/vapi-voice-agent/issues/1#issuecomment-4403442303)。对方现在只需按模板粘贴一条脱敏 transcript,或明确标注 synthetic/public sample。

## 2026-05-09 reply check

检查时间:2026-05-09 13:58 CST。

- `streamcoreai/streamcore-server#4`:对方明确同意测试,提示可先测 `streamcore.ai` demo,但 demo 只有基础 Streamcore knowledge;已回复请求可脚本化 HTTP/WebSocket 测试入口或脱敏 transcript:[issuecomment-4411642368](https://github.com/streamcoreai/streamcore-server/issues/4#issuecomment-4411642368)。
- `codewithmuh/ai-voice-agent#2`:对方给出弱正向回复;已回复请求 dev/test endpoint 或一条脱敏 booking / missed-call / handoff transcript:[issuecomment-4411642357](https://github.com/codewithmuh/ai-voice-agent/issues/2#issuecomment-4411642357)。

当前优先级:Streamcore > codewithmuh。拿到 endpoint 走 `doctor` / `run`;拿到 transcript 先跑 `transcript-intake`,只公开 aggregate 结果,不贴原始 transcript。
10 changes: 9 additions & 1 deletion docs/ops/external-pilot-runbook.zh-CN.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,15 @@
- 测试数据已脱敏,不包含真实身份证、完整地址、病历、交易账号、客户真实姓名等敏感信息。
- 至少选择一个 starter 行业:`real_estate`、`dental_clinic`、`home_design`、`insurance`、`restaurant`。

如果对方只能提供保险 / 监管服务 transcript,先让对方按 [Insurance transcript intake pack](insurance-transcript-intake.md) 填写一条脱敏失败或边界通话,再进入 `from-transcript` / `draft-regressions` 流程。
如果对方只能先提供一条脱敏 transcript,优先用 `transcript-intake` 生成私有 suite 草稿、商家草稿和 triage summary:

```bash
pbpaste | npx voice-agent-testops transcript-intake \
--stdin \
--summary .voice-testops/transcript-intake/summary.md
```

summary 只输出统计、风险信号、隐私警告和下一步命令,不引用原始 transcript 文本。保险 / 监管服务 transcript 先让对方按 [Insurance transcript intake pack](insurance-transcript-intake.md) 填写一条脱敏失败或边界通话,再加 `--intake insurance` 进入 `from-transcript` / `draft-regressions` 流程。

如果对方提供的是一批原始录音链接或 call replay URL,先按 [录音资源 Intake Runbook](recording-resource-intake.zh-CN.md) 建私有 manifest,完成授权、脱敏和 `keep` / `maybe` / `discard` 筛选后,再挑样本转 transcript。

Expand Down
3 changes: 2 additions & 1 deletion docs/ops/external-pilot-tracker.zh-CN.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,7 @@

| 日期 | 对象 | 入口 | 行业 starter | 接入方式 | 数据授权 | 当前状态 | 下一步动作 |
|---|---|---|---|---|---|---|---|
| 2026-05-06 | Streamcore server | https://github.com/streamcoreai/streamcore-server/issues/4 | custom platform / realtime voice agent | HTTP / WebSocket / Transcript import | 未确认 | replied | 已回复需求;等待可脚本化 demo endpoint、WebSocket route 或 sanitized transcript |
| 2026-05-08 | Awaisali36 outbound real-estate Vapi agent | https://github.com/Awaisali36/Outbound-Real-State-Voice-AI-Agent-/issues/6 | real_estate / outbound_leadgen | Vapi / Transcript import | 未确认 | contacted | 2026-05-10 follow-up:endpoint 或 1 条脱敏 transcript |
| 2026-05-08 | santmun Sofia voice agent | https://github.com/santmun/sofia-voice-agent/issues/2 | real_estate | Retell / Twilio / Transcript import | 未确认 | contacted | 2026-05-10 follow-up:endpoint 或 1 条脱敏 transcript |
| 2026-05-08 | askjohngeorge Pipecat lead qualifier | https://github.com/askjohngeorge/pipecat-lead-qualifier/issues/1 | outbound_leadgen | Pipecat / HTTP | 未确认 | contacted | 2026-05-10 follow-up:lead qualifier endpoint 或 transcript |
Expand All @@ -49,7 +50,7 @@
| 2026-05-09 | videosdk WhatsApp AI calling agent | https://github.com/videosdk-community/videosdk-whatsapp-ai-calling-agent/issues/2 | outbound_leadgen / custom channel | WhatsApp / Twilio / VideoSDK / Transcript import | 未确认 | contacted | 2026-05-11 follow-up:0.1.19 comment 已补;等 endpoint 或 sanitized transcript |
| 2026-05-09 | VoiceBlender | https://github.com/VoiceBlender/voiceblender/issues/28 | outbound_leadgen / platform adapter | REST / Webhook / WebSocket adapter | 未确认 | contacted | 2026-05-11 follow-up:0.1.19 comment 已补;问 adapter interest 或 demo endpoint |
| 2026-05-09 | theaifutureguy LiveKit voice agent | https://github.com/theaifutureguy/livekit-voice-agent/issues/6 | outbound_leadgen / receptionist | LiveKit / Telephony / HTTP | 未确认 | contacted | 2026-05-11 follow-up:0.1.19 comment 已补;等 dev endpoint 或 one sanitized call |
| 2026-05-09 | codewithmuh AI voice receptionist | https://github.com/codewithmuh/ai-voice-agent/issues/2 | restaurant / custom receptionist | Vapi / HTTP / Transcript import | 未确认 | contacted | 2026-05-11 follow-up:booking endpoint 或 one sanitized transcript |
| 2026-05-09 | codewithmuh AI voice receptionist | https://github.com/codewithmuh/ai-voice-agent/issues/2 | restaurant / custom receptionist | Vapi / HTTP / Transcript import | 未确认 | replied | 已回复需求;等待 booking endpoint 或 one sanitized transcript |
| 2026-05-09 | Teleglobals voicebot calling agent | https://github.com/Teleglobals-org/voicebot-calling-agent/issues/1 | real_estate | Twilio / AWS / HTTP / Transcript import | 未确认 | contacted | 2026-05-11 follow-up:real-estate endpoint 或 one sanitized transcript |
| 2026-05-09 | frejun Teler Vapi bridge | https://github.com/frejun-tech/teler-vapi-bridge/issues/6 | outbound_leadgen / platform bridge | Vapi bridge / HTTP / WebSocket | 未确认 | contacted | 2026-05-11 follow-up:contract-test adapter interest 或 dev endpoint |

Expand Down
1 change: 1 addition & 0 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,7 @@
"voice-test": "tsx src/testops/cli.ts",
"judge:calibrate": "tsx src/testops/cli.ts calibrate-judge",
"suite:from-transcript": "tsx src/testops/cli.ts from-transcript",
"transcript:intake": "tsx src/testops/cli.ts transcript-intake",
"calls:import": "tsx src/testops/cli.ts import-calls",
"voice-test:openclaw": "scripts/openclaw-docker.sh voice-test",
"voice-test:photo-demo": "scripts/openclaw-docker.sh voice-test examples/voice-testops/photo-studio-multiturn-suite.json",
Expand Down
167 changes: 166 additions & 1 deletion src/testops/cli.ts
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,13 @@ import {
renderSemanticJudgeCalibrationMarkdown,
} from "./semanticJudgeCalibration";
import { loadVoiceTestSuite } from "./suiteLoader";
import { getTranscriptIntakeDefaults, parseTranscriptIntakePreset, type TranscriptIntakePreset } from "./transcriptIntake";
import {
analyzeTranscriptIntake,
getTranscriptIntakeDefaults,
parseTranscriptIntakePreset,
renderTranscriptIntakeMarkdown,
type TranscriptIntakePreset,
} from "./transcriptIntake";
import { buildDraftMerchantFromTranscript, buildVoiceTestSuiteFromTranscript } from "./transcriptSuite";

const severityRank: Record<VoiceTestSeverity, number> = {
Expand Down Expand Up @@ -84,6 +90,10 @@ async function main(argv: string[]): Promise<number> {
return recordingIntake(argv.slice(1));
}

if (argv[0] === "transcript-intake") {
return transcriptIntake(argv.slice(1));
}

if (argv[0] === "pilot-report") {
return generatePilotReport(argv.slice(1));
}
Expand Down Expand Up @@ -531,6 +541,23 @@ type RecordingIntakeArgs = {
summaryPath?: string;
};

type TranscriptIntakeArgs = {
transcriptPath?: string;
readFromStdin: boolean;
suitePath: string;
merchantPath?: string;
merchantOutPath?: string;
summaryPath: string;
merchantName?: string;
industry?: Industry;
name?: string;
scenarioId?: string;
scenarioTitle?: string;
source: LeadSource;
intake?: TranscriptIntakePreset;
turnRole: FromTranscriptTurnRole;
};

type PilotReportArgs = {
reportPath: string;
commercialPath?: string;
Expand Down Expand Up @@ -707,6 +734,144 @@ function parseRecordingIntakeArgs(argv: string[]): RecordingIntakeArgs {
};
}

async function transcriptIntake(argv: string[]): Promise<number> {
const args = parseTranscriptIntakeArgs(argv);
const transcript = args.readFromStdin
? await readFromStdin()
: await readFile(await resolveReadablePath(args.transcriptPath ?? ""), "utf8");
const intakeDefaults = args.intake ? getTranscriptIntakeDefaults(args.intake) : undefined;
const merchantName = args.merchantName ?? intakeDefaults?.merchantName;
const industry = args.industry ?? intakeDefaults?.industry;
const merchant = args.merchantPath
? merchantConfigSchema.parse(JSON.parse(await readFile(await resolveReadablePath(args.merchantPath), "utf8")))
: buildDraftMerchantFromTranscript({ transcript, name: merchantName, industry });
const suite = buildVoiceTestSuiteFromTranscript({
transcript,
merchant,
name: args.name ?? intakeDefaults?.suiteName,
scenarioId: args.scenarioId ?? intakeDefaults?.scenarioId,
scenarioTitle: args.scenarioTitle ?? intakeDefaults?.scenarioTitle,
source: args.source,
turnRole: args.turnRole,
});
const suiteOutput = args.merchantOutPath
? buildSuiteWithMerchantRef(suite, relativeMerchantRef(args.suitePath, args.merchantOutPath))
: suite;
const report = analyzeTranscriptIntake({
transcript,
suite,
sourcePath: args.readFromStdin ? undefined : args.transcriptPath,
selectedTurnRole: args.turnRole,
artifacts: {
suitePath: args.suitePath,
merchantPath: args.merchantOutPath,
summaryPath: args.summaryPath,
},
});

if (args.merchantOutPath) {
await writeReport(args.merchantOutPath, `${JSON.stringify(merchant, null, 2)}\n`);
}
await writeReport(args.suitePath, `${JSON.stringify(suiteOutput, null, 2)}\n`);
await writeReport(args.summaryPath, renderTranscriptIntakeMarkdown(report));

console.log(`Transcript intake summary: ${args.summaryPath}`);
console.log(`Generated suite draft: ${args.suitePath}`);
if (args.merchantOutPath) {
console.log(`${args.merchantPath ? "Merchant profile" : "Merchant draft"}: ${args.merchantOutPath}`);
}
console.log(`Transcript: ${args.readFromStdin ? "read from stdin" : args.transcriptPath}`);
if (args.intake) {
console.log(`Transcript intake: ${args.intake}`);
}
console.log(`Suite: ${suite.name}`);
console.log(`Scenario: ${suite.scenarios[0].id} - ${suite.scenarios[0].title}`);
printTurnCount(args.turnRole, suite.scenarios[0].turns.length);
console.log(`Assertions: ${report.assertionCount}`);
console.log(`Risk signals: ${report.riskSignals.length}`);
console.log(`Privacy warnings: ${report.privacyWarnings.length}`);

return 0;
}

function parseTranscriptIntakeArgs(argv: string[]): TranscriptIntakeArgs {
const values = new Map<string, string>();
const flags = new Set<string>();
const knownValues = new Set([
"transcript",
"input",
"suite",
"out",
"merchant",
"merchant-out",
"summary",
"merchant-name",
"industry",
"name",
"scenario-id",
"scenario-title",
"source",
"intake",
"turn-role",
]);

for (let index = 0; index < argv.length; index += 1) {
const arg = argv[index];
if (!arg.startsWith("--")) {
throw new Error(`Unexpected argument: ${arg}`);
}

const name = arg.slice(2);
if (name === "stdin") {
flags.add(name);
continue;
}
if (!knownValues.has(name)) {
throw new Error(`Unknown transcript-intake option: --${name}`);
}

const value = argv[index + 1];
if (!value || value.startsWith("--")) {
throw new Error(`${arg} requires a value`);
}

values.set(name, value);
index += 1;
}

const readFromStdin = flags.has("stdin");
const transcriptPath = values.get("transcript") ?? values.get("input");
if (readFromStdin && transcriptPath) {
throw new Error("--stdin cannot be combined with --transcript or --input");
}
if (!readFromStdin && !transcriptPath) {
throw new Error("--transcript, --input, or --stdin is required");
}

const suitePath = values.get("suite") ?? values.get("out") ?? ".voice-testops/transcript-intake/suite.json";
const merchantPath = values.get("merchant");
const merchantOutPath = values.get("merchant-out") ?? (merchantPath ? undefined : ".voice-testops/transcript-intake/merchant.json");
const summaryPath = values.get("summary") ?? ".voice-testops/transcript-intake/summary.md";
const intake = values.get("intake");

return {
transcriptPath,
readFromStdin,
suitePath,
merchantPath,
merchantOutPath,
summaryPath,
merchantName: values.get("merchant-name"),
industry: values.has("industry") ? industrySchema.parse(values.get("industry")) : undefined,
name: values.get("name"),
scenarioId: values.get("scenario-id"),
scenarioTitle: values.get("scenario-title"),
source: leadSourceSchema.parse(values.get("source") ?? "website"),
intake: intake ? parseTranscriptIntakePreset(intake) : undefined,
turnRole: parseTranscriptTurnRole(values.get("turn-role") ?? "customer"),
};
}

function parsePilotReportArgs(argv: string[]): PilotReportArgs {
const values = parseKeyValueArgs(argv);

Expand Down
Loading