Agent Mode Performance Test

Automated performance testing for Postman Agent Mode AI tools. Two testing approaches available:

Approach	Description	Use When
UX Testing	Tests through the Postman UI using Playwright + Electron	You need to test the full user experience, including UI interactions
API Testing	Direct API calls to the agent endpoint	You need faster feedback, lower overhead, or want to validate tool call parameters

Project Structure

agent-mode-perf-test/
├── perf-test-ux/           # UI-based testing (Playwright + Electron)
│   ├── src/
│   │   ├── config.ts       # Test configuration
│   │   ├── test.spec.ts    # Main Playwright test
│   │   └── ...
│   ├── playwright.config.ts
│   └── package.json
│
├── perf-test-api/          # API-based testing (direct HTTP calls)
│   ├── src/
│   │   ├── config.ts       # Configuration from .env
│   │   ├── client.ts       # HTTP client with SSE streaming
│   │   ├── validator.ts    # JSON Schema validation
│   │   └── index.ts        # Main test runner
│   ├── .env.example        # Environment template
│   └── package.json
│
├── results/                # Shared results directory
└── README.md               # This file

---

Quick Start

UX Testing (through Postman UI)

cd perf-test-ux
npm install
npx playwright install
npm test

Requirements:

• Postman app installed and logged in
• Auto-approve tools enabled (recommended)

See perf-test-ux/README.md for full documentation.

API Testing (direct API calls)

cd perf-test-api
npm install

# Copy and configure environment
cp .env.example .env
# Edit .env with your GATEWAY_URL and ACCESS_TOKEN

npm start

Requirements:

• Access token from Postman (get from CurrentUser.accessToken)
• Gateway URL (production: https://gateway.postman.com, beta: https://gateway.postman-beta.com)

See perf-test-api/README.md for full documentation.

---

Comparison of Approaches

Feature	UX Testing	API Testing
Speed	Slower (launches app each run)	Faster (direct HTTP)
Fidelity	Full UI experience	API-only
Tool Execution	Executes tools in real app	Validates params only
Setup	Needs Postman app	Needs access token
Metrics	Time to completion, UI timing	Time to first token, tool call timing
Validation	Success indicators in chat	JSON Schema validation

When to Use Each

Use UX Testing when:

• Testing the complete user experience
• Verifying UI behavior and rendering
• Tools must actually execute (file editing, etc.)
• You need to test multi-step conversations with tool execution

Use API Testing when:

• You want faster iteration and feedback
• Testing tool call parameter generation
• Comparing model response times
• You don't need actual tool execution
• Running many iterations quickly

---

Metrics Captured

UX Testing

• Time to first response
• Time to tool call start
• Time to tool call end
• Time to completion (cancel button disappears)
• Streaming chunk statistics
• Success/failure based on chat text indicators

API Testing

• Time to first token (textChunk event)
• Time to tool call (toolCall event)
• Tool argument streaming time
• Time to completion ([DONE] signal)
• Tool call parameter validation (JSON Schema)
• Validation success rate

---

Results

Both approaches save results to timestamped folders:

results/
├── perf-test-2026-01-30T10-00-00/     # UX test results
│   ├── results.json
│   ├── report.txt
│   └── summary.md
│
└── api-test-2026-01-30T10-30-00/      # API test results
    ├── results.json
    ├── report.txt
    └── summary.md

---

Example: Testing File Editing

UX Approach

Tests actual file editing through the UI:

// perf-test-ux/src/config.ts
const config = {
  models: ['GPT-4o'],
  fileReset: {
    targetFilePath: '/project/server.js',
    resetBehavior: 'restore',
  },
  prompts: ['Remove the /health endpoint from server.js'],
  successIndicator: 'Edited server.js',
};

API Approach

Tests that the model generates valid editFile parameters:

// perf-test-api/src/config.ts
const config = {
  models: ['GPT-4o'],
  prompts: ['Remove the /health endpoint from server.js'],
  // Validates: editFile({ filePath: "...", lnDiff: "..." })
};

---

Development

Both projects are independent npm packages. Develop them separately:

# UX testing development
cd perf-test-ux
npm install
npm test

# API testing development
cd perf-test-api
npm install
npm start