Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 24 additions & 8 deletions docs/configuration/reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -290,38 +290,52 @@ The `extra_instructions` field allows you to append custom instructions to the s
- Adding domain-specific evaluation criteria to the judge
- Customizing synthetic user behavior for specific test scenarios
- Fine-tuning evaluation strictness
- Setting project-wide guidelines that apply to all test runs

**Important:** Extra instructions are **appended** (not replaced) across all configuration levels. Instructions from multiple sources are concatenated with newlines.

**Configuration priority for extra_instructions:**
1. CLI arguments (highest)
2. Per-scenario config (in scenario YAML)
3. Component-specific config (`judge:` or `synthetic_user:` sections)
4. Shared config (`llm:` section)
4. Shared config (`llm:` section) - applies to both judge and synthetic_user

**Example - Strict judge instructions:**
**Example - Project-wide instructions (shared config):**
```yaml
# mcprobe.yaml
judge:
llm:
provider: ollama
model: llama3.2
extra_instructions: |
Be strict about tool parameter validation.
Any missing or incorrect parameters should result in failure.
Do not accept approximate answers - require exact matches.
This project tests a financial data API.
All monetary values should be validated for correct currency formatting.
Be strict about decimal precision in financial calculations.
```

**Example - Custom synthetic user behavior:**
This applies to both the judge and synthetic user across all scenarios.

**Example - Component-specific instructions:**
```yaml
# mcprobe.yaml
synthetic_user:
llm:
provider: ollama
model: llama3.2
extra_instructions: |
Project-wide: All responses should be evaluated for accuracy.

judge:
extra_instructions: |
Be strict about tool parameter validation.
Any missing or incorrect parameters should result in failure.

synthetic_user:
extra_instructions: |
Always express urgency in your requests.
If the agent asks more than 2 clarifying questions, show impatience.
```

In this example, the judge receives both the project-wide and judge-specific instructions concatenated together.

**Example - Per-scenario override (in scenario YAML):**
```yaml
# scenario.yaml
Expand All @@ -333,6 +347,8 @@ config:
This is a strict validation scenario.
```

This adds to any instructions already set in mcprobe.yaml.

### Supported Providers

#### Ollama
Expand Down