Skip to content

Fix config file allowing partial overrides for judge/synthetic_user#61

Merged
richardkiene merged 1 commit into
mainfrom
fix/partial-llm-config-overrides
Jan 24, 2026
Merged

Fix config file allowing partial overrides for judge/synthetic_user#61
richardkiene merged 1 commit into
mainfrom
fix/partial-llm-config-overrides

Conversation

@richardkiene
Copy link
Copy Markdown
Contributor

Problem

The judge: and synthetic_user: sections in mcprobe.yaml required provider and model fields, even when the user just wanted to add extra_instructions and inherit everything else from the shared llm: config.

Error:

judge.provider
  Field required [type=missing, input_value={'extra_instructions': '...'}, input_type=dict]

Solution

Added FileLLMConfigOverride model with all optional fields, similar to ScenarioLLMOverride. This allows partial overrides in the config file.

Example

Now this works:

llm:
  provider: ollama
  model: llama3.2

judge:
  extra_instructions: |
    Custom judge instructions that inherit provider/model from llm section

Test plan

  • All 258 unit tests pass
  • Ruff linting passes
  • Mypy type checking passes

The judge: and synthetic_user: sections in mcprobe.yaml now allow
partial overrides. Previously they required provider and model fields
even when the user just wanted to set extra_instructions.

Changes:
- Add FileLLMConfigOverride model with all optional fields
- Use FileLLMConfigOverride for judge/synthetic_user in FileConfig
- Add _apply_file_override method for partial config application
- Export FileLLMConfigOverride from config package
- Update tests to use new model

This allows configs like:
```yaml
llm:
  provider: ollama
  model: llama3.2

judge:
  extra_instructions: |
    Custom judge instructions...
```
@richardkiene richardkiene merged commit b1e4e31 into main Jan 24, 2026
@richardkiene richardkiene deleted the fix/partial-llm-config-overrides branch January 24, 2026 00:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant