Epoch persona conflicts with hardcoded fixture data

[roykollensvendsen]

Problem

When eval_pack.py generates an epoch persona (e.g., "Riley Johnson, Design Lead @ Cascade Systems"), the USER.md template placeholders ({{USER_NAME}}, {{USER_ROLE}}, {{COMPANY}}) are substituted correctly. However, all fixture data remains hardcoded to the default scenario context:

• Emails come from @techcorp.com addresses (marcus.johnson@techcorp.com, sarah.kim@techcorp.com)
• Slack messages reference TechCorp employees and TechCorp-specific context
• Calendar events list TechCorp attendees
• Task board references TechCorp projects

This creates an inconsistency where the agent is "Riley Johnson @ Cascade Systems" but receives data from a completely different company's systems.

Impact

• May confuse reasoning models into spending extra tokens resolving the inconsistency
• Could cause timeouts on complex scenarios (observed inbox_to_action and team_standup timing out at 120s with epoch personas)
• Undermines the purpose of persona variation if the agent can pattern-match on hardcoded fixture names regardless of persona

Reproduction

OPENCLAW_URL=http://localhost:18790 \
  python scripts/eval_pack.py --pack pack.json \
  --clawbench-path /path/to/clawbench \
  --seed 32389 -v

Observe: persona is "Riley Johnson, Design Lead @ Cascade Systems" but all fixture data references TechCorp.

Suggestion

Either:

1. Template-substitute company/people names in fixture data (inbox.json, slack_messages.json, etc.) the same way USER.md is substituted
2. Document that persona variation only affects USER.md and the epoch preamble, not fixture data — so miners know not to rely on persona-specific names in their policies

🤖 Generated with Claude Code