Copied from codex's report from its slow-powers:investigating-bugs iteration 2 run:
## eval-magic 0.9.0 friction
The following issues or rough edges occurred during the run:
1. A Codex smoke dispatch failed inside the operator's outer workspace sandbox with
`Operation not permitted`; the exact generated task succeeded outside that outer
sandbox while retaining the task's workspace-write sandbox and eval guard.
2. Ingesting the one-task smoke iteration ran held-out command setup/checks across all
80 prepared environments, including undispatched tasks. That made the remaining
smoke workspace unsafe to resume, so the fleet used a clean iteration 2.
3. Generated post-run commands omit the explicitly supplied `--harness-file`, even
though the preregistration requires the UTC descriptor on every command. Ingest,
finalize, and teardown retained it manually.
4. The `conditions.json` file records models, harness, label, and `TZ`, but it has no
guard-status field even though the run was generated with `--guard`. Guard state
had to be confirmed from the armed hook and its denial artifacts.
5. The guard blocks routed Git commands, package-install commands, and output
redirection outside the task workspace, but an explicit `touch` command targeting
`/private/tmp` was not blocked or logged. The documented redirection probe was
blocked, and post-run stray-write detection found no fleet violation. The two
zero-byte probe files were removed after inspection.
6. Frozen transcript checks use Claude-style `Bash|Read` tool names. Codex transcripts
use `command_execution`, producing 0/10 mechanical results in both arms despite
direct evidence of the behavior in every run.
7. Codex has usable tool-invocation transcripts, but the skill-invocation meta-check
falls back to final-message fingerprint judging. This produced 15 false negatives
and a raw 62.5% invocation rate despite 40/40 direct staged-skill reads.
8. Three judge processes logged a non-fatal model-catalog refresh timeout. Each still
exited successfully and wrote a valid verdict.
9. Codex duration data is unavailable for all 80 runs, so eval-magic reports timing
sample size zero.
Copied from codex's report from its slow-powers:investigating-bugs iteration 2 run: