Skip to content

[AIGTWY-4882] Cover OS-managed input preservation in CUJ7 at Claude/Codex parity - #1060

Open
tt-le wants to merge 18 commits into
mainfrom
tt-le/aigtwy-4882-cuj7-tui-family-defaults
Open

tt-le wants to merge 18 commits into
mainfrom
tt-le/aigtwy-4882-cuj7-tui-family-defaults

Conversation

@tt-le

@tt-le tt-le commented Oct 8, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Stacked on #930. Adds the CUJ7 coverage #930's headless cases cannot reach, and keeps CUJ7 at Claude/Codex parity. Tracks AIGTWY-4882.

ug only rewrites an agent's OS-managed file when it has a TTY: managed_writes_allowed() is sys.stdin.isatty() (src/ucode/managed_files.py). Both _reconcile_managed_settings (Claude) and _reconcile_managed_config (Codex) return early unless ug's desired settings conflict with the file, and a missing owned path is not a conflict. UserSession.run starts children with stdin DEVNULL or a pipe, so every headless case covers only the no-write path. No test asserted that the interactive rewrite keeps pre-existing admin input.

CUJ7 now collects twelve cases, six per agent:

Journey Claude Codex
Configured discovery case 07 case 08
Fresh --model-location task ×2 ✓ (#930) ✓ (#930)
Headless: preserves OS-managed input ×2 test_ug_claude_preserves_preexisting_managed_family_defaults (#930) test_ug_codex_preserves_preexisting_managed_config (new)
TUI: ug rewrites the file, keeps input, revert restores test_ug_claude_tui_rewrites_managed_settings_preserving_family_defaults (new) test_ug_codex_tui_rewrites_managed_config_preserving_admin_settings (new)

Shared fixture and assertions. #930's family_defaults fixture is now an agent-keyed managed_input fixture. It seeds /etc/claude-code/managed-settings.json with distinct family defaults, or /etc/codex/managed_config.toml with inert admin settings outside ug's owned keys: a sibling AdminProvider model provider and file_opener. Teardown is unchanged: recording stops, then the guarded revert, file removal and parent-directory cleanup (only if the fixture created the directory) run. Every preservation case requires:

  • each seeded leaf unchanged, via a recursive _assert_contains;
  • task-correlated, nonempty HTTP 200 inference through the recorder, via assert_inference_evidence (Codex also with databricks-model-service-parent-schema = ug_e2e.models, plus a completed turn reporting the scoped Luna service; Claude headless also checks modelUsage);
  • for the TUI cases, ug's gateway base URL in the rewritten file (env.ANTHROPIC_BASE_URL / model_providers.Databricks.base_url = <recorder>/ai-gateway/{anthropic,codex/v1}). This proves the interactive write happened.
  • for the TUI cases, ug revert must restore the parsed seeded document. Revert may reserialize the file, so the comparison is on parsed JSON/TOML, not bytes.

Both READMEs now describe CUJ7 as eight journeys and twelve cases, and the scenario table has rows for the three new cases.

Not asserted, intentionally. Codex strips a pre-existing model / model_reasoning_effort from the managed file when no model is chosen. That is the opposite of Claude's family-default preservation, and the CUJ seed avoids those keys. See the comment below; it's a product question, not something to lock in here.

Validation

  • All e2e_cuj cases collect (90), including 12 CUJ7 cases, with --confcutdir=tests/e2e_cuj and pexpect/pyte, matching CI.
  • Offline sanity check of the new helpers: each seed parses, _assert_contains accepts ug's additions and rejects a changed or missing seeded leaf, and _assert_rewritten rejects a file ug did not write.
  • ruff check and ruff format --check pass on tests.
  • Unit suite: 3,518 passed. The two failures are tests/test_e2e_user_agent.py::TestClaudeUserAgent::{test_user_agent_arrives_at_gateway,test_managed_http_header_arrives_at_gateway}, which also fail on origin/main on the authoring host: it has its own /etc/claude-code/managed-settings.json. They pass in CI.
  • Live CUJ7 has not run for the new cases. The host's managed settings make a local live run unrepresentative, and ci.yml only triggers on PRs targeting main, so this stacked PR gets no automatic CUJ run. It will run once [AIGTWY-4882] Add fresh-launch journeys and dedicated CUJ7 coverage #930 merges and this PR retargets main, or via a manual integration.yml dispatch on this branch.

This pull request and its description were written by Isaac.

@tt-le tt-le changed the title [AIGTWY-4882] Cover interactive managed-settings rewrite in CUJ7 [AIGTWY-4882] Cover OS-managed input preservation in CUJ7 at Claude/Codex parity Oct 8, 2026
@tt-le

tt-le commented Oct 8, 2026

Copy link
Copy Markdown
Collaborator Author

Product question (not addressed in this PR): Codex strips an admin's managed model, while Claude preserves family defaults.

On an unmanaged launch with no model chosen, Codex's compose (src/ucode/agents/codex.py, write_tool_config) runs base.pop("model") and base.pop("model_reasoning_effort"). _reconcile_managed_config applies the same compose to /etc/codex/managed_config.toml, so a pre-existing admin model there is removed whenever ug rewrites the file (any TTY launch). This is intended today: tests/test_agent_codex.py::test_managed_config_preserves_other_keys seeds model = "my-own" and asserts it's gone ("ucode removes its stale model pin"), introduced in #429.

Two things look inconsistent:

  1. Claude does the opposite. With no CodingAgentConfig, should_preserve_preexisting_claude_family_defaults treats an existing family default in the OS-managed file as belonging to the developer and keeps it.
  2. The "stale pin" framing doesn't match the test. The test seeds a value ug never wrote. Also, model_reasoning_effort isn't in Codex's MANAGED_KEYS, so ug removes a key it doesn't claim to own.

This PR's Codex cases deliberately seed only keys outside ug's owned paths, so they neither lock in nor contradict the current behavior. If the intended behavior is to preserve admin-set model / model_reasoning_effort when ug never wrote them (parity with Claude), that's a product change plus a unit-test update. A CUJ7 assertion can follow once it's decided.

@tt-le
tt-le force-pushed the tt-le/aigtwy-4882-integ-tests branch 2 times, most recently from 11fbb2b to 3a1440d Compare October 9, 2026 18:49
@tt-le
tt-le force-pushed the tt-le/aigtwy-4882-cuj7-tui-family-defaults branch from 58a6c8b to d0c996e Compare October 9, 2026 18:53
tt-le and others added 16 commits October 9, 2026 20:03
read_jsonl used Path.read_text() with the platform default encoding.
On the Windows integration lane that is cp1252, which cannot decode
UTF-8 model text such as a closing curly quote (0xE2 0x80 0x9D), so
every Codex headless case that reads its session transcript through
assert_completed_task_model failed with UnicodeDecodeError.

Read transcripts as UTF-8 and add a reader regression test with
non-ASCII content; it fails under a non-UTF-8 default without the fix.

Co-authored-by: Isaac <no-reply@databricks.com>
`codex exec` cannot prompt for a Windows sandbox on first run and rejects
every shell command until one is configured, so on the Windows lane the
fresh-workspace task could not read its file ("file-system access is
blocked by the current sandbox/policy"). Call
choose_codex_windows_sandbox() as the other Codex headless journeys do;
it is a no-op elsewhere.

Co-authored-by: Isaac <no-reply@databricks.com>
Recount from collection on top of #1051/#1062/#1039: 85 live cases
(39 Claude, 46 Codex), 11 marked TUI, OpenCode adding one live headless
and two managed_fixture cases, and 129 executions (126 with Claude and
Codex selected).

Co-authored-by: Isaac <no-reply@databricks.com>
@tt-le
tt-le force-pushed the tt-le/aigtwy-4882-integ-tests branch from d844ef6 to 1fbc5d7 Compare October 9, 2026 20:04
tt-le and others added 2 commits October 9, 2026 20:04
Headless launches have no TTY, so ug leaves Claude's OS-managed settings
untouched unless they conflict; only an interactive launch rewrites that
file. Add a CUJ7 TUI case that seeds OS-managed family defaults, selects
Sonnet in the real Claude TUI, and asserts that ug rewrote the file (its
ANTHROPIC_BASE_URL is present) while keeping every seeded default, that
the TUI task reached the Sonnet service through the recorder, and that
`ug revert` restores the seeded file.

Co-authored-by: Isaac <no-reply@databricks.com>
Keep CUJ7 at Claude/Codex parity: add Codex's headless preservation
case (before ug's separator and in exec) and its interactive rewrite
case, each over a seeded /etc/codex/managed_config.toml holding inert
admin settings outside ug's owned keys (a sibling AdminProvider model
provider and file_opener).

Both agents now share one seeded-file fixture and the same assertions:
every seeded leaf unchanged, task-correlated inference through the
recorder (Codex also with the parent-schema header), and for the
interactive cases ug's gateway base URL in the rewritten file plus
`ug revert` restoring the seeded document. CUJ7 now collects twelve
cases, six per agent.

Co-authored-by: Isaac <no-reply@databricks.com>
@tt-le
tt-le force-pushed the tt-le/aigtwy-4882-cuj7-tui-family-defaults branch from d0c996e to 53000c6 Compare October 9, 2026 20:06
Base automatically changed from tt-le/aigtwy-4882-integ-tests to main October 10, 2026 02:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant