fix(provider-copilot): drop the obsolete Vertex structured-output strips - #271
Conversation
|
Agent note (Claude Code, Opus 5): the earlier claim in this PR that a live re-probe was impossible was wrong, and the reason is instructive — I probed with direct egress after excluding the upstream's HKG-scoped What upstream actually does today (2026-07-27, 42 samples + a 28-sample confirmation round)
Deterministic, not per-request flapping: every model was 6/6 consistent in the first deep run and 2/2 again in the confirmation round.
What this means for the gate in this PRThe Two hypotheses are now dead:
A 5-day sampler is now running every ~28 minutes across all catalog Claude models to establish whether the 4.7 carve-out is stable or a transient state of Copilot's fleet. I would rather not encode a one-model exception off a single day's data. Holding this PR as-is: the conservative gate is correct and shippable today, and the sampler decides whether to widen it. Marking it draft until that data lands. |
The Messages boundary deleted `output_config.format` and tool `strict: true` from every request. Both existed because a Vertex-routed request tripped the GCP org policy `constraints/vertexai.allowedPartnerModelFeatures`, which denied the `structured_outputs` partner feature with a 400 FAILED_PRECONDITION. Copilot picks the backend per request, so callers lost the grammar-constrained guarantee on every model to insure against a backend they might not even draw. That policy no longer denies the feature. A sampler that has run every two hours since Jul 27 against the live catalog, from two egresses, recorded the change as a single clean transition: the last org-policy rejection landed at 2026-07-31T14:37Z and every Vertex-served request since 16:38Z has returned 200, on both egresses, with the rejection counters flat across the rounds that followed. The message-id infix (`msg_vrtx_*` / `msg_bdrk_*` / bare `msg_01*`) attributes each sample to its serving deployment, so these are Vertex-served successes rather than requests that happened to route elsewhere. With the upstream accepting the fields, stripping them is pure loss: clients ask for a schema-constrained reply and silently receive an unconstrained one. Remove both interceptors rather than gate them — a gate would still need a model-id predicate, and the failure never tracked the model. The policy lives in GitHub's own GCP project and can be flipped back without notice, so the sampler keeps running against the same matrix. Reinstating the strips is a revert of this commit.
b0eb7cb to
d1beeba
Compare
Menci
changed the title
1 participant
Deletes
withStructuredOutputFormatStrippedandwithToolStrictStrippedfrom the Copilot Messages boundary, along with their tests and their two registrations in the boundary chain.What the strips did
Both interceptors deleted a field from every outbound Messages request:
output_config.format, which carries the JSON Schema a reply is constrained to, andstrict: trueon tools, which compilesinput_schemainto a grammar. Anthropic serves both on the GA Messages wire with no beta header.They existed because Copilot load-balances its Claude catalog across three Anthropic deployments — AWS Bedrock, Google Vertex, and Anthropic's own API — picking one per request. A Vertex-routed request tripped the GCP organization policy
constraints/vertexai.allowedPartnerModelFeatures, which denies thestructured_outputspartner feature by default and answers a 400FAILED_PRECONDITION. Because the backend is chosen per request and is not visible beforehand, the only deterministic defence was to strip the fields from everything — so every caller on every model lost the grammar-constrained guarantee to insure against a backend it might not even draw.Why the policy no longer requires it
A sampler ran every two hours from 2026-07-27 to 2026-08-01 against the live catalog, from two independent network egresses, sending
output_config.formatand astrict: truetool to every Claude model Copilot serves on/v1/messages. Each sample is attributed to its serving deployment by the Anthropic message-id infix —msg_vrtx_*for Vertex,msg_bdrk_*for Bedrock, baremsg_01*for Anthropic-direct — which is the only backend signal available, since Copilot strips Anthropic'srequest-idresponse header.The policy change shows up as a single clean transition:
32 Vertex-served successes across four consecutive rounds on both egresses, with the rejection counters flat throughout. These are genuinely Vertex-served, not requests that happened to route elsewhere.
Why removal rather than a model-id gate
The failure never tracked the model.
claude-opus-4.6drew Vertex and Bedrock inside the same window and failed only on the Vertex samples; both egresses converged on the same per-model backend mix, so the client's source address is not the variable either. Any gate would have to predict a per-request routing decision that no client can observe before the response. With the upstream accepting the fields, there is nothing left to gate, and keeping a stripped path would only preserve the silent downgrade.Risk and rollback
The organization policy lives in GitHub's own GCP project (
projects/524636045653), so it can be flipped back without notice. The sampler continues to run the same matrix; a regression surfaces as the return ofconstraints/vertexai.allowedPartnerModelFeaturesrejections. Reinstating the strips is a revert of this commit.One unexplained failure mode is unrelated to this change and predates it:
output_config.format: Extra inputs are not permittedonclaude-opus-4.7(12 samples) andclaude-opus-4.8(3 samples), last seen 2026-07-28T03:11Z, served by Anthropic-direct rather than Vertex. It has not recurred in over four days and its cause is unknown.Test Plan
pnpm run typecheck— cleanpnpm run lint— cleanpnpm run test— 4947 passed / 421 filesoutput_config.formatand toolstrict— 32 Vertex-attributed 200s across four rounds and two egresses since 2026-07-31T16:38Z, zero organization-policy rejections in that spanclaude-opus-4.6,claude-sonnet-4.6) all drew Bedrock. Backend selection is upstream-controlled, so a Vertex sample cannot be forced on demand; the item above covers the same claim from the sampler's accumulated draws.