This repository was archived by the owner on Sep 18, 2026. It is now read-only.
Repository navigation
v0.5.0: verification scope, native runner, verify-patch, honest verdicts, tested endpoint translation - #3
Merged
Merged
Conversation
A prompt written as adjacent string literals inside parentheses is one
string to the interpreter — the compiler joins them with nothing. adapt
joined every chunk with a newline, so a prompt like
SYSTEM = ("You are a screener. " "Return JSON with keys a, b.")
came out with a newline the upstream agent never sends, while both chunks
were reported as verbatim-verified.
The gate now records each surviving chunk's exact span inside the source
string literal it came from (verify.locate_chunks_in_source, reading the
file's ast.Constant values, where Python has already performed the
concatenation). generate.build_system_prompt joins two chunks with ""
when they sit end to end inside the same literal, and keeps the
documented single-newline join everywhere else: separate statements,
non-Python sources, templated chunks, unresolvable citations.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
`upshift cost` reported "unknown rate" for every run on gpt-5.6-luna and gpt-5.6-terra, which turns the cost column of a real run into a guess. Standard-tier rates per 1M tokens from https://developers.openai.com/api/docs/pricing, fetched 2026-09-03: terra $2.00 input / $0.20 cached / $12.00 output, luna $0.20 / $0.02 / $1.20. gpt-5.6-sol's $4.00 / $0.40 / $20.00 is unchanged and confirmed by the same fetch. The published flex and batch rows are exactly half the standard row and every cached-input row is exactly 10% of the model's input rate, so TIER_MULTIPLIER and CACHED_INPUT_FRACTION already reproduce the table; the test pins that against the published numbers. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
/v1/responses takes forced tool choice flat ({"type": "function", "name": X}), the way
its tool definitions are flat; chat/completions nests it under "function". map_params
translated tool_choice for `messages` but passed it through untouched for `responses`,
so any agent that forces a tool — the shape ChatOpenAI.bind_tools(tool_choice="X")
emits — got `400 Missing required parameter: 'tool_choice.name'` the moment it was
routed to /v1/responses.
That lands on the endpoint_routing repair candidate, which is the documented fix for the
gpt-5.6 family's "use /v1/responses or set reasoning_effort to 'none'" break: the repair
upshift most needs to be able to make was unusable for every forced-tool agent. Found on
a real target (DataTunerX/dac's SD member-capability judge) where the routing repair
screened 0/65 for this reason alone.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Routing a chat/completions agent to /v1/responses left max_tokens / max_completion_tokens in the request; the Responses SDK raises TypeError on either before sending, so the run crashed instead of recording a failed rep, and endpoint routing -- the documented repair for the gpt-5.5+/gpt-5.6 'function tools ... in /v1/chat/completions' 400 -- could not be applied to any agent that sets an output cap. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
`upshift cost` printed "unknown rate" for anything outside the 5.5/5.6 families, and an unpriced leg is precisely what a spend ceiling must not read as free. Standard-tier USD per 1M tokens from https://developers.openai.com/api/docs/pricing, fetched 2026-09-03: gpt-5.2 1.75/14.00 (cached 0.175), gpt-5.2-pro 21.00/168.00, gpt-5-mini 0.25/2.00 (cached 0.025). The published flex and batch rows are exactly half of standard and the cached rows exactly 10% of input, which is what TIER_MULTIPLIER and CACHED_INPUT_FRACTION already compute. gpt-5.2-pro carries its own entry because it is a separate published row at 12x gpt-5.2: without it, longest-prefix matching prices a -pro run as gpt-5.2 and understates the spend twelvefold — a fail-open under a cost ceiling. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
`adapt` makes one paid call and has always had a ceiling. `run` and `upgrade` make thousands — baseline reps, candidate reps, then a screen and a full-suite verify per repair candidate — and had none, so a lab overran a $5 per-case cap inside a single `upgrade` invocation. The ceiling is priced, not estimated: budget.py sums the token usage already recorded under this run id (for `upgrade`, the whole `--tag` family, since every run id in the pipeline is derived from the tag) through the same `pricing` module `upshift cost` uses, so the number enforced and the number later printed are the same number. It is consulted before each rep is dispatched, and again between phases and between repair candidates so a phase landing exactly on the line does not start the next one. On reaching it, nothing is rolled back: the next API call is simply not made. Every finished rep was already written atomically, so rerunning the same command with a higher ceiling resumes and skips them. The command prints the priced total and the phase that stopped and exits 3 — distinct from a STAY PINNED verdict (1) and a usage/API error (2). No verdict is emitted; a COST_STOPPED.json marker lands beside diff.json so a partial pipeline cannot be read as a finished one, and is deleted once a pipeline does finish. Unpriced models fail closed. pricing.ceiling_price charges a model with no published rate at the highest rate in the table rather than returning None, and the command warns loudly at startup that the total is an upper bound: "we do not know" must never be spent as "$0". It also stops letting the provider name decide what is free — sim is free because no rate exists for its model ids, so injecting a rate table makes the whole ceiling testable without a paid provider, which is how tests/test_cost_ceiling.py exercises the stop and the resume. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
`upshift cost` reported "unknown rate" for every gpt-5.4 run, which also means `--max-cost-usd` cannot enforce a ceiling on a leg billed to that model: an unpriced model contributes nothing to the running total the ceiling checks. gpt-5.4 is the baseline a target that predates the temperature deprecation has to use (gpt-5.5 and the whole gpt-5.6 family reject any non-default `temperature`), so it is a model the lab actually runs, not a hypothetical. All four published gpt-5.4 rows are added, not just the base one, because RATES is prefix-matched: `gpt-5.4-pro` bills at 12x `gpt-5.4` and `gpt-5.4-mini` at 0.3x, and a single `gpt-5.4` entry would silently price both as the base rate. Rates from https://developers.openai.com/api/docs/pricing, fetched 2026-09-05. Every published cached-input rate in the family is exactly 10% of that model's input rate, so CACHED_INPUT_FRACTION reproduces the table with no override. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
`upshift cost` reported "unknown rate" for every gpt-4o-mini run, and with `--max-cost-usd` set an unpriced model is charged at the most expensive rate upshift knows. On case ghc-223 that stopped an `upgrade` before its first paid call: the baseline leg's 109k input / 4k output tokens on gpt-4o-mini were billed against the ceiling at $21/$168 per MTok, reported as $1.5337 against a $1.50 ceiling, over a leg whose published price is about $0.013. gpt-4o-mini is not hypothetical: it is what ardhaecosystem/synapse ships as its configured default (`src/synapse/config.py:26`), so it is the last model that accepts graphiti's literal `max_tokens=` and therefore the BASELINE leg of that migration. gpt-4.1-nano is added with it because the same repo's open PR #34 proposes it as the reranker default. Both carry their own published cached-input rates ($0.075 of $0.15 and $0.025 of $0.10), so they need MODEL_CACHED_INPUT_FRACTION overrides: the 90% caching discount is a gpt-5-era rate and would under-report a cache-heavy baseline leg by 4-8x on the cached part. Rates from https://developers.openai.com/api/docs/pricing (Standard tier), verified 2026-09-05. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Every gpt-5-family model on api.openai.com rejects a chat.completions call
carrying `max_tokens`:
400 Unsupported parameter: 'max_tokens' is not supported with this model.
Use 'max_completion_tokens' instead.
That 400 fell into `api_error_other`, which generates no candidates at all.
On case ghc-223 (ardhaecosystem/synapse, whose ingestion path is graphiti's
OpenAIGenericClient sending a literal `max_tokens=16384`) the suite regressed
8/8 cases at p=0.00397 with that byte-identical message on all 50 reps, and
`upshift upgrade` printed STAY PINNED after screening ZERO candidates — while
the fix is one key in agent.json params, inside the `model_params` repair
type, and is exactly the edit the target's own open PR makes.
- differ: new signature `api_error_unsupported_token_cap`, matched on the
API's own wording (`'<param>' is not supported`) so a 400 that merely
mentions a cap ("max_tokens must be an integer") is not swept up, and a
400 naming `temperature` still reads as a sampling rejection.
- playbook: `rename-token-cap-param` renames whichever cap the agent declares
to the one its endpoint accepts (chat_completions ->
max_completion_tokens, responses -> max_output_tokens, messages ->
max_tokens), keeping the author's value, as a one-line textual edit so the
emitted git diff is the line a maintainer would change; skipped when the
agent already uses the right name or declares both. `drop-token-cap-param`
follows as the fallback that loses the author's limit.
- tests/test_token_cap_repair.py pins detector and candidates.
Stacked on fix/gpt-4o-mini-pricing (that case's baseline leg needs a rate).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…gnatures The loop keeps a second copy of the differ's SIGNATURE_PRIORITY and filters the diff's signatures through it before calling generate_candidates. A signature absent from that copy is dropped silently: the playbook is never asked, and the run reports "no further candidates for the observed failure signatures" even though candidates exist. Caught live on case ghc-223: after the previous commit the differ emitted api_error_unsupported_token_cap and the playbook produced rename-token-cap-param for that agent, but `upshift upgrade` still screened ZERO candidates and printed STAY PINNED. Adds the signature to the list and a test that the two lists stay identical except for the one deliberate omission (thinking_block_invalid, which the loop refuses on purpose), so the next signature cannot be swallowed the same way. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Brings in the whole OpenAI-track stack: the adapt implicit-concat join, gpt-5.6-terra/luna + gpt-5.2/5.2-pro/5-mini pricing, the /v1/responses forced-tool_choice flattening and output-token-cap translation, and the --max-cost-usd spend ceiling (src/upshift/budget.py). Conflicts, all resolved as unions — nothing from either side dropped: - CHANGELOG.md: main's `v0.4.1-dev — unreleased` section kept verbatim; the branch's `## Unreleased` bullets folded into it under a lead-in naming the OpenAI migration-rescue track, above `v0.4.0-dev`. - src/upshift/agent_loop.py: kept BOTH module constants (TERMINAL_TOOL_RESULT, TOKEN_CAP_PARAMS), both map_params accumulators (`sampling`, `cap`), all four endpoint branches (they are on disjoint endpoints: MESSAGES sampling params vs RESPONSES tool_choice/token cap), both post-loop steps (_place_sampling_params then the max_output_tokens translation) and both helpers (_place_sampling_params, _responses_tool_choice). - src/upshift/cli.py: kept both import groups (budget + capture) and both cmd_upgrade features — the cost ceiling is built before the baseline run and the whole pipeline stays inside the branch's try/stopped() block, with main's BASELINE_BROKEN warning kept between the two legs (re-indented into the try, reflowed to the line limit). - tests/test_agent_loop.py, tests/test_pricing.py: union of both sides' test sections; no test from either side lost, none duplicated. No test needed changing. 1528 tests pass; `ruff check src tests` clean.
Adds gpt-4o-mini / gpt-4.1-nano pricing (with their own published cached-input
fractions), the gpt-5 token-cap-parameter 400 detector + repair candidates in
differ/playbook, and the loop fix that stops the private signature priority list
from swallowing newly added signatures.
Conflicts, all resolved as unions:
- src/upshift/pricing.py: module docstring rewritten to state BOTH verifications
(the whole gpt-5.6 family re-verified 2026-09-03, gpt-4o-mini/gpt-4.1-nano
2026-09-05) and to keep the branch's caveat that the two pre-gpt-5 models carry
their own cached-input fractions rather than the gpt-5-era 10% default; RATES
keeps both blocks — gpt-5.6-terra/luna + gpt-5.2/5.2-pro/5-mini from the
previous merge AND gpt-4o-mini/gpt-4.1-nano. 18 entries, no duplicates.
- src/upshift/repair/playbook.py: both helpers kept
(_leaves_an_empty_extra_body, _agent_json_rename_param) — unrelated functions.
- tests/test_pricing.py: union of imports (`pytest` + `cached_input_fraction`)
and of both sides' test blocks; no test lost.
One source fix, not a test fix: main's capture framework-mapping table did not
know the branch's two new patch ids, and tests/test_capture_mapping.py asserts
every model_params patch id is classified. Added a TOKEN_CAP category
("output-token cap") to capture/mapping.py and mapped `rename-token-cap-param`
and `drop-token-cap-param` to it. No FRAMEWORKS entry names it yet, so it renders
as "not mapped" — the module's documented behaviour for a knob whose per-framework
spelling has not been verified in source, and honest rather than silent.
1549 tests pass; `ruff check src tests` clean.
Adds the published gpt-5.4 / -mini / -nano / -pro rates (fetched 2026-09-05). Conflict: - tests/test_pricing.py: git interleaved the two sides' independent additions across three hunks. Resolved by taking the integration branch's file whole (which already carries the gpt-5.6-family, gpt-5.2-family, gpt-4o-mini / gpt-4.1-nano and Anthropic-model tests plus the `cached_input_fraction` import) and appending the branch's gpt-5.4 section verbatim — GPT_54_PUBLISHED and its three tests. Verified no test function from either side is missing from the result. src/upshift/pricing.py auto-merged: the gpt-5.4 rows sit above gpt-5.5 and do not collide with any other branch's rows. 1558 tests pass; `ruff check src tests` clean.
…ner, honest verdicts, patch verification, capture continuation, translation matrix) + working file Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ow-ups Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…repair disclosures DESIGN.md §G. Three changes that all serve one rule: a run must never report a harness or configuration failure as something the model did. 1. agent_loop gains a declarative TRANSLATION_TABLE (reasoning, output cap, tool_choice, sampling, seed, state-linking) with per-endpoint spelling, placement, legal values and a documented precedence rule per row. translate_params returns the request fields AND everything the record must say: dropped_params (with reasons), passthrough_params, notes, and determinism: best_effort for a seeded run - never 'deterministic'. An effort value the target endpoint has no rung for is a TranslationError, never a substitution: substituting 'none' would turn reasoning off and report the result as the same agent. State-linking params (previous_response_id, conversation, store, metadata, user) are dropped on upshift's one-shot replays and every drop is recorded. map_params keeps its exact signature and byte-for-byte behaviour. 2. Providers distinguish an HTTP error the provider returned (status_code set, message verbatim) from a LOCAL SDK validation error raised before any request went out (error_type sdk_validation, status_code None). The anthropic provider no longer manufactures a 400 for an unexpected keyword: nothing was sent, and the same in-process TypeError hits both models of an upgrade pair, so it never distinguished them. Three tests that encoded that manufactured 400 are updated and say why. 3. differ gains harness_error - sdk_validation and translation_error, detail 'the request never reached the provider' - ranked first in SIGNATURE_PRIORITY, plus api_error_unsupported_effort_value for the real gpt-5.6 'Supported values are: ...' 400. SIGNATURES_WITHOUT_REPAIRS is now the named contract for what the repair loop may legitimately not act on. 4. playbook gains disclosures_for(patch_id): a module-level registry of changes_capability / changes_cost sentences for the repairs that restore a suite by changing what the agent is guaranteed to do (remove-forced-tool- choice, drop-sampling-params, drop-token-cap-param, reasoning-effort-none) or what it costs (the effort raises). generate_candidates now stable-sorts transport/spelling fixes first and capability changes last, so routing an agent to an endpoint that supports reasoning is always tried before turning its reasoning off. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
tests/test_translation_matrix.py (96 cases): every row of TRANSLATION_TABLE gets a positive, a negative and a precedence test, and two coverage tests fail if a row is added without them or forgets an endpoint. Also covers the record itself (dropped_params / passthrough_params / notes / determinism, and one note per API call), the sdk_validation vs HTTP-status split, harness_error, the new api_error_unsupported_effort_value classification, and the repair disclosures and ordering. docs/provider-matrix.md: model family x accepted params/values/endpoint, every row cited to the vendor's own current documentation with the date checked (2026-09-07), and an explicit UNVERIFIED section for what could not be fetched - notably the full /v1/responses parameter table, so nothing about seed, temperature or service_tier on that endpoint is asserted from memory. Where a row is unverified the table forwards the value and records a note rather than encoding a guessed rule. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…dation/harness_error classification, repair disclosures + ranking, provider matrix Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
DESIGN.md §B, §D and §E, plus the statistics wording of §D. - recorder: evidence_id_for() / evidence_id_of() / recompute_evidence_id(); every manifest carries evidence_id, scope, patch_sha256 and agent_dir, and a resume whose identity inputs changed is refused by name instead of silently mixing experiments. Pre-v0.5 manifests (no evidence_id) still resume: runs/ is committed evidence. - verdict: INCONCLUSIVE with ten reason codes. The line the codes protect is that a model's 400 is BEHAVIOUR (A-075's tool_choice 400 is still a regression, still repaired) while billing/auth/harness/short-run/mixed-provider failures are accidents and can never produce SAFE. SAFE no longer claims a "drop-in replacement"; every verdict carries the smallest effect its N could detect. - repair loop: a FRESH final verification of the stacked patch on the full suite at salted seeds, which the verdict rests on; the screen/verify/adjudication runs become selection_runs. A candidate that wins selection and loses the fresh run is STAY PINNED. --no-final-verify equivalent is labelled selection_evidence_only. - collateral: measured, not assumed. broken_by_patch is now a result of the final run, and a run with nothing to protect says so (the rescue-ops shape: "0 protected passing case(s)" in ghi56-006, ghisdk-052, ghc-223 and a dozen more). - patch: make_patch refuses to export when patched_agent/ diverges in a way a diff cannot carry (a non-patchable file, a mode change) and emits git's no-newline marker, so the exported patch IS what was verified. - verify_patch: new `upshift verify-patch` (add_parser here, wiring is the integrator's) — clean copy, git apply --check + apply, rebuild every case's first request and compare with the recorded one; exit 2 with the differing field path. Sends nothing. - report: verification scope, the collateral sentence, evidence ids, fresh-final vs selection evidence, a STALE flag when an agent dir no longer hashes to its run, and a footnote stating the number of unadjusted tests performed. - stats: smallest_detectable_drop(); reference tests against an independent rational Fisher enumerator and Newcombe (1998) Wilson intervals. 1694 tests (was 1558), ruff clean. No committed verdict changes.
…esh final verification, evidence identity, verify-patch, patch-export integrity Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Part A — tests/rescue/: a permanent regression suite for the 12 failure modes the 2026-09 migration-rescue campaign found. Fixtures (tests/rescue/fixtures/incidents.json) encode API request parameters only — endpoint, model pair, params dict — for eight incidents whose upstream repos are MIT or Apache-2.0; LICENSES.md records where each license was read and names the three excluded ones (autonima, atlas-ui-3, plastiq). Every test is marked sim / mocked_transport / recorded_fixture; `live` is registered but conftest.py skips it unless `-m live` is selected. Where DESIGN §v0.5 specifies an interface another stream is still landing, the test imports defensively and skips naming the exact symbol — nothing about the product is stubbed. Part B — tests/test_security.py: adds credential-leak greps over the runs root and the capture store, the container-environment guarantee for the shell_gpt sandbox, and a precise statement of `upshift capture --upstream`'s real behaviour (operator-supplied, http(s)-only, never inferred from request content). Fixes the 24 pre-existing ruff errors under agents/ without changing behaviour, and adds `agents` to the CI ruff step. Part C — docs: replaces every "nothing leaves your machine" claim with the actual data flow (provider calls carry prompts and credentials; `adapt` sends cited code slices to the extraction model; local transcripts can hold sensitive data; nothing reaches a Mechanism backend), adds the three verification scopes to the README, and adds docs/capabilities.md — providers/endpoints/runners against implemented+offline-tested, live-smoke-tested and end-to-end-migration-verified. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The application runs itself (DESIGN.md §C). `agent.json` gains a `runner` block; upshift
invokes the developer's own command once per (case, rep), hands it the case on stdin as
upshift result protocol v1 and reads one JSON object back. Checks, N reps, records,
statistics and verdict are the existing engine — there is no second verification engine.
`runner` + `cases/` is the whole authoring surface: no prompt, tools or backend.py.
Why: 25 OpenAI-track and 36 Anthropic-track rescue cases closed UNSUPPORTED_FRAMEWORK, and
four of five hand-built adapters were hand-built because the target was not Python. Two of
those projects (auditk A-052, policybench A-032) ARE benchmark harnesses — the adapter was a
reconstruction of something that already ran.
- src/upshift/native/{protocol,exec,runner,cli}.py — the wire contract, safe execution,
one episode, the flag surface + authorization gate.
- Failure classes: runner_error (bad exit, timeout, malformed/absent/truncated protocol) and
continuation_exhausted are NON-behavioural; a flaky test command cannot manufacture a
regression.
- Security: no execution without --allow-runner; argv never a shell; minimal env (a child
never sees an unrelated secret, or another provider's key); per-rep temp workdir copy,
deleted after; in-place needs --allow-in-place; stdout cap; timeout kills the process
group; --capture binds loopback only and the child cannot be pointed elsewhere.
- scope (DESIGN.md §A) derived, never declared, and passed to write_manifest when that
parameter exists; every rep carries scope + episode_source.
- Capture continuation (§F): recorded_turns + continuation fail|repeat_last, all policy in
capture/continuation.py, three lines in the agent loop.
- Capture fidelity: capture/fidelity.py names every recorded field the adapter cannot carry
(dropped params, server tools, beta headers, unrepresentable response blocks, gateway
upstreams, unrecorded paths) into unsupported_fields.json + an ADAPT_EDITS.md table.
Denylist-free, so tomorrow's field is a finding on its first capture, not silence.
- examples/runners/: Python and Node reference runners over a stub provider, both driven
through run_suite by tests; ADAPTER.md "Native runner" + examples/runners/README.md.
1622 tests (+64), ruff clean.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…no-final-verify Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ata-flow docs, capabilities table, agents lint Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…nuation agent_loop.py resolved as a union: the continuation policy hook (stream 2, DESIGN §F) runs before the turn is built, then stream 1's table-driven build_request_with_translation/TranslationError path builds and records it.
…VE, honest SAFE sentence - agent_loop: build_request folded translated params in LAST, so a params key could overwrite a field upshift builds (`store: true` re-enabling retention, a stray `tools` replacing the suite's own). MANAGED_REQUEST_FIELDS now win and the override is recorded in dropped_params with reason "managed by upshift" (DESIGN §G). - verdict.decide: expose `inconclusive_reason` (first entry of `reasons`) so the reason code is readable as a scalar; `reasons` stays the authority. - report: the SAFE sentence reads "no regression detected on N case(s) at N reps". - tests/rescue: 04/05/09 adjusted where they guessed an interface name the spec left open (see report); new coverage for the managed-field guard and for a seed dropped on an endpoint that has none.
…ure taxonomy CLI (DESIGN §C): - `--allow-runner` / `--allow-in-place` / `--capture` on `run` and `upgrade` via native.cli.add_runner_args; check_authorized runs before any provider work and its exit code is returned, so an unauthorized native agent prints its argv and stops. - runner_options (and a CaptureSession when --capture is given) are built once per command and passed into every run_suite call, including the four inside repair(). - validate_agent_dir: an agent.json with a `runner` block needs only name/model plus a block that RunnerSpec.from_dict accepts — the prompt, tools and backend belong to the application. Native runs also skip the provider model preflight: the model string is the application's, not upshift's. - an empty eval suite is now INCONCLUSIVE(empty_suite) with a verdict on disk and exit 3 for `upgrade` (DESIGN §D), still a plain usage error for `run`. differ/repair: - native.protocol.is_non_behavioural is consulted BEFORE the 400 wording checks, so a runner_error/continuation_exhausted whose message happens to contain a documented 400 phrase can never be filed as a behavioural signature. - repair/loop's private priority list is derived from differ.SIGNATURE_PRIORITY minus SIGNATURES_WITHOUT_REPAIRS; a test proves a signature added to the differ is seen. tests: tests/test_cli_e2e_v05.py drives all four v0.5 proofs through cli.main on the simulator ($0), plus runner-error-to-INCONCLUSIVE and the differ/loop contract.
…27, ghc-062) Greedy acceptance let the playbook's emission order decide the verdict. In the rescue lab that published two wrong answers: waku (ghisdk-127) accepted a candidate restoring 8 of 9 cases and reported STAY PINNED while a sibling in the same generation restored all nine, and crispen (ghc-062) shipped a repair carrying a disclosed change of guarantee ahead of the maintainer's own equivalent fix. Each signature round now screens every candidate on the still-broken cases first, then ranks the restorers by (a) full restoration then cases restored, (b) no disclosure, (c) playbook rank, (d) playbook order, and verifies them in that order — falling to the next on rejection. Acceptance thresholds and adjudication are untouched; ranking chooses only who faces them. Screening costs one screen run per sibling and `--budget` still bounds candidates tried, so the default budget goes 6 -> 24 and a candidate that lost a round is re-screened after an acceptance (the stacked agent it would apply to is a different agent).
…cepted (ghisdk-052) When a verify run left a protected or earlier-restored case below PASS, the N adjudication reps that settle it were part of the acceptance test only when they could be paid for. In rescue-ops ghisdk-052 hunk 2's contested cases sat at 2/5 and 3/5 and were never adjudicated because the case went over budget. The loop now catches CostCeilingExceeded around the adjudication run: the candidate is rejected, the suspects are recorded as `adjudication_skipped`, the fresh final run is skipped and the outcome is flagged `cost_stopped` — which `verdict.decide` reads, so the run ends INCONCLUSIVE(cost_ceiling) rather than SAFE WITH PATCH even when the caller never saw the exception. An interruption is not swallowed and still accepts nothing. Also pins the restoration side with an explicit test: 2/5 on the screen plus 3/5 on the verify is 5/10, which is not a restoration at the unchanged 0.8 threshold.
…(ghi56-019/006/021)
A flex capacity 429 ("Flex does not have sufficient resources") was recorded three times in
the rescue lab as a permanently failing rep, moving the pass rate the verdict is computed
from. Nothing about the model was measured; the queue was full.
run_suite now retries the whole episode on 429/5xx/timeout/connection errors — up to 3
attempts, exponential jittered backoff, total bounded at 90s, seeded per (case, rep) so a
rerun waits the same — outside the SDK's own retries. A rep that still fails is recorded as
`transient_provider_error`, which native.protocol lists as non-behavioural: the differ files
it under harness_error and the verdict is INCONCLUSIVE(transient_provider_error). Billing
still aborts, auth is not retried, and a 400 is never retried or reclassified.
`--retry-errored` (runner_cli.add_retry_args, wired into run and upgrade) re-runs, on a
resume, only the reps whose recorded error was non-behavioural. The rep file is replaced only
on a pass or a new non-transient outcome, and the replacement names what it replaced in
`retried_from`.
…ion (ghisdk-051, -052)
ghisdk-051 (getzep/graphiti) was a structured-output incident whose 400 read "Invalid schema
for response_format 'ExtractedEdges'". `response_format` was in adapt's BLOCKED_PARAMS, so
adapt deleted the entire subject of the failure and produced an agent that asked for no
structured output at all — for $0.31, and with a green run available at the end of it.
It is now a carried parameter: out of BLOCKED_PARAMS, into DICT_PARAMS (with `reasoning`),
into capture's CAPTURED_PARAMS, and into agent_loop's translation table with the three
required tests — verbatim on chat/completions, `text.format` with the json_schema object
flattened one level on /v1/responses (OpenAI's migration guide: "Instead of response_format,
use text.format in Responses"), dropped with a recorded reason on Anthropic's messages, and an
explicit `text` object the agent wrote wins. capture/fidelity stops listing it as a loss,
because it stopped being one.
ghisdk-052: a `str | None` tool parameter reached the model as {"type": "string"}, the model
had no way to express "nothing" and sent "" instead, which the tool read as an ordinary lookup.
New upshift.jsonschema canonicalises every nullable spelling to anyOf-with-null, recursively,
and adapt applies it to every generated tool schema.
…51, p2-001) graphiti's three-week silent outage (ghisdk-051) was a 400 reading "Invalid schema for response_format 'ExtractedEdges': ... 'additionalProperties' is required to be supplied and to be false". upshift had no signature for it, so the loop had nothing to try. openchamber (p2-001) went 0/13 -> 0/13 on endpoint routing alone and 0/13 -> 12/13 on routing plus a two-line tool-schema edit; "no repair candidate edits a tool or response JSON Schema" was recorded as the product's biggest gap after that case. New differ signature `api_error_schema_invalid`, matched on the API's own four fragments and tested not to steal the sampling, tool-choice, token-cap or tools+reasoning 400s. New playbook candidate `schema-strict-compat:<schema name>` — one per named schema the agent supplies — applying OpenAI's three documented strict rules (every property required, additionalProperties: false, optional properties nullable via anyOf) with the doc URL cited in the code, a minimal textual edit where the serialization allows one, and no candidate at all for a schema that is already strict. Ranked RANK_BEHAVIOURAL: after transport fixes, before capability-disabling ones. Disclosed: "schema made strict — optional fields are now required-and-nullable; downstream code must accept null". Patch ids may now name their target after a colon, and rank/disclosure lookups read the family, so a per-schema candidate cannot lose its disclosure.
…s rows, README native runner / verify-patch / operational notes Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ve --patch failed on a fresh install) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The PR could not be merged, so GitHub never built a merge ref and the `ci` workflow never ran on it. The only conflict was the checkout action pin. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The README described the three-step integration; nothing in the repository showed the finished result, so a reader had to assemble it from prose and nothing stopped it rotting. `examples/runners/python-agent/` and `node-agent/` are complete agent directories — an `agent.json` with a `runner` block and `cases/cases.json`, and nothing else — that run today against the stub provider: `upshift run --agent examples/runners/python-agent --provider sim --allow-runner`, three cases, no key, no network, $0. Point either at `stub-model-b` and every case fails with the documented 400: baseline -> candidate in two commands. Each agent carries the reference runner inside its own `app/` workdir. A `runner.workdir` may never escape the agent directory and `workdir-copy` copies only that directory, so a path to the sibling file would not exist in the copy the command runs in; tests/test_examples_runners.py pins each copy to its sibling byte for byte instead, and runs both agents through `cli.main` asserting scope: native_application and every rep passing. CI's ruff step gains `examples` for the same reason it has `agents`: this is code a user is told to copy and the suite executes. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Three CodeQL actions/missing-workflow-permissions alerts, one per job. Every job checks out, installs, lints, tests or builds; none writes anything back to the repository. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Release integration branch for v0.5.0 — the reliability release. Every change came out of
running upshift against real migration incidents (the private rescue campaign: 109 OpenAI-track
and 54 Anthropic-track cases) and fixing what broke, with a regression test per finding.
What's in it
request_contract/adapted_agent/native_application). No output claims application-level verification unless the applicationactually ran.
agent.jsonrunnerblock lets the application run itself — protocolv1, Python and Node reference runners, an authorization gate (
--allow-runner), a minimalchild environment, a timeout and an output cap, a fresh workdir copy per rep.
upshift verify-patch. The exact exported patch is applied to a clean copy and everycase's first request is rebuilt and compared against the run that verified it. Exit 0 only on
a byte-for-byte match (cache keys and seeds excluded). Sends nothing, costs nothing.
INCONCLUSIVEwith reason codes; collateral protection measured andreported rather than assumed; a fresh final verification run separate from candidate
selection; evidence identity that refuses to resume a run whose inputs changed; SAFE wording
that states what was measured.
seed, state-linking,
response_format. Every drop is recorded; a local SDK failure issdk_validation, never a manufactured HTTP status; repairs that change a capability or acost carry a disclosure and rank last.
acceptance had let emission order decide a verdict); adjudication cannot be skipped; transient
provider errors are retried and never counted as behaviour (
--retry-erroredre-runs exactlythose reps on a resume).
turn_paramsso a capture-derived agent replays the framework's realshape, continuation policy, machine-readable unsupported-fields findings.
matrix with unverified rows marked.
Note:
--budget(max repair candidates) now defaults to 24 because sibling screening countsevery candidate screened. Use
--max-cost-usdto bound spend.Full detail, with the case ids each finding came from:
reports/reliability-upgrade-v0.5.md
and the v0.5.0 section of CHANGELOG.md.
Verification
uv run ruff check src tests agentsclean;uv run pytest -q→ 1999 passed, 0 skipped.uv build+twine checkpass on both artifacts.python:3.12-slimcontainer, non-root, installing only the built wheel:upshift --version→ 0.5.0;init+ the README sim upgrade → exit 0, SAFE WITH PATCH, 36/36 restored,0 broken, scope
adapted_agent;verify-patchwith relative paths → exit 0; the nativerunner refuses without
--allow-runner(exit 2, nothing recorded) and recordsnative_applicationwith it, for both the Python and the Node reference runner; empty suite →exit 3 INCONCLUSIVE(empty_suite); resume with the same config rewrites nothing, and a changed
--nis refused by the evidence-identity guard naming the differing input.The last commit on this branch fixes a bug that the fresh-install check found:
verify-patchran
git applywith cwd set to the clean temp copy, so a relative--patchpath — what theREADME prints and every user types — could not be opened. Paths are resolved before the cwd
changes, with a regression test at both the function and the command level.