Repository navigation
perf: send GitHub only the changed paths when publishing a tree - #73
Conversation
A development task on a large repository failed after five attempts with
POST /git/trees answering 502 or 504 ("We couldn't respond to your request
in time"). Without skipped files the bot listed every path of the
repository and no base tree, so GitHub had to rebuild the whole tree and
ran past its ten-second request limit on every retry. It was not a rate
limit: those answer 403/429.
The tree now carries only changed, added and deleted paths on top of the
snapshot's source tree, and the result is still checked against the
candidate's tree SHA. A path that turns from a file into a directory or
back, or a tree GitHub builds differently, falls back to the complete
listing.
Automatic retries now wait 5, 15, 45 and 135 seconds instead of five
seconds each, and the give-up guidance mentions oversized or slow requests.
![agent-ci-infra-bot[bot]](https://avatars.githubusercontent.com/u/343315635?s=60&v=4){kind=small}
There was a problem hiding this comment.
Reviewed commit: 82093c8
Coverage: Snapshot: 199 files, 3383335 bytes; skipped 0 entries; Initial model context: 22 files; other accepted files available through tools; Git workspace contains a fixed snapshot, not full upstream history
| const before=[...upstream.keys(),...skipped.map((f:any)=>f.path)],files=new Set(before),directories=new Set(before.flatMap(parents)); | ||
| const reshaped=changes.some((f:any)=>directories.has(f.path)||parents(f.path).some(path=>files.has(path))); | ||
| const create=(body:any)=>this.github.request('POST',this.github.path('/git/trees'),body); | ||
| let tree=candidate.source_tree&&changes.length+deletions.length&&!reshaped?await create({base_tree:candidate.source_tree,tree:[...changes,...deletions]}):null; |
There was a problem hiding this comment.
F1 — medium, suggestion — robustness — a rejected incremental tree request never falls back to the complete listing
- Where:
src/github/publisher.ts:90 - Problem: the fallback on the next line only runs when GitHub answers with a tree SHA different from
expected. If GitHub rejects the POST that carriesbase_treeinstead (4xx such as 422 for an entry it cannot apply on top of the base tree, or a 5xx),create()throws and the complete listing is never tried. Before this change a repository without skipped files always sent a complete listing withoutbase_tree, so there was no base tree for GitHub to refuse. - Conditions: a candidate whose delta GitHub refuses to apply to
source_tree— a reshape the localreshapedcheck does not model, or a base tree GitHub no longer resolves. The local check covers file↔directory conversion for every changed/deleted path of the snapshot, so this is hardening rather than a known failing input; I could not produce a concrete rejection on a real repository. - Impact: the publish fails (
github_http_422→github_rejected) or repeats the identical request through all five automatic retries, which is the same failure loop this PR is removing, just with a different trigger. - Fix: treat a rejected incremental request like a SHA mismatch — try the complete listing once, then fail:
let tree=null;
if(candidate.source_tree&&changes.length+deletions.length&&!reshaped){
try{tree=await create({base_tree:candidate.source_tree,tree:[...changes,...deletions]});}
catch(error){if(!(error instanceof TaskError))throw error;tree=null;}
}An unreferenced tree object left behind is harmless; trees are content-addressed, so the following complete-listing request cannot corrupt the branch.
|
The PR sends GitHub an incremental tree (base_tree plus changed/added/deleted entries, with reshaped candidates and SHA mismatches falling back to the complete listing) and makes automatic retry_wait retries back off 5/15/45/135 s with clearer give-up guidance. The logic, docs and tests look sound: I ran Reviewed commit: 82093c8 |
A development task on a large private repository failed after five automatic attempts. Every attempt failed on the same request:
This was not a rate limit, which answers 403/429 and is reported as
github_rate_limited. When a candidate has no skipped files,Publisher.branch()sent GitHub the complete list of every path in the repository and nobase_tree, so GitHub had to rebuild the whole tree. On a large repository that takes longer than GitHub's ten-second request limit, so each retry hit the same timeout. GitHub's status page shows no incident at the time.Change
sha: null) on top of the snapshot'ssource_tree. Unchanged files are inherited from the base tree, as skipped entries already were. GitHub's answer is still checked against the candidate's tree SHA.github_tree_mismatch.retry_waitretries back off: 5, 15, 45 and 135 seconds, instead of 5 seconds each. A short GitHub or model outage is outlasted rather than using up all five attempts in about a minute and a half. Rate limits still followRetry-After.Validation
npm run checke2e/failure-retry.spec.mjspassed.github_tree_mismatch.base_treelike GitHub (null SHA removes a path). The development flow test asserts the published tree is one entry on top of the source tree, and the existing development tests still publish matching trees through it.