Skip to content

fix(import): delete GitHub skill blobs when import fails - #3612

Merged
Patrick-Erichsen merged 8 commits into
openclaw:mainfrom
SebTardif:fix/github-import-blob-cleanup
Sep 16, 2026
Merged

Patrick-Erichsen merged 8 commits into
openclaw:mainfrom
SebTardif:fix/github-import-blob-cleanup

Conversation

@SebTardif

@SebTardif SebTardif commented Sep 5, 2026 •

Copy link
Copy Markdown
Contributor

GitHub skill imports now validate required metadata and resolve ownership before storing files, then delete partial or rejected uploads only while the request owns them. The shared publishing helper transfers ownership immediately when a pending or published version commits, preserving imported files through later scheduler or compensation failures. This completes Sebastien Tardif's existing cleanup patch.

Fixes #3672. Uses the shared persistence signal landed in #3549.

Native local Convex proof: invalid-version imports went from two orphan blobs to no stores; partial-store and rejected-insert failures now retain zero blobs. The previous PR deleted both committed files after a scheduler error (download 404); the repaired version retained them (download 200, 989 and 43 bytes). Pending success, successful compensation, failed compensation and normal publication also passed. The proof uses synthetic GitHub API/archive responses with real Convex identity, storage and publication; no production data was used.

Validation: 54 importer/shared-publishing tests passed after merging current main; static, full types/build and explicit schema/CLI/Convex typechecks passed; structured repair review returned no findings. Full coverage passed with 6,976 tests using four workers; the initial parallel run timed out in an existing large-archive test. No test or timeout was weakened.

Refreshed on main after #3549 merged. The final diff contains only the importer, its tests and its ownership invariant. The refresh preserves the exact previously validated file tree.

importGitHubSkillForUser stored Convex blobs before validating
slug/owner/semver, and publish failures left those ids unreferenced.

Signed-off-by: Sebastien Tardif <SebTardif@ncf.ca>
@clawsweeper

clawsweeper Bot commented Sep 5, 2026 •

Copy link
Copy Markdown
Contributor

🦞👀
ClawSweeper picked this up.

Pull request received. I will update this pull request when review starts.

ClawSweeper review complete

ClawSweeper finished reviewing this revision. The review result is being finalized.

View the workflow run.

@vercel

vercel Bot commented Sep 5, 2026

Copy link
Copy Markdown
Contributor

@SebTardif is attempting to deploy a commit to the OpenClaw Foundation Team on Vercel.

A member of the Team first needs to authorize it.

@clawsweeper clawsweeper Bot added P2 Normal backlog priority with limited blast radius. merge-risk: 🚨 compatibility 🚨 Merging this PR could break existing users, config, migrations, defaults, or upgrades. rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. labels Sep 5, 2026
@clawsweeper

clawsweeper Bot commented Sep 5, 2026 •

Copy link
Copy Markdown
Contributor

Codex review: needs maintainer review before merge. Reviewed September 16, 2026, 1:31 AM ET / 05:31 UTC (Revision 3).

ClawSweeper review

What this changes

GitHub skill imports validate metadata before uploading files and delete failed uploads only until publication takes ownership of them.

Merge readiness

✅ Ready for maintainer review

No blocking findings. The earlier committed-file deletion defect is resolved, and the updated local Convex evidence covers cleanup and preservation after persistence. Main still lacks cleanup for GitHub imports, so this PR remains necessary.

Priority: P2
Reviewed head: 3a56921bc1501e66f6a3eba4e3d86e92857a3556

Review scores

Measure Result What it means
Overall readiness 🐚 platinum hermit (4/6) A focused repair with four regression cases and relevant native Convex results resolves the previous correctness and proof blockers.
Proof confidence 🐚 platinum hermit (4/6) Sufficient (live_output): The updated evidence exercises the importer with synthetic GitHub responses but real local Convex identity, storage and publication: failed uploads leave no blobs, and committed files remain downloadable after scheduler failure. Pending and compensation cases cover the shared ownership handoff.
Patch quality 🐚 platinum hermit (4/6) No actionable review findings were identified.

Verification

Check Result Evidence
Real behavior Verified Sufficient (live_output): The updated evidence exercises the importer with synthetic GitHub responses but real local Convex identity, storage and publication: failed uploads leave no blobs, and committed files remain downloadable after scheduler failure. Pending and compensation cases cover the shared ownership handoff.
Evidence reviewed 10 items Reviewed scope and policy: Read the complete root AGENTS.md, ClawHub maintainer and Convex skills, and generated Convex guidelines. No nested AGENTS.md files were found under convex/ or specs/, and .agents/maintainer-notes/ is absent. The pinned introduced diff is three files, +280/-73; the checkout remained clean.
Current main still leaks failed import uploads: Main stores selected files before metadata validation and has no cleanup around partial stores or publication failure. The merged multipart fix at #3549 supplies the shared persistence signal but does not repair this import entrypoint.
Latest release comparison: The v0.23.3 importer also stores files before validating metadata and lacks failure cleanup. Its source was inspected through the GitHub contents API after local historical blob retrieval failed.
Findings None None.
Security None None.

How this fits together

ClawHub imports selected files from a publisher’s GitHub repository into Convex storage, then creates a skill version. Cleanup must remove rejected uploads while preserving files referenced by pending or published versions.

flowchart TD
  A[Selected GitHub files] --> B[Validate metadata and publisher]
  B --> C[Store import files]
  C --> D[Publish skill version]
  D --> E{Version committed?}
  E -->|No, import failed| F[Delete request uploads]
  E -->|Yes| G[Version owns files]
  G --> H[Followups or pending compensation]
Loading

Before merge

None.

Agent review details

Security

None.

Review metrics

Metric Value Why it matters
Diff size +280/-73 across 3 files The final scope is limited to the importer, regression tests and its documented ownership invariant.
Production and test growth Production +15 net; tests +188; specs +4 Production growth implements bounded failure cleanup and reuses the existing persistence signal.
Importer regression cases 4 added Coverage distinguishes failed uploads from files already owned by a committed version.

Root-cause cluster

Relationship: fixed_by_candidate
Canonical: #3672
Summary: This PR implements the import-specific cleanup tracked by the open issue; the merged multipart repair supplies a shared prerequisite.

Members:

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Technical review

Best possible solution:

Keep import cleanup local to newly uploaded files and use the shared commit signal as the definitive handoff to version ownership.

Do we have a high-confidence way to reproduce the issue?

Yes. Main visibly stores files before metadata validation and leaves earlier uploads behind after later failures; the supplied local Convex results corroborate that path, although this review did not execute it.

Is this the best way to solve the issue?

Yes. Reusing the shared persistence callback is the narrowest fix: deleting on every publication error corrupts committed versions, while moving cleanup into the shared helper would lose the importer’s request-specific ownership boundary.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning medium; reviewed against e5f60853e9dd.

Labels

Label changes:

  • add proof: sufficient: Contributor real behavior proof is sufficient. The updated evidence exercises the importer with synthetic GitHub responses but real local Convex identity, storage and publication: failed uploads leave no blobs, and committed files remain downloadable after scheduler failure. Pending and compensation cases cover the shared ownership handoff.
  • add rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🐚 platinum hermit and patch quality is 🐚 platinum hermit.
  • add status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (live_output): The updated evidence exercises the importer with synthetic GitHub responses but real local Convex identity, storage and publication: failed uploads leave no blobs, and committed files remain downloadable after scheduler failure. Pending and compensation cases cover the shared ownership handoff.
  • remove merge-risk: 🚨 compatibility: Current PR review selected no merge-risk labels.
  • remove status: 📣 needs proof: Current PR status label is status: 👀 ready for maintainer look.
  • remove rating: 🦪 silver shellfish: Current PR rating is rating: 🐚 platinum hermit, so this older rating label is no longer current.

Label justifications:

  • P2: This repairs a bounded authenticated-import storage leak without evidence of an urgent production outage.
  • rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🐚 platinum hermit and patch quality is 🐚 platinum hermit.
  • status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (live_output): The updated evidence exercises the importer with synthetic GitHub responses but real local Convex identity, storage and publication: failed uploads leave no blobs, and committed files remain downloadable after scheduler failure. Pending and compensation cases cover the shared ownership handoff.
  • proof: sufficient: Contributor real behavior proof is sufficient. The updated evidence exercises the importer with synthetic GitHub responses but real local Convex identity, storage and publication: failed uploads leave no blobs, and committed files remain downloadable after scheduler failure. Pending and compensation cases cover the shared ownership handoff.

Evidence

What I checked:

  • Reviewed scope and policy: Read the complete root AGENTS.md, ClawHub maintainer and Convex skills, and generated Convex guidelines. No nested AGENTS.md files were found under convex/ or specs/, and .agents/maintainer-notes/ is absent. The pinned introduced diff is three files, +280/-73; the checkout remained clean. (AGENTS.md, 3a56921bc150)
  • Current main still leaks failed import uploads: Main stores selected files before metadata validation and has no cleanup around partial stores or publication failure. The merged multipart fix at fix: delete unpublished skill blobs when publish fails #3549 supplies the shared persistence signal but does not repair this import entrypoint. (convex/githubImport.ts:308, e5f60853e9dd)
  • Latest release comparison: The v0.23.3 importer also stores files before validating metadata and lacks failure cleanup. Its source was inspected through the GitHub contents API after local historical blob retrieval failed. (convex/githubImport.ts:308, 87ca030c30f3)
  • Earlier blocking finding resolved: The importer now sets filesPersisted through onFilesPersisted and skips request cleanup afterward. The shared helper invokes that callback immediately after both published and pending insertion, before scheduling or attempt creation. Pending compensation remains responsible for its own deletion. (convex/githubImport.ts:401, 3a56921bc150)
  • Shared ownership contract provenance: The GitHub commit patch adds the persistence callback at both insertion boundaries in the merged multipart fix. Raw commit inspection records parent be87bfd. Local blame could not complete because historical blobs were unavailable; the exact patch was inspected through REST instead. (convex/lib/skillPublish.ts:588, e5f60853e9dd)
  • Native Convex behavior evidence: The captured PR body, unchanged in the live REST response at the pinned head, reports synthetic GitHub responses with real local Convex identity, storage and publication. Invalid metadata produces no stores; partial-store and rejected-insert failures retain zero blobs; post-commit scheduler failure preserves both files with HTTP 200 downloads of 989 and 43 bytes instead of the earlier 404s. Pending success, successful and failed compensation, and normal publication were also exercised. This replaces the earlier in-memory-only evidence; no media artifacts were supplied. (3a56921bc150)

Likely related people:

  • unknown: The claimed source-line change could not be verified from bounded local history. (role: source history unknown; confidence: low)
  • vyctorbrzezowski: Suggested for follow-up; no historical authorship or introduction is verified. (role: unverified routing candidate; confidence: low)

Rating scale

Score Internal tier Crab rank Meaning
6/6 S 🦀 challenger crab Exceptional readiness
5/6 A 🦞 diamond lobster Very strong readiness
4/6 B 🐚 platinum hermit Good normal PR; ordinary maintainer review
3/6 C 🦐 gold shrimp Useful, but confidence is limited
2/6 D 🦪 silver shellfish Proof or implementation needs work
1/6 F 🧂 unranked krab Not merge-ready
N/A NA 🌊 off-meta tidepool Rating does not apply

Overall follows the weaker of proof and patch quality.
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

Workflow

  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

History

Review history (2 earlier review cycles)
  • reviewed 2026-09-05T12:20:46.539Z sha 094ebd4 :: needs real behavior proof before merge. :: [P1] Preserve blobs once publication has committed
  • reviewed 2026-09-11T14:08:18.574Z sha 094ebd4 :: needs real behavior proof before merge. :: [P1] Preserve blobs once publication has committed

Multipart POST /api/v1/skills stored each uploaded file, then parsed
the payload. An invalid body, a later oversized file, or a 400 after
parse (license reject, owner resolution) left Convex blobs unreferenced.

Match parseMultipartSkillScan: reject oversized parts before store,
delete stored ids if store or parse fails, and delete them when the
handler returns 400 after a successful parse.

Replayed onto upstream/main d3bde70.

Signed-off-by: Sebastien Tardif <sebtardif@ncf.ca>
@clawsweeper clawsweeper Bot added rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. and removed rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. labels Sep 11, 2026
@Patrick-Erichsen
Patrick-Erichsen merged commit 19bb43f into openclaw:main Sep 16, 2026
34 of 36 checks passed
@clawsweeper clawsweeper Bot added proof: sufficient Contributor real behavior proof is sufficient. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. and removed merge-risk: 🚨 compatibility 🚨 Merging this PR could break existing users, config, migrations, defaults, or upgrades. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. labels Sep 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

P2 Normal backlog priority with limited blast radius. proof: sufficient Contributor real behavior proof is sufficient. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

GitHub skill import leaves Convex blobs after a failed import

2 participants