Repository navigation
fix(uploads): SkillForge text fallback + queue self-heal - #26
Merged
Merged
Conversation
Two related production failures observed in the 2nd-pass diagnostic:
NEW-1 — `processBulkUpload` returned `uploaded:0, failed:1` for every
inline call. Both fallback models died with provider-strict-mode
rejections of the structured-output JSON schema generated from
PackageDraftSchema (open `record<string, any>` slots → 400 on
gpt-4o-mini, "No object generated" on gemini-2.5-flash). The catch
swallowed the upstream body, so operators only saw "Bad Request".
- Capture `e.text`, `e.responseBody`, `e.cause` per attempt so the
real reason is in the logs instead of an opaque message.
- After structured attempts exhaust, fall back to plain `generateText`
asking for raw JSON, then parse + zod-validate locally. Loses
provider-side schema enforcement but yields a draft instead of a
fatal error. Tolerant extractor handles bare JSON, fenced JSON,
and brace-balanced first-object recovery.
NEW-2 — uploads were stuck in `queued` indefinitely (>14h). Two bugs:
the worker hardcoded `attempts: 1` on each claim (so MAX_ATTEMPTS
gating could never trigger), and a job that crashed mid-flight stayed
in `processing` forever with no recovery path.
- Each drain pass now scans for jobs in `processing` older than 2min
and either requeues them (attempts < 3) or marks `failed` with an
explicit "abandoned" reason (attempts >= 3). Returns `requeued`
in the summary so cron logs make the healing visible.
- Claim increments `attempts` from the prior value instead of
overwriting to 1.
https://claude.ai/code/session_019gMoupKKTVydpNwiiACQRd
criptogus
marked this pull request as ready for review
May 28, 2026 02:31
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Diagnostic round 2 surfaced two production failures behind the freshly-shipped REST upload path. Same root areas, two different layers.
NEW-1: SkillForge author 0/1 on every inline upload
processBulkUploadwas returninguploaded:0, failed:1for every call. Both structured-output fallback models died:openai/gpt-4o-mini→Bad Request(provider strict mode rejecting the JSON schema generated from PackageDraftSchema's openrecord<string, any>slots).google/gemini-2.5-flash→No object generated: response did not match schema.e.messagereached logs, so the symptom was a generic "Bad Request" with no signal.Fixes
describeAttemptErrorextractse.text(NoObjectGeneratedError),e.responseBody(HTTP errors), ande.cause.message. Real provider body now lands in logs.generateTextasking for raw JSON, then parsed locally and validated withPackageDraftSchema. Tolerant extractor handles bare JSON, ```json fences, and brace-balanced first-object recovery. Loses provider-side schema enforcement but yields a draft instead of a fatal error.NEW-2: queue zombies — uploads stuck in
queuedindefinitelyattempts: 1on every claim, soMAX_ATTEMPTSgating could never trigger.processingforever with no recovery path.Fixes
processingjobs withstarted_at < now-2min:attempts < 3→ flip back toqueued, clearstarted_at.attempts >= 3→ markfailedwith an "abandoned" reason so the user sees a real outcome.attemptsfrom the prior value instead of overwriting.requeuedso cron logs surface the healing.Out of scope (still TODO)
Bearer sas_…rejection (suspected separate auth context inside the MCP tool, not the route). Will tackle next.super-agentCLI on npm — needs a publish or a snippet revision; not a code-in-this-repo fix.Test plan
POST /api/packages/uploadwith a real file returnsuploaded:1, failed:0— text fallback should catch any remaining structured-output strictness issues.results[0].errornow containstext=…orbody=…snippets pointing at the upstream cause, not just "Bad Request".UPDATE package_upload_jobs SET status='processing', started_at=now()-interval '5 minutes' WHERE …; hit/api/jobs/drain-upload-queue?secret=…. Job flips back toqueued(attempt 1) orfailedwith "abandoned" (attempt 3).requeuedcounter./account/packagesclear within one cron tick after this deploys.https://claude.ai/code/session_019gMoupKKTVydpNwiiACQRd
Generated by Claude Code