Skip to content

Codex release task: qualify final BharatCode build and publish evidence (no speculative features) #72

New issue
New issue
Open
Open
Codex release task: qualify final BharatCode build and publish evidence (no speculative features)#72

Description

@Pavithran-R-A
Pavithran-R-A
opened on Nov 8, 2026

Objective

Qualify MergeSutra for the BharatCode Build League by 10 Nov 2026, 23:59 IST. Work from the latest main and review PR #71 (judge-facing README) first. The goal is 100% of explicitly defined submission gates, not a claim of perfect production reliability.

Use this issue as the Codex engineering task. A Codex coding environment needs repository access; writing this issue does not itself start Codex.

Non-negotiable guardrails

  • Work on topic branches. No unreviewed pushes or merges to main.
  • Preserve sandbox isolation, path containment, request ledger, secret redaction, digest-bound approval, and the refusal to force-push/merge.
  • Never open remote draft PRs through MergeSutra or invoke pr --publish against a real customer repo.
  • No API credits or paid requests without explicit owner authorization, even when a secret is configured. Never print credentials.
  • No speculative rewrite, dashboard, IDE extension or feature creep.
  • Maintain immutable evidence: exact SHA, command, OS, Node, timestamp, outcome, reason and public workflow/job link for each gate.

Gate A — README and cross-platform engineering

  1. Review PR docs: rebuild README as a judge-first, evidence-backed landing page #71's diff, run formatting, lint, typecheck, build, full deterministic suite; catch any README promise/config/platform/shipped-pointer guard failures.
  2. Check all six GitHub CI jobs: Ubuntu, Windows and macOS on Node 22 and 24.
  3. Run npm run qualify:path-shapes, npm run verify:package, npm run test:artifact, and npm audit --omit=dev --audit-level=high.
  4. Run npm run qualify:customer in the documented Docker fixture (8 scripted journeys), verify independent oracle and the customer-gate summary.
  5. Distinguish tests executed at PR head from those at the eventual merged main SHA. If a gate fails, add a reproducible regression test, smallest safe fix and focused PR.

Gate B — controlled live BharatCode preflight

Read .github/workflows/live-validation.yml, docs/REAL_MODEL_E2E_PROCEDURE.md, docs/REAL_TASK_BENCHMARK.md and docs/SANDBOX_EXECUTION.md.
Verify all:

  • refs/heads/main plus explicit run_live=true guard.
  • Catalog preflight lists the chosen eligible model (qwen-3.8-27b only if account has access).
  • GitHub token is step-scoped to trusted driver only; no credential reaches npm/install/tests/repository commands or artifacts.
  • Docker is digest-pinned and network-disabled for repo-owned commands.
  • Atomic request ledger bounds model completions to 12; check no reruns reset it.
  • Independent oracle includes baseline, changed-file, behavior, review, wall-clock and negative-control checks.
  • The workflow cannot create/push/merge a real PR and does not leak secrets.

Only after the owner explicitly authorizes the possible request cost and the API key is securely configured, dispatch one controlled full live-model run on the exact green merged main SHA. Stop on failed model eligibility or any preflight. Do not blindly rerun.

Gate C — final presentation evidence

  1. Use actual model/fixture results in the judge guide and README; do not turn a scripted run into a live claim.
  2. Confirm BharatCode.txt at root and public repo accessible.
  3. Confirm public BharatCode submission already exists and the category is Coding & developer tools (user screenshot shows entry).
  4. Check demonstration link, badges, npm package-scope links and accessibility/readability.
  5. Record current limitations clearly: unimplemented unattended run, no native IDE plugin, known final-check-to-rename race, 0/10 live benchmark until executed.

Final deliverable

A GitHub markdown release report with a PASS/FAIL/BLOCKED table and exact evidence refs:

  • Qualification SHA, six CI jobs, eight scripted customer journeys.
  • npm run check, package, artifact, path qualification, production audit counts.
  • Real-model run ID/model/completions/budget/oracle/secret-scan result, if authorized and performed.
  • Open blockers with minimal fix PRs.
  • Explicit verdict: SUBMISSION_GATES_PASS or BLOCKED; never assert success from skipped tests.
  • Do not use a rounded percentage as a substitute for named passing gates.

Relevant: PR #71 #71 ; previous green main CI https://github.com/Pavithran-R-A/mergesutra/actions/runs/42492506208 .

Activity

  1. Pavithran-R-A commented on Nov 8, 2026

    @Pavithran-R-A
    Pavithran-R-A
    on Nov 8, 2026 – with ChatGPT Codex Connector
    OwnerAuthor

    Release checkpoint — 8 Nov 2026

    • README overhaul PR #71 merged into main at 801aab0de847db4d5237a68b5f3d195aaaa62afc.
    • Fresh merged-main CI run #42495944794 PASS (6/6) across Ubuntu/Windows/macOS × Node 22/24. A completed Ubuntu job reports 2,976 passed / 18 skipped deterministic tests, 115 package tests, 15 artifact tests, 0 production audit vulnerabilities. No claim that skipped live tests ran.
    • PR docs: rebuild README as a judge-first, evidence-backed landing page #71's eight scripted customer journeys PASS at pre-merge SHA 57cecc64. The model in that gate is scripted, not real.
    • One controlled live model qualification was dispatched on current main as workflow #42497165107 with run_live=true, qwen-3.8-27b, run attempt 1, and a ceiling of 12 completion requests. Result pending. No PR publish step.
    • Repository About metadata now uses the evidence-first value proposition; no source code was changed.

    Codex continuation: Inspect workflow #42497165107's terminal outcome, independent oracle, request ledger, and sanitized evidence before updating release verdict or the README. Do not rerun a failed workflow attempt (the guard explicitly refuses request-ledger reset). If live qualification fails, root-cause and reproduce before a minimal PR; keep all security gates. Ten-case benchmark is still a distinct unexecuted reliability gate.

  2. Pavithran-R-A commented on Nov 8, 2026

    @Pavithran-R-A
    Pavithran-R-A
    on Nov 8, 2026 – with ChatGPT Codex Connector
    OwnerAuthor

    Confirmed qualification failure — 8 Nov 2026, 21:56 IST

    Verdict remains BLOCKED. Cross-checked GitHub Actions on exact main SHA 801aab0de847db4d5237a68b5f3d195aaaa62afc:

    • Live run #42497165107 completed failure on first attempt; both Linux and Windows no-key jobs succeeded.
    • Catalog GET /models returned HTTP 200 with qwen-3.8-27b listed.
    • The bounded plan request returned: BharatCode answered, but its response body did not finish arriving within 290000ms. The model output did not complete. No implementation, independent acceptance, full baseline preservation or review occurred.
    • Failure artifact #13624078273 uploaded. It did not record the ledger reservation count; the twelve-completion ceiling configuration alone is not evidence of requests consumed.
    • The candidate's six-platform CI #42495944794 succeeded.
    • A Codex/local correction to include failure-artifact reservation counts was reported at 743a0c66b93c5b7be64253df20506b8c48faecd0 on local branch fix/live-failure-ledger-evidence-20261108. That commit and branch are not in the GitHub remote as of this verification; they cannot be reviewed/qualified here yet.

    Next Codex steps

    1. Inspect the local diff and regression tests; push only the reviewed fix branch to origin, open a minimal PR, and run fresh hosted CI and artifact gates. Never claim the local result is hosted.
    2. Preserve null for unreadable ledger counts; report both configured request ceiling and observed reservations on all failure exits, without disclosing the absolute ledger path, API key or response contents.
    3. Separately investigate the actual long-running response: confirm whether headers/SSE frames/terminal events were received and whether a safe x-request-id can be included in a redacted diagnostic report; do not conflate 'stream opened' with a completed model response. Avoid logging raw model payload or credentials.
    4. Consider reducing the planner's request complexity / output budget or explicitly disabling reasoning only if existing schemas and acceptance coverage remain intact and with deterministic regression evidence. BharatCode officially documents max_output_tokens and reasoning:{effort:'none'} for the Responses endpoint. Do not claim either will fix this timeout without evidence.
    5. Do not rerun workflow #42497165107, raise budgets, make paid changes or initiate another live call without fresh owner authorization. A new trial would require a new run with its own explicit cost/allowance decision.
    6. Keep the README/judge guide honest: old historical live success remains historical; current HEAD qualification and the ten-task independent benchmark are unproven. A test timeout is not evidence of a defective code patch, model identity, or provider root cause.

    Official API contract: https://bharatcode.ai/build ; provider capacity/queue caveat: https://bharatcode.ai/terms .

    Release verdict: BLOCKED until the independent oracle, baseline check, review and auditable request count are evidenced on a qualified SHA.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions