Skip to content

fix(bugbot): ground JSON values in reader contracts - #884

Merged
efraespada merged 3 commits into
developfrom
bugfix/855-bugbot-false-positive-treats-jsondocument-decode-object-as-text-and-suggests-a-breaking-cast
Nov 4, 2026
Merged

efraespada merged 3 commits into
developfrom
bugfix/855-bugbot-false-positive-treats-jsondocument-decode-object-as-text-and-suggests-a-breaking-cast

Conversation

@efraespada

@efraespada efraespada commented Nov 4, 2026 •

Copy link
Copy Markdown
Member

Summary

Bugbot can mistake an object-valued decode result for JSON text and suggest an identity cast that breaks a working codec. Trace the actual producer, transformations and reader contract before publishing that claim, and verify any replacement against the same value. Preserve genuine extra encoding and object/text boundary defects.

One shared instruction reaches ordinary/framed source review, existing Python candidate verification and the real-agent benchmark. No consumer-specific suppression, executor, query trigger, permission, history authority, threshold or coverage exclusion is added.

Validation

  • 22 new deterministic regressions bind twelve exact-source Python controls to independently measured values/errors, opposite-verdict scoring, prompt delivery/privacy, identity/history, bounds and previous corpus preservation.
  • Three fixed-input baseline trials: no false positives; two missed defects across 18 positive observations. One individual trial fails recall/F1; the predefined aggregate passes unchanged floors. This does not reproduce the original historical false positive.
  • All three candidate trials pass individually and as the predefined cohort: zero false positives, 18/18 defects and exact anchors; unchanged default quality floors.
  • Full local validation: 900 Jest suites/17,621 passing tests (35 preexisting skips), 267 extension tests at 100% of all metrics, 4/4 modified coverable lines with no misses/partials, and all global/exhaustive budgets. Types, lint, architecture, specs/catalog/docs/workflows and standard 99-file package validation pass. Official current-head native checks/review and Codecov remain required before merge.
  • Fresh exact original-head qualification with its immutable lock and Python 3.14.6: the six requested consumer regressions pass; all four complete-codec variants round-trip unchanged, while the exact harmful replacement made only in memory fails all four. All 517 original files remain intact.
  • Primitive execution, selected consumer tests, reporter-owned whole-backend results, authored runtime controls and actual model predictions remain separate evidence. No historical full-review replay, full consumer acceptance or population accuracy guarantee is claimed.

The SDD, catalog, detection/quality documentation, LF-bound probe and shipped bundles are updated together. See the versioned quality/bugbot/evidence/issue-855.json for frozen identities and every declared measurement.

Closes #855

…-text-and-suggests-a-breaking-cast: ground JSON values in reader contracts
…-text-and-suggests-a-breaking-cast: align owning specification verification date
@docs-page

docs-page Bot commented Nov 4, 2026

Copy link
Copy Markdown

To preview the documentation for this pull request, visit the following URL:

docs.page/vypdev/copilot~884

Documentation is deployed and generated using docs.page

@vypbot
vypbot self-requested a review November 4, 2026 05:32
@vypbot vypbot added the size: XXL Indicates an extremely large task or issue that requires significant time and effort to complete. label Nov 4, 2026
@vypbot vypbot added this to vypdev Nov 4, 2026
@vypbot vypbot moved this to In Progress in vypdev Nov 4, 2026
@vypbot
vypbot changed the base branch from develop to master November 4, 2026 05:33
@vypbot
vypbot changed the base branch from master to develop November 4, 2026 05:33
@vypbot vypbot added the 90% Progress: 90% label Nov 4, 2026
@vypbot

vypbot commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

Bugbot: review complete

Current status: No active findings on a4080b0.

Pull request · Verified commit · Workflow run

@vypbot

vypbot commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

Copilot approval · Human review needed

Status: A protected workflow, policy, or trust-boundary file changed.
Completed: 1/1 observed checks succeeded; Bugbot complete.
Action required: Ask a human reviewer to decide.
Impact: No new approval is confirmed for this revision.
See checks

Technical evidence

Revision: a4080b02dfbb; base: d546c370231c; reason: protected-path.

@codecov-commenter

codecov-commenter commented Nov 4, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.49%. Comparing base (d546c37) to head (a4080b0).

Additional details and impacted files
@@           Coverage Diff            @@
##           develop     #884   +/-   ##
========================================
  Coverage    95.49%   95.49%           
========================================
  Files         1161     1162    +1     
  Lines        38322    38326    +4     
  Branches     11437    11437           
========================================
+ Hits         36594    36598    +4     
  Misses         665      665           
  Partials      1063     1063           
Files with missing lines Coverage Δ
...commit/bugbot/verify_generated_literal_findings.ts 100.00% <100.00%> (ø)
src/prompts/bugbot.ts 100.00% <100.00%> (ø)
src/prompts/bugbot_json_value_evidence.ts 100.00% <100.00%> (ø)
src/tooling/bugbot_benchmark_runner.ts 100.00% <100.00%> (ø)
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

…-text-and-suggests-a-breaking-cast: verify original JSON codec regressions and harmful suggestion
@vypbot vypbot added state:ai-processing 70% Progress: 70% and removed state:ai-processing 90% Progress: 90% labels Nov 4, 2026
@efraespada
efraespada merged commit c0e7a13 into develop Nov 4, 2026
13 checks passed
@github-project-automation github-project-automation Bot moved this from In Progress to Done in vypdev Nov 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

70% Progress: 70% size: XXL Indicates an extremely large task or issue that requires significant time and effort to complete. state:verified

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

🐛🧑‍💻 - Bugbot false positive treats JsonDocument.decode object as text and suggests a breaking cast

3 participants