fix(epcis): keep a truncated export truncated when the fallback adds nothing; close the #3714 review leftovers by PetrefiedThunder · Pull Request #3767 · regengine/RegEngine

PetrefiedThunder · 2026-10-31T05:01:46Z

Summary

Closes the ten CodeRabbit threads left open when #3714 (S3 EPCIS export cap, S7 FDA package evidence context) merged. Each was verified against main before changing anything. Lane: backend (export).

Bug (Major): a truncated export could read as complete. GET /api/v1/epcis/export in services/ingestion/app/epcis/router.py: when the database page was truncated and the product_id filter emptied it, the in-memory fallback branch reset regengine:limit / regengine:truncated to null / false. The reset now happens only when the fallback actually supplies events; otherwise the database page's facts stand.
Fallback ordering. The fallback returned store iteration order while advertising event_timestamp desc; it now sorts by the stored event time, newest first, undated records last.
Savepoint isolation. snapshot_rule_versions in services/shared/rules/versions.py runs each read on its own begin_nested() savepoint when the session supports it, so a failed catalog read cannot leave the caller's transaction in InFailedSqlTransaction and poison the evaluated read and later queries. Fake sessions without begin_nested run the statement directly.
rule_versions_status in the request-workflow package. services/shared/request_workflow/assembly.py stores the snapshot's catalog_status (default unavailable) beside rule_versions in package_contents, so an empty catalog that was read stays distinguishable from one that could not be read; it is hashed with the rest of the package.
Docs. docs/PRODUCT_DESCRIPTION.md names the real cutoff field (regengine:truncated inside regengine:metadata). docs/runbooks/FDA_RECALL_RESPONSE.md: check catalog_status / evaluated_status before treating rule-version evidence as complete; the 250-entry cap behind VALIDATION_ERRORS_*.log and the known_gaps.missing_required_events_truncated / missing_required_events_listed fields; the integrity sentence now says the verifiers prove internal consistency against manifest.json, not the issuer, and points to X-Package-Hash.
Tests. Truncated page emptied by the filter stays truncated with the limit; fallback newest-first with undated last; failing catalog read rolled back on its own savepoint while the evaluated read still runs; rule_versions_status ok vs unavailable; recall package forwards the snapshot helper's result and arguments; proof and admin verifiers reject a same-size edit of evidence_context on the hash, not the size.

Thread map (all on #3714): docs/PRODUCT_DESCRIPTION.md L34 (field name); FDA_RECALL_RESPONSE.md L168 (statuses), L169 (250 cap), L174-175 (integrity wording); epcis/router.py L371-372 (Major, truncation reset) and L397 (ordering); test_fda_export_recall_1342.py L435-437 (stub the snapshot helper); test_proof_verify_router.py L977-978 and the admin verifier test (same-size mutation); assembly.py L161-163 (rule_versions_status); versions.py L75-86 (savepoints).

Type of Change

Test Plan

Unit tests added/updated
Integration tests pass
Manual testing performed
N/A (docs, config, etc.)

Run locally, all green:

python3 -m pytest services/ingestion/tests/test_epcis_router_1342.py services/ingestion/tests/test_epcis_persistence_1342.py services/ingestion/tests/test_fda_export_recall_1342.py services/ingestion/tests/test_fda_export_service_1342.py services/ingestion/tests/test_fda_export_formatters_1342.py services/ingestion/tests/test_proof_verify_router.py services/shared/tests/test_rule_versions_snapshot.py tests/test_request_workflow_unit.py tests/test_e2e_package_manifest.py services/admin/tests/test_verify_audit_pack.py --no-cov

python3 scripts/security/scan_tenant_scope.py   # the pinned EPCIS query text is unchanged
python3 scripts/check_docs_links.py
python3 scripts/check_trust_safety_copy.py

Plus ruff check --select I,E9,F63,F7,F82 on the changed files and git diff --check.

Checklist

My code follows the existing patterns in the codebase
I have not introduced hardcoded secrets or credentials
If this PR adds a test that uses sys.modules[X] = stub, I justified it in the PR description (none)
I have updated relevant documentation (if applicable)
My changes generate no new warnings or lint errors

Related Issues

Relates to #3714 (its ten open review threads) and #3701 (stories S3 and S7).

Generated by Claude Code

…nothing, and close the #3714 review leftovers Ten CodeRabbit threads were left open when #3714 (EPCIS export cap, FDA package evidence context) merged. Each was verified against main first. services/ingestion/app/epcis/router.py (GET /api/v1/epcis/export) - A truncated database page that the product_id filter emptied entered the in-memory fallback branch and had its regengine:limit / regengine:truncated reset to null / false, so an incomplete export read as complete. The reset now happens only when the fallback actually supplies events; when it supplies nothing the database page's facts stand. - The fallback returned store iteration order while advertising "event_timestamp desc". Fallback records are now sorted by their stored event time, newest first, undated records last. services/shared/rules/versions.py - Each snapshot read runs on its own SAVEPOINT (begin_nested) when the session supports it, so a failed catalog read cannot leave the caller's transaction in InFailedSqlTransaction and poison the evaluated read and every later query on that session. Fake sessions without begin_nested run the statement directly, as before. services/shared/request_workflow/assembly.py - package_contents carries rule_versions_status (the snapshot's catalog_status, "unavailable" when absent) beside rule_versions, so an empty catalog that was read stays distinguishable from one that could not be read; the status is hashed with the rest of the package. Docs - docs/PRODUCT_DESCRIPTION.md names the real cutoff field (regengine:truncated inside regengine:metadata). - docs/runbooks/FDA_RECALL_RESPONSE.md tells reviewers to check catalog_status / evaluated_status before treating rule-version evidence as complete, documents the 250-entry cap behind VALIDATION_ERRORS and the known_gaps fields that say when it bit, and qualifies the package integrity sentence: the verifiers prove internal consistency against manifest.json, not the issuer; compare X-Package-Hash to tie a ZIP to the original export. Tests - epcis router: truncated page emptied by the filter stays truncated with the limit; fallback events come back newest first with undated last. - rule_versions: a failing catalog read is rolled back on its own savepoint and the evaluated read still runs on a second one. - request workflow: rule_versions_status is "ok" for an empty readable catalog and "unavailable" when the catalog read fails. - recall package: the snapshot helper receives (session, tenant, events) and its return value is forwarded as rule_versions. - proof verifier and admin verifier: a same-size edit inside evidence_context is rejected by the hash check, not the size check. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GyhDYG3ddQvVKyu2qtYfCj

coderabbitai · 2026-10-31T05:01:56Z

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

github-actions · 2026-10-31T05:01:56Z

PR Blocker Checklist

Head SHA: ea0ce2f21f67d9bb6507a46ab07eabeb417d7e15

Status: Ready

All review threads resolved
No reviews requesting changes
No blocked review engines

…n-8fijxh-s3-s7-followup

The Security Audit, Production Readiness Gate and per-service pip-audit jobs went red on this PR after advisories for pypdf 6.21.2, pyasn1 0.6.4 and follow-redirects 1.16.0-1.16.1 were published. Main is red on the same jobs. This ports #3773 byte for byte (pypdf 6.21.4, pyasn1 0.6.5, npm overrides follow-redirects >=1.16.2 and minimatch >=11.0.1). It no-ops once main carries #3773. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GyhDYG3ddQvVKyu2qtYfCj

vercel · 2026-10-31T13:03:17Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
regengine	Ready	Preview	Oct 31, 2026 1:05pm UTC

PetrefiedThunder · 2026-10-31T13:03:48Z

Status 14:12 UTC: ea0ce2f is green on every required check (PR Gate: Backend, PR Gate: Frontend, PR Gate: Workflow Contract, CI Status Gate). The only red is Engine Review Pass, which needs the founder's REVIEW: PASS <engine> ea0ce2f… marker.

The first PR Gate: Frontend attempt on ea0ce2f timed out after 12 minutes waiting for a queued Frontend CI run. pr-gate-frontend-rerun.yml re-ran it automatically after Frontend CI passed, and attempt 2 succeeded.

Original report on 961487f: red on dependency audits, not on this PR's code. Fix ported in ea0ce2f.

Failing checks on 961487f:

Production Readiness Gate and Security Audit (frontend): npm audit --omit=dev --audit-level=high fails on follow-redirects 1.16.0–1.16.1 (GHSA-mrwp-f3q8-3g9p, high).
Security Audit admin / compliance / graph / ingestion / nlp / scheduler: pip-audit -r requirements.lock fails on pyasn1 0.6.4 (four CVEs, fixed in 0.6.5) and pypdf 6.21.2 (seven CVEs, fixed in 6.21.4).
Frontend CI Status Gate, Backend CI Status Gate, PR Gate: Frontend, PR Gate: Backend, CI Status Gate: downstream of the above.

Why it is not this PR's: this PR changes no dependency file. The advisories were published after the last green main run. Main 1d9c995 carries the same pins (pyasn1==0.6.4, pypdf==6.21.2, follow-redirects 1.16.1), and npm audit --audit-level=high --omit=dev on main's lockfile exits 1 locally.

What I pushed:

9f6ea0e merges main 1d9c995 into the branch (clean merge).
ea0ce2f ports fix(deps): bump pypdf to 6.21.4 for factory-local pip-audit #3773 byte for byte: pypdf 6.21.4, pyasn1 0.6.5, npm overrides follow-redirects >=1.16.2 and minimatch >=11.0.1. It no-ops once fix(deps): bump pypdf to 6.21.4 for factory-local pip-audit #3773 merges.

Local checks on ea0ce2f:

pip-audit -r requirements.lock with the CI ignore list: no known vulnerabilities.
npm audit --audit-level=high --omit=dev: exit 0 (4 moderate remain, same as fix(frontend): patch follow-redirects and minimatch for npm audit #3770 and fix(deps): bump pypdf to 6.21.4 for factory-local pip-audit #3773).
This PR's ingestion, shared and admin test subsets and scan_tenant_scope.py: pass.

Generated by Claude Code

github-actions · 2026-10-31T13:48:33Z

Vercel preview: https://regengine-6jg40nlr8-petrefiedthunders-projects.vercel.app

Access is via Vercel login (Deployment Protection).

github-actions · 2026-10-31T13:49:57Z

✅ Bundle Size Report

Metric	Size
Total JS	5581.9KB (5.45MB)
Total CSS	131.9KB
Limit	10.0MB

Bundle is within the 10.0MB limit.

This was referenced Oct 31, 2026

feat(export): state the EPCIS export cap and carry rule versions and known gaps in the FDA package #3714

Merged

feat(evidence): enforce the traceability_events mutation policy and derive waiver approvers from the principal #3717

Open

github-actions Bot added backend docs labels Oct 31, 2026

PetrefiedThunder and others added 2 commits October 31, 2026 13:01

Merge remote-tracking branch 'origin/main' into claude/gracious-ediso…

9f6ea0e

…n-8fijxh-s3-s7-followup

vercel Bot deployed to Preview October 31, 2026 13:05

View deployment

github-actions Bot added dependencies Pull requests that update a dependency file frontend labels Oct 31, 2026

Navigation Menu

Repository navigation

fix(epcis): keep a truncated export truncated when the fallback adds nothing; close the #3714 review leftovers - #3767

fix(epcis): keep a truncated export truncated when the fallback adds nothing; close the #3714 review leftovers#3767
PetrefiedThunder wants to merge 3 commits into
mainfrom
claude/gracious-edison-8fijxh-s3-s7-followup

PetrefiedThunder commented Oct 31, 2026

Uh oh!

coderabbitai Bot commented Oct 31, 2026

Draft PR not reviewed

Uh oh!

github-actions Bot commented Oct 31, 2026 •

edited

Loading

Uh oh!

vercel Bot commented Oct 31, 2026 •

edited

Loading

Uh oh!

PetrefiedThunder commented Oct 31, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Oct 31, 2026

Uh oh!

github-actions Bot commented Oct 31, 2026

Uh oh!

This branch was successfully deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

PetrefiedThunder commented Oct 31, 2026

Summary

Type of Change

Test Plan

Checklist

Related Issues

Uh oh!

coderabbitai Bot commented Oct 31, 2026

Draft PR not reviewed

Uh oh!

github-actions Bot commented Oct 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Blocker Checklist

Status: Ready

Uh oh!

vercel Bot commented Oct 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

PetrefiedThunder commented Oct 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Oct 31, 2026

Uh oh!

github-actions Bot commented Oct 31, 2026

✅ Bundle Size Report

Uh oh!

This branch was successfully deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

github-actions Bot commented Oct 31, 2026 •

edited

Loading

vercel Bot commented Oct 31, 2026 •

edited

Loading

PetrefiedThunder commented Oct 31, 2026 •

edited

Loading