From 453598f960023f0ef1c77c6b6cf960983c60baaf Mon Sep 17 00:00:00 2001 From: Cursor Agent Date: Mon, 26 Oct 2026 16:12:21 +0000 Subject: [PATCH] docs: record EDI event time, supplier quality, and export limits The intake page still said a missing or old EDI date became the ingest clock. Supplier data quality, gap time to fix, spreadsheet Event Time, and the trace and export caps had no engineering page. Co-authored-by: christophersellers --- README.md | 8 +- docs/README.md | 3 + docs/engineering/SUPPLIER_DATA_QUALITY.md | 159 +++++++++++++++++++ docs/engineering/TRACEABILITY_FILE_INTAKE.md | 97 +++++++++-- docs/engineering/TRACE_AND_EXPORT_LIMITS.md | 139 ++++++++++++++++ 5 files changed, 388 insertions(+), 18 deletions(-) create mode 100644 docs/engineering/SUPPLIER_DATA_QUALITY.md create mode 100644 docs/engineering/TRACE_AND_EXPORT_LIMITS.md diff --git a/README.md b/README.md index 77e422977..fe1e0e8a5 100644 --- a/README.md +++ b/README.md @@ -49,11 +49,13 @@ Implemented capabilities in the current tree. Route availability is controlled b | CTE persistence, hash-chain storage, and verification (legacy and canonical stores; dual-write retirement mid-cutover) | [services/shared/cte_persistence/core.py](services/shared/cte_persistence/core.py), [services/shared/canonical_persistence/writer.py](services/shared/canonical_persistence/writer.py), [docs/persistence/dual-write-cutover.md](docs/persistence/dual-write-cutover.md) | | Compliance score reads from stored FSMA CTE data | [services/ingestion/app/compliance_score.py](services/ingestion/app/compliance_score.py) | | FDA-oriented CSV and evidence-bundle export routes | [services/ingestion/app/fda_export/router.py](services/ingestion/app/fda_export/router.py) | +| Compliance sortable spreadsheet: zone-aware Event Time written in UTC with a `Z` suffix, empty window 404, over 50,000 events 413 | [services/compliance/app/fsma_spreadsheet.py](services/compliance/app/fsma_spreadsheet.py), [docs/engineering/TRACE_AND_EXPORT_LIMITS.md](docs/engineering/TRACE_AND_EXPORT_LIMITS.md) | | Request-response cases, scoped evidence package assembly, and auditor workflows | [services/ingestion/app/request_workflow_router.py](services/ingestion/app/request_workflow_router.py), [services/ingestion/app/auditor_router.py](services/ingestion/app/auditor_router.py) | | Recall readiness assessments, gap checklists, and recall reports | [services/ingestion/app/readiness_router.py](services/ingestion/app/readiness_router.py), [services/ingestion/app/recall_report.py](services/ingestion/app/recall_report.py) | -| EPCIS JSON-LD export and broader recall simulations (experimental routers; excluded from the production allowlist) | [services/ingestion/app/epcis_export.py](services/ingestion/app/epcis_export.py), [services/ingestion/app/recall_simulations.py](services/ingestion/app/recall_simulations.py) | +| EPCIS JSON-LD export and broader recall simulations (experimental routers; excluded from the production allowlist). A storage error on the EPCIS/FDA export is HTTP 503, and more than 50,000 events is HTTP 413 | [services/ingestion/app/epcis_export.py](services/ingestion/app/epcis_export.py), [services/ingestion/app/recall_simulations.py](services/ingestion/app/recall_simulations.py), [docs/engineering/TRACE_AND_EXPORT_LIMITS.md](docs/engineering/TRACE_AND_EXPORT_LIMITS.md) | | Subscription-gated access and persisted usage metering; Stripe usage reporting requires an operator call | [services/ingestion/app/subscription_gate.py](services/ingestion/app/subscription_gate.py), [DESIGN_metering-billing.md](DESIGN_metering-billing.md) | | Tenant/admin/user/API-key management, including a sysadmin rename of an organization and the tenant row that shares its id | [services/admin/app/](services/admin/app/), [docs/engineering/ADMIN_ORGANIZATIONS.md](docs/engineering/ADMIN_ORGANIZATIONS.md) | +| Per-facility supplier data quality and data-gap time to fix on `/dashboard/suppliers` | [services/admin/app/supplier_compliance_routes.py](services/admin/app/supplier_compliance_routes.py), [docs/engineering/SUPPLIER_DATA_QUALITY.md](docs/engineering/SUPPLIER_DATA_QUALITY.md) | | Persisted admin tenant provisioning | [services/admin/app/routes.py](services/admin/app/routes.py) | | Tamper-evident admin audit log hashing | [services/admin/app/audit.py](services/admin/app/audit.py) | | Audit-chain verification for admin audit exports | [services/admin/app/audit_integrity.py](services/admin/app/audit_integrity.py) | @@ -357,7 +359,9 @@ These are current repo facts, not speculation: - [docs/PRODUCT_DESCRIPTION.md](docs/PRODUCT_DESCRIPTION.md) — canonical positioning and capability availability. - [docs/engineering/JEV_HEADER_MAPPING.md](docs/engineering/JEV_HEADER_MAPPING.md) — optional CSV suggestions, review policy, privacy defaults, and evaluation tools. - [docs/engineering/DOCUMENT_EXTRACT.md](docs/engineering/DOCUMENT_EXTRACT.md) — optional document data extraction (LlamaIndex Extract v2.5 on supplier PDFs): request flow, failure modes, data handling, configuration, command line. -- [docs/engineering/TRACEABILITY_FILE_INTAKE.md](docs/engineering/TRACEABILITY_FILE_INTAKE.md) — Excel `.xlsx` on the CSV ingest route, the dashboard EPCIS 2.0 upload, and X12 944/945 warehouse advices: routes, field mapping, and rejection rules. +- [docs/engineering/TRACEABILITY_FILE_INTAKE.md](docs/engineering/TRACEABILITY_FILE_INTAKE.md) — Excel `.xlsx` on the CSV ingest route, the dashboard EPCIS 2.0 upload, and X12 intake: routes, field mapping, EDI event time, TLC shapes, and rejection rules. +- [docs/engineering/SUPPLIER_DATA_QUALITY.md](docs/engineering/SUPPLIER_DATA_QUALITY.md) — per-facility supplier data quality, agreed sending hours, and data-gap time to fix. +- [docs/engineering/TRACE_AND_EXPORT_LIMITS.md](docs/engineering/TRACE_AND_EXPORT_LIMITS.md) — FDA spreadsheet Event Time, export caps, and graph trace truncation. The graph and `/api/v1/export` routers are off the production allowlist. - [frontend/README.md](frontend/README.md) — current UI surfaces and consolidated-backend development setup. - [2026-09-04 GitHub issue review and rollout](docs/audits/GITHUB_ISSUE_REVIEW_2026_09_04.md) — fixes, validation, and outstanding provisioning. diff --git a/docs/README.md b/docs/README.md index 55ba4f7cc..beb6d2031 100644 --- a/docs/README.md +++ b/docs/README.md @@ -42,6 +42,9 @@ - [Schema Change Policy](engineering/SCHEMA_CHANGE_POLICY.md) - [Content Ingestion](engineering/CONTENT_INGESTION.md) - [Audit Events](engineering/AUDIT_EVENTS.md) +- [Traceability file intake](engineering/TRACEABILITY_FILE_INTAKE.md) +- [Supplier data quality](engineering/SUPPLIER_DATA_QUALITY.md) +- [Trace and export limits](engineering/TRACE_AND_EXPORT_LIMITS.md) - [Authentication](engineering/Authentication.md) - [NFR Requirements](engineering/NFR_REQUIREMENTS.md) - [Testing Requirements](engineering/TESTING_REQUIREMENTS.md) diff --git a/docs/engineering/SUPPLIER_DATA_QUALITY.md b/docs/engineering/SUPPLIER_DATA_QUALITY.md new file mode 100644 index 000000000..927cae23f --- /dev/null +++ b/docs/engineering/SUPPLIER_DATA_QUALITY.md @@ -0,0 +1,159 @@ +# Supplier data quality and gap time to fix + +The suppliers dashboard (`/dashboard/suppliers`) shows two readouts for +the facilities the signed-in supplier user owns. Both routes live on +the admin supplier router (`services/admin/app/supplier_compliance_routes.py`), +mounted at `/v1` on the consolidated app. `admin` is on the production +router allowlist. + +A supplier here is one `supplier_facilities` row. `supplier_key` is that +row's id. The caller must have a tenant context. A missing tenant is +HTTP 400 `Tenant context required`. Another user's facility is not in +the summary. The records route returns HTTP 404 for a facility the +caller does not own, and HTTP 400 for an id that is not a UUID. + +## Data quality + +`GET /v1/supplier/data-quality?window_days=30` + +`window_days` defaults to 30 and must be from 1 to 365. The window +opens at `now` minus that many days. The response repeats +`window_days`, `min_records` (20), `late_cte_label`, and +`evaluated_at`, then one entry per owned facility. + +Four metrics stay separate. There is no combined score. A share is +present only when the denominator is at least 20. Below that, +`status` is `not_enough_data`, `share` is null, and the numerator and +denominator are still returned. + +| `key` | What the share counts | Drilldown | +| --- | --- | --- | +| `missing_kde` | CTE records (`supplier_cte_events`) received in the window that lack a required KDE in `kde_data` | yes | +| `unlinked_lot` | Lot codes active in the window that do not link upstream | yes | +| `late_cte` | Records received (`created_at`) more than the facility's agreed sending hours after `event_time` | yes | +| `epcis_failed` | EPCIS documents received in the window whose latest receipt is `rejected` | no | + +`missing_kde` uses the webhook required-KDE table, minus the fields a +supplier record already stores as columns: the lot code, the event +time, and the recording facility's own location. The denominator is +every CTE record received in the window. An unknown CTE type has an +empty required list, so that record stays in the denominator and adds +nothing to the numerator. + +`unlinked_lot` uses one denominator per distinct lot, not per record. + +A lot counts as linked when it is the output of +`fsma.transformation_links`, when one of the caller's records for it +carries a linkage KDE (`tlc_source_reference`, `tlc_source_gln`, +`immediate_previous_source`, `input_lot_codes`, +`input_traceability_lot_codes`, `input_tlcs`, +`harvester_business_name`), or when the chain starts at that facility +(`harvesting`, or `first_receiver` including the +`first_land_based_receiving` alias). Linkage looks at all of the +caller's records for the lot, including records older than the window. +A lot is active only when the caller sent a record for it in the +window. Another supplier's record does not activate it. + +`late_cte` is the product's measure against the agreed sending time. +The response label says it is not a regulatory requirement. The +denominator is records in the window that have both `event_time` and +`created_at`. A record is late when `created_at - event_time` is +greater than the agreed hours. With no agreed hours, status is +`not_enough_data`, the counts are null, and `reason` says the sending +time is unset. + +`epcis_failed` is the tenant's count, repeated on every facility. +An EPCIS upload has no facility. Retries of the same request body count +once: the metric uses the latest receipt per document SHA-256 in +`fsma.epcis_document_receipts`. `reason` says the count is account-wide. +When that table cannot be read, status is `unavailable` and the other +metrics still return. Writing a receipt during ingest is fail-open and +does not change the ingest response. + +### Records behind a metric + +```bash +curl "$BASE/v1/supplier/data-quality/$FACILITY_ID/records?metric=missing_kde&window_days=30&limit=50&offset=0" \ + -H "Cookie: __Host-regengine_session=$SESSION" +``` + +`metric` is `missing_kde`, `unlinked_lot`, or `late_cte`. `limit` +defaults to 50, maximum 200. `offset` defaults to 0. `total` is the +full flagged set; `items` is the page. `late_cte` for a facility with +no agreed hours is HTTP 409 `No agreed sending time is set for this +facility`. + +`missing_kde` items include `record_id`, `lot_code`, `cte_type`, and +`missing_kdes`. `late_cte` items include `event_time` and `received_at`. +`unlinked_lot` items include `lot_id` and `lot_code`. + +The dashboard loads this from +`frontend/src/app/dashboard/suppliers/data-quality.tsx` in pages of 25. + +### Agreed sending hours + +`PUT /v1/supplier/admin/facilities/{facility_id}/agreed-sending-hours` + +Body: `{ "agreed_sending_hours": 24 }` or `{ "agreed_sending_hours": null }` +to clear it. The value is a strict integer greater than 0, up to the +Postgres integer maximum. A string, float, or boolean is HTTP 422. + +The caller must be an Owner or Admin of the bound tenant +(`tenant.admin`). Anyone else is HTTP 403. A missing tenant context is +HTTP 403 `E_NO_TENANT_CONTEXT`. A facility outside the tenant is HTTP +404. The supplier data-quality view can read the value and cannot +change it. The dashboard labels it as set by the customer. + +## Gap time to fix + +`GET /v1/supplier/gap-time-to-fix` + +One row per owned facility: + +| Field | Meaning | +| --- | --- | +| `open_gap_count` | Open rows in `public.data_gap_events` for that facility | +| `median_time_to_fix_seconds` | Median of `closed_at - opened_at` for closed gaps. Null when nothing has been fixed. | +| `oldest_open_gap_opened_at` | Earliest `opened_at` still open | +| `past_internal_target` | True when any open gap has been open for more than 24 hours | +| `at_risk` / `at_risk_gap_count` | Gaps whose lot is on an unanswered request case and that are still open at that case's `response_due_at` | +| `has_data` | True when the facility has an open gap or a closed one | +| `internal_target_seconds` | 86400. This target is internal. It is not the FDA response deadline. | + +`at_risk` joins `fsma.request_cases` on the lot. A case counts while +`package_status` is outside `submitted` and `amended`. A gap counts +when it opened at or before `response_due_at` and had not closed by +then. + +### How a gap opens and closes + +`services/shared/data_gap_tracking.py` records two kinds: +`missing_kde` (one gap per missing KDE name) and +`cte_validation_failed` (keyed on the rule id). A failed record opens +gaps and leaves existing gaps open. A record that passes validation +closes every open gap for that lot and CTE. + +A close stores `closing_record_id` and `closing_record_hash`. The hash +must be a lowercase SHA-256 hex digest of the persisted record. A pass +without that receipt does not close the gap. + +The writer runs after the ingest transaction commits. It catches its +own errors, logs them, and returns. A gap-log failure does not roll +back an accepted event. An open insert is idempotent on the partial +unique index of open rows. A lot with no supplier facility is stored +under `supplier_key` `unassigned`. That key is not a facility id, so +the per-facility summary does not list it. + +The dashboard panel is +`frontend/src/app/dashboard/suppliers/gap-time-to-fix.tsx`. + +## Tests + +```bash +python3 -m pytest services/admin/tests/test_supplier_data_quality_routes.py -q --no-cov +python3 -m pytest services/shared/tests/test_supplier_data_quality.py \ + services/shared/tests/test_data_gap_tracking.py \ + services/shared/tests/test_epcis_receipts.py -q --no-cov +cd frontend && npm run test:run -- src/__tests__/components/supplier-data-quality.test.tsx \ + src/__tests__/components/gap-time-to-fix.test.tsx +``` diff --git a/docs/engineering/TRACEABILITY_FILE_INTAKE.md b/docs/engineering/TRACEABILITY_FILE_INTAKE.md index 26db7d97c..6f47b740f 100644 --- a/docs/engineering/TRACEABILITY_FILE_INTAKE.md +++ b/docs/engineering/TRACEABILITY_FILE_INTAKE.md @@ -17,6 +17,70 @@ assertion and must match that principal. `POST /api/v1/ingest/edi` accepts a single 856 transaction set. 861, 944, 945, 850, and 810 belong on `/document`. +## EDI event time + +Both EDI routes store the source date as the event time. +`_ship_timestamp_and_date` in +`services/ingestion/app/edi_ingestion/utils.py` parses CCYYMMDD or YYMMDD +into a UTC timestamp (`YYYY-MM-DDTHH:MM:SS+00:00`). A date with no time +is midnight UTC. A YYMMDD year uses `EDI_CENTURY_PIVOT` (default 50): +YY below the pivot is 20YY, and YY at or above it is 19YY. + +The preflight runs after the partner allowlist and before the ISA13 +control number is reserved. A date rejection does not reserve that +control number, so a corrected resend is not refused as a 409 replay. +Each rejection is written to the EDI rejection log with +`needs_review: true`, and the HTTP detail includes `rejection_id`. + +| Source date | Status | `error` | +| --- | --- | --- | +| Missing or blank | 400 | `missing_edi_date` | +| Not CCYYMMDD or YYMMDD | 400 | `invalid_edi_date` | +| 856 time (BSN04) has at least four digits and the hour or minute is impossible | 400 | `invalid_edi_time` | +| More than 24 hours after the current UTC time | 400 | `edi_date_in_future` | +| A parseable date at or before that cutoff, including a date older than 90 days | accepted | timestamp is that source date | + +856 is the only set that reads a time. The first four digits of BSN04 +are the hour and minute. Fewer than four digits leaves the time at +midnight UTC. Seconds in a longer BSN04 are ignored. 944 and 945 pass +no time into the parser. A date more than 24 hours ahead is rejected. +A date exactly 24 hours ahead is kept. + +| Set | Date element | Time | +| --- | --- | --- | +| 856 | BSN03 | BSN04 | +| 861 | BRA02 | none | +| 944 | W1702. A G62 date is used only when W1702 is empty. | none | +| 945 | W0603. A G62 date is used only when W0603 is empty. | none | +| 850 | BEG05 | none | +| 810 | BIG01 | none | + +On `POST /api/v1/ingest/edi` a missing BSN date is this 400. The later +856 placeholder check (`edi_856_missing_required_kdes`) still rejects a +missing ship-from, ship-to, quantity, unit, or product, and it does not +get a chance to report the date. + +### Accepted TLC shapes + +EDI persistence validates the event TLC with `FSMAEvent` in +`services/shared/schemas.py`. A failure is HTTP 422 +`fsma_validation_failed`. The validator accepts the submitted string +when it is one of these: + +- Fourteen digits followed by a lot suffix of letters, digits, `-`, or + `.`. The whole value is at least 15 characters. +- An LGTIN class URN: `urn:epc:class:lgtin:`, two digit groups whose + lengths add up to 13, a dot, then the lot. +- A GS1 Digital Link: `http://` or `https://`, a host, optional extra + path segments, `/01/` and 14 digits, `/10/` and the lot, then an + optional further path and query. + +On the URN and Digital Link forms the lot, after percent-decoding, is +1 to 20 characters from GS1 character set 82 (digits, both letter +cases, and the twenty specials in that set; no space). A character the +carrier requires to be percent-encoded, or a `%` that is not a `%HH` +escape, fails the match. The stored TLC is the submitted string. + ## Excel `.xlsx` on `POST /api/v1/ingest/csv` `services/ingestion/app/shared/xlsx_rows.py` turns the workbook into the CSV @@ -149,7 +213,9 @@ line, or to the draft when no line is open yet. The line lot is the event TLC, copied verbatim. A GTIN or UPC on the line does not prefix it. A line with no lot uses the required form field `traceability_lot_code`. A lot that fails TLC format is HTTP 422 -`fsma_validation_failed`, including a one-character lot. Two lines that +`fsma_validation_failed`, including a one-character lot. The shapes +`FSMAEvent` accepts are listed under +[Accepted TLC shapes](#accepted-tlc-shapes). Two lines that share a lot both persist. `edi_line_index` on the KDEs separates them. N104 is stored as the location id whether or not N103 is `UL`. The event @@ -157,10 +223,8 @@ N104 is stored as the location id whether or not N103 is `UL`. The event check digit. The 856 event location is the ship-from GLN. The 861 event location is the receiving-location GLN. -A missing date uses the ingest time. A date that is present and is not -CCYYMMDD or YYMMDD is HTTP 400 `invalid_edi_date`. A parsed date older -than 90 days, or more than 24 hours in the future, keeps that calendar -date on the event and snaps the ingest timestamp to now. +The event time is the BSN date (and BSN04 time when present). See +[EDI event time](#edi-event-time). ### `POST /api/v1/ingest/edi` (856 only) @@ -170,11 +234,13 @@ and 861, `/document` emits one CTE per item line. Required segments are ISA, GS, ST, BSN, HL, SE, GE, and IEA. Each line must carry a positive quantity, a recognized unit, and a product id or -description. The set must carry a ship-from name, a ship-to name, and a -BSN date. A miss is HTTP 422 `edi_856_missing_required_kdes`, the -rejection is logged, and nothing is persisted. The route does not fill -quantity `1.0`, unit `units`, product `EDI 856 Shipment`, an unknown -location, or today's date. +description. The set must carry a ship-from name and a ship-to name. A +miss on those fields is HTTP 422 `edi_856_missing_required_kdes`, the +rejection is logged, and nothing is persisted. A missing BSN date is +HTTP 400 `missing_edi_date` from the event-time preflight, which runs +before that placeholder check. The route does not fill quantity `1.0`, +unit `units`, product `EDI 856 Shipment`, an unknown location, or +today's date. `product_description`, `quantity_override`, `unit_of_measure_override`, `ship_from_location_override`, and `ship_to_location_override` apply to @@ -287,11 +353,8 @@ assigned id and the GLN stays empty. if none of those appear, the depositor (N1*DE) is the ship-from. - 945 ship-from is N1*SF when present, otherwise the warehouse (N1*WH). - The event date prefers the header date (W1702 or W0603). A G62 date is - used only when that header date is empty. A missing date uses the ingest - time. A date that is present and is not CCYYMMDD or YYMMDD is HTTP 400 - `invalid_edi_date`. A parsed date older than 90 days, or more than 24 - hours in the future, keeps that calendar date and snaps the ingest - timestamp to now. + used only when that header date is empty. Parsing, rejection codes, and + the stored UTC timestamp are in [EDI event time](#edi-event-time). - 944 reference document number is the warehouse receipt number, else the shipment id. 945 uses the shipment id, else the depositor order number. N9 fills a blank reference (`BM`, `SI`, `PO`, `PK`, `CN`, in that order). @@ -323,7 +386,9 @@ assigned id and the GLN stays empty. python3 -m pytest services/ingestion/tests/test_csv_xlsx_upload.py -q --no-cov python3 -m pytest services/ingestion/tests/test_edi_warehouse_944_945.py \ services/ingestion/tests/test_edi_golden.py \ - services/ingestion/tests/test_edi_wire_format_2974_2976_2978.py -q --no-cov + services/ingestion/tests/test_edi_wire_format_2974_2976_2978.py \ + services/ingestion/tests/test_edi_preserve_event_time.py \ + services/ingestion/tests/test_edi_date_reject_resend.py -q --no-cov cd frontend && npm run test:run -- src/__tests__/lib/epcis-upload.test.ts \ src/__tests__/pages/ingest-epcis.test.tsx ``` diff --git a/docs/engineering/TRACE_AND_EXPORT_LIMITS.md b/docs/engineering/TRACE_AND_EXPORT_LIMITS.md new file mode 100644 index 000000000..3d9a8e965 --- /dev/null +++ b/docs/engineering/TRACE_AND_EXPORT_LIMITS.md @@ -0,0 +1,139 @@ +# Trace and export limits + +What a caller receives when a trace or an FDA-oriented export is larger +than the query cap, and how the compliance spreadsheet writes Event +Time. Confirm the router is enabled before calling it. +`server/router_policy.py` is the production allowlist. + +## FDA sortable spreadsheet + +`GET /v1/fsma/audit/spreadsheet` is on the `fsma_compliance` router, +which is on the production allowlist. Implementation: +`services/compliance/app/routes.py` and +`services/compliance/app/fsma_spreadsheet.py`. + +The caller is an API key. Tenant scope is the key's tenant. An +`X-Tenant-ID` or `X-RegEngine-Tenant-ID` header that names a different +tenant is HTTP 409 `E_TENANT_MISMATCH`. A key with no tenant is HTTP +403. + +`start_date` and `end_date` are required `YYYY-MM-DD` values. +`validate_export_window` requires `end_date` on or after `start_date` +and a span of at most 90 days (`(end - start).days` greater than 90 is +HTTP 400). Optional `tlc` filters one lot code. Optional +`requesting_entity` must match `^[A-Za-z0-9 .,&'\-]{0,120}$`. + +| Outcome | Status | +| --- | --- | +| At least one event | 200, CSV attachment | +| No events in the window | 404 | +| More than 50,000 events | 413. Narrow the dates or the TLC. | +| Event storage cannot be reached | 503 | +| An event date or event shape cannot be rendered | 400. The export stops. | + +### Event Time + +Zone-aware timestamps are converted to UTC before they are written. +`Event Date` is the UTC date, which can be the next or previous +calendar day. `Event Time` is `HH:MM:SSZ`. Rows sort by that date, then +that time, so two events from different offsets sort in real order. + +A timestamp with no zone is written as `HH:MM:SS` with no suffix. The +spreadsheet does not label it UTC. A date-only value becomes midnight +with no zone (`00:00:00`). + +When `kdes.event_time` is present and not blank, that string is the +Event Time cell, including a value with no zone. The date still comes +from the parsed event timestamp. + +A malformed `event_date` on this path raises `FSMATimestampError` and +the route returns HTTP 400. It does not truncate the raw string into +the date column. + +```bash +python3 -m pytest services/compliance/tests/test_fda_event_time_utc_sel519.py \ + services/compliance/tests/test_fda_spreadsheet.py -q --no-cov +``` + +## EPCIS and FDA export on the experimental export router + +`POST /api/v1/export/epcis` and `POST /api/v1/export/fda` are +`services/ingestion/app/epcis_export.py`. The owner is `export`, which +is in `EXPERIMENTAL_ROUTERS` and is off the production allowlist. + +Both routes read `fsma.cte_events` for the authenticated tenant +context. The JSON `tenant_id` field does not select a different tenant. +`lot_code`, `date_from`, and `date_to` filter the query. `date_from` is +inclusive and `date_to` is exclusive (`event_timestamp < date_to`). + +| Outcome | Status | Body | +| --- | --- | --- | +| The query succeeds and the row count is within 50,000 | 200 | EPCIS document, or the FDA CSV from this module | +| The query succeeds and returns no rows | 200 | EPCIS metadata says `No data yet`. The FDA route returns the header row, `X-RegEngine-Events-Count: 0`, and `X-RegEngine-Message: No data yet` | +| The database read fails | 503 | `Event storage unavailable; export not generated` | +| The query returns more than 50,000 rows | 413 | Names the 50,000-event cap and asks for a narrower date range or lot code | + +The cap query fetches 50,001 rows. Crossing the cap refuses the export. +It does not return the newest 50,000 and omit the rest. The compliance +spreadsheet above uses the same cap and, unlike this router, returns +404 when the window is empty. + +The FDA CSV on this router formats `event_timestamp` with +`%Y-%m-%d` and `%H:%M:%SZ`. That is a separate writer from the +compliance spreadsheet in the previous section. + +```bash +python3 -m pytest services/ingestion/tests/test_epcis_export_1342.py -q --no-cov +``` + +## Graph traces and gap queries + +The graph router is in `EXPERIMENTAL_ROUTERS`. On the consolidated app +its prefix is `/api/v1`. + +### Forward and backward trace + +`GET /api/v1/fsma/traceability/trace/forward/{tlc}` and +`GET /api/v1/fsma/traceability/trace/backward/{tlc}`. + +Each capped query reads `TRACE_ROW_LIMIT + 1` rows. The limit is 5000 +(`services/graph/app/fsma/tracer.py`). + +| Rows the capped query returned | Response | +| --- | --- | +| 5000 or fewer | `truncated` is false and `truncation_limit` is null | +| 5001 | `truncated` is true, `truncation_limit` is 5000, and the extra row is dropped | + +The HTTP status stays 200. The body is a partial trace: the events that +fit, plus `truncated: true`. A recall response that ignores the flag +treats a cut-off chain as the whole chain. + +The association query that rebuilds lot, event, and facility rows uses +the same cap. That query alone can set `truncated` even when the event +list is shorter than 5000. + +### Gap, chain, and orphan reads + +`find_gaps`, `find_broken_chains`, and `find_orphaned_lots` read Neo4j +in keyset pages of 2000 (`_PAGE_SIZE` in +`services/graph/app/fsma/chain_integrity.py`). Each page starts after +the last key already returned. The functions yield every matching row +into one result. A result larger than 2000 is the full set. + +HTTP routes that call them: + +- `GET /api/v1/fsma/compliance/gaps` and + `GET /api/v1/fsma/compliance/export/gaps` call `find_gaps` +- `GET /api/v1/fsma/compliance/gaps/orphans` calls `find_orphaned_lots` + +`find_broken_chains` uses the same pager. `find_all_gaps` calls it +together with `find_gaps`. The compliance router has no separate +broken-chain route. The responses above have no page cursor. The caller +receives the assembled list. A missing tenant id raises `ValueError` +inside these functions, so a null tenant does not become a cross-tenant +read. + +```bash +python3 -m pytest services/graph/tests/test_trace_truncated_flag_sel515.py \ + services/graph/tests/test_graph_row_paging_r1.py -q --no-cov +```