[Privacy/Operability] Govern provider-side batch file expiration and deletion

[seonghobae]

PARTIAL — protected provider-file lifecycle primitives

Protected main contains the reviewed remote-file lifecycle primitives in pg_llm_batch/batch_api_client.py, with OpenTelemetry parity and retention tests/doctoring:

• BatchAPIClient.upload_jsonl(..., expires_after_seconds=...) supports bounded input-file expiry with local fail-before-provider validation;
• BatchAPIClient.create_batch_job(..., output_expires_after_seconds=...) supports bounded generated output/error-file expiry;
• BatchAPIClient.delete_file() is an explicit caller-authorized provider deletion primitive, not inferred cleanup authority; and
• OpenTelemetryBatchAPIClient preserves those controls without adding content-bearing telemetry authority.

These are implemented primitives, not end-to-end retention/erasure. They do not establish provider capability negotiation, automatic cleanup, already-deleted/not-found reconciliation, durable cleanup audit, local PostgreSQL retention/erasure, backup deletion or legal-erasure guarantees.

Residual bounded target

Complete the remaining remote content-lifecycle policy test-first:

• establish an explicit provider capability/compatibility contract for input/output expiry rather than assuming every OpenAI-compatible gateway accepts optional retention fields;
• define reviewed already-deleted/not-found reconciliation semantics; uncertain transport failure remains reconciliation work rather than success;
• never infer deletion authority from batch terminal state alone;
• if automatic cleanup is introduced, require explicit retention/reconciliation policy after compatible terminal/retrieval state, bounded crash/restart-safe execution, and no distributed exactly-once claim;
• retain provider input/output/error file identity plus bounded cleanup/expiry evidence needed for audit after remote content disappears instead of overwriting history with NULL;
• keep remote provider retention, local PostgreSQL retention/export/erasure, backups and host logs as separate authorities;
• keep diagnostics/telemetry/audit free of prompts/results/provider bodies, credentials, DSNs, arbitrary provider text and dynamic exception names; and
• cover supported/unsupported capability, already-deleted/not-found reconciliation, transient failure, cancellation with partial output, cleanup ordering and restart/recovery with realistic tests.

Current writer boundary

Do not start a competing batch_api_client.py retention branch while Draft #317 is active. #317 is the canonical overlapping provider-client writer for #301/#302/#347. Its exact current head c7d62daccb746999a269917e6120d3d3fe6260d1 is repository-local exact-head GREEN after the #347 provider resource-ID authority repair and its induced test-fixture repair, but #317 remains Draft/unintegrated. Branch-local GREEN does not create writer vacancy and is not protected-main or immutable-release authority. Mutable exact workflow/review state belongs in #317/#244.

That overlap is a repair/serialization finding, not a reason to close #136 or discard valid predecessor work. Residual retention source work may begin only after #317 ceases to be the active writer through normal protected integration or verified complete successor inheritance, followed by a fresh intended-path inventory of every open and no-PR branch touching provider-client/schema/package/test surfaces. #316 remains the branch-lifecycle authority; historical branch counts are not writer-safety evidence.

Documentation and completion boundary

#229 owns canonical PRD/TRD/documentation-fitness, #324 owns root architecture/CHANGELOG/product-gap documentation, and #321 owns root README/docs index. Do not create a competing provider-retention documentation branch. #195/#196 own repository-wide threat/data-governance convergence. Remote provider expiry/deletion must not be promoted into local PostgreSQL erasure, backup erasure, cryptographic erasure or legal-erasure claims.

#233 remains the earliest protected integration root. Mutable central workflows/ruleset/review state belongs to #244 and live owners. Close only after residual capability/reconciliation behavior is normally integrated from writer-safe ancestry, canonical docs agree with exact protected source, all then-live exact-head gates pass, and any released claim is bound through #198 to immutable protected version/tag/package/SBOM/provenance/reproducibility/rollback identity.

Refs #136, #172, #195, #196, #198, #229, #233, #244, #301, #302, #316, #317, #321, #324, #347.

[seonghobae]

Fresh writer-boundary RCA — provider-retention source lane remains occupied (2026-08-28)

Protected main was refetched immediately before this write and remains exact b84f0c94154043a3473939c01bb6471de5a129ae.

The two retained refs named in this issue's source-lane guard were independently resolved again:

• agent/bounded-jsonl-result-streaming@64662a4410ff0ff83f150d72fee07b78c4e82d5b
• agent/bounded-jsonl-result-streaming-reconcile@5ec158ad7855a70ecebbc2ca886d81e3f1cdd3cc

Fresh exact-branch open-PR searches return no open PR for either. A direct exact-head ancestry comparison proves they are diverged, not a linear predecessor/successor pair: merge base 6a6fac2a99134f2bf0271d0d550dffd36ddb6d9c; the reconcile ref is 1 ahead / 5 behind the other retained ref. Its unique commit is 5ec158ad... (chore(stack): replay bounded streaming contributor contract). Therefore -reconcile is a name, not evidence that the other retained branch has been fully absorbed or made inert.

For this issue's residual provider-file lifecycle implementation, that means the existing source freeze remains correct and is now more precise: do not open a new competing provider-client/package/workflow/docs source lane merely because one retained ref appears to be a reconcile successor. Before any source mutation, freshly compare both exact refs plus every then-current PR/no-PR ref touching the intended batch_api_client.py / provider result-streaming / package / workflow / test surface against then-protected main. If overlap remains ambiguous, freeze only that source lane and rotate rather than deleting/rebasing/force-updating retained refs to manufacture exclusivity.

This does not promote either retained ref to protected implementation authority and does not transfer historical checks/reviews. It records the root-cause evidence needed to avoid racing two diverged retained lineages while #136 remains residual/partial.

[seonghobae]

2026-08-29 provider-retention writer correction — closed #58 head is historical; reconcile/current provider-client ownership still blocks source work

Fresh branch provenance narrows this issue's writer boundary without changing the residual privacy/operability target.

• retained agent/bounded-jsonl-result-streaming@64662a4410ff0ff83f150d72fee07b78c4e82d5b is the exact head of closed-unmerged feat(streaming): bound incremental result records #58. Merged feat(streaming): rebuild bounded result records on current main #172 is its current-main bounded-streaming replacement, so [Governance/Acquisition] Establish auditable non-default branch lifecycle and writer-retirement evidence #316 classifies the closed feat(streaming): bound incremental result records #58 ref SUPERSEDED_RETAINED for current bounded-streaming authority. Branch persistence alone is not a present provider-client writer lease.
• the distinct no-PR agent/bounded-jsonl-result-streaming-reconcile@5ec158ad7855a70ecebbc2ca886d81e3f1cdd3cc remains present and is current writer evidence pending exact intended-path/purpose reconciliation.

Accordingly the residual batch_api_client.py retention/capability lane is still not proven free: a complete then-current provider-client/schema/package/test PR/no-PR census is required, and the reconcile lineage must be reconciled or shown non-overlapping before a source branch can be created. The protected expiry/deletion primitives remain IMPLEMENTED; capability negotiation, automatic cleanup/reconciliation and durable audit remain residual work.

No source/docs branch, retained-ref deletion/rename/force-update/destructive rebase, or predecessor evidence transfer is authorized. Canonical retention/privacy docs remain separately owned by the then-current documentation lane.

[seonghobae]

Fresh writer-boundary correction against protected main@b84f0c94154043a3473939c01bb6471de5a129ae:

The residual provider-retention lane should no longer treat both retained bounded-jsonl-result-streaming names as provider-client writer authority merely from historical comparison/name lookup.

• agent/bounded-jsonl-result-streaming@64662a4410ff0ff83f150d72fee07b78c4e82d5b is the retained exact head of closed-unmerged Draft feat(streaming): bound incremental result records #58 and is SUPERSEDED_RETAINED after the bounded-streaming slice was reconstructed and merged through feat(streaming): rebuild bounded result records on current main #172 on protected ancestry. Its old provider-client diff is historical provenance, not a current lease.
• agent/bounded-jsonl-result-streaming-reconcile@5ec158ad7855a70ecebbc2ca886d81e3f1cdd3cc remains a no-PR divergent ref (179 ahead / 181 behind from merge base bf2cc2e140dc3ff4a56c3203f80f41bb9fed5d10), but its fresh exact changed-path set does not include pg_llm_batch/batch_api_client.py. It does overlap database/schema/package/docs surfaces, so its authority must be scoped to those exact paths rather than inferred from its branch name.

Preserve both refs for provenance; do not delete, rename, rebase, force-update, or transfer historical checks/reviews.

This correction does not authorize a #136 source branch. The repository still has 123 non-default refs and the residual slice can touch provider-client, lifecycle/schema/package, tests and canonical docs. A complete invocation-scoped exact-head path reconciliation for the actual intended implementation must first prove the chosen source subset writer-safe; Draft #229 separately occupies canonical documentation. Keep the current issue open and fail closed until that proof exists.

[seonghobae]

Fresh blocker refinement against unchanged protected main@b84f0c94154043a3473939c01bb6471de5a129ae: the complete branch census is now 125 refs total (main + 124 non-default), and residual work in this issue is not source-lane-free. Open Draft #317 is exact 273a785642adcc5cb7ab72272a113e1ba6de8eae from that protected base and actively owns pg_llm_batch/batch_api_client.py plus tests/test_batch_api_client_log_privacy.py for the provider-ID logging privacy repair. Because this issue's residual provider-retention/capability work explicitly targets the same provider-client surface, status: blocked is now the correct current classification. This is collision-avoidance evidence only: #317 does not implement or supersede the residual retention/deletion policy, and no competing source branch should be opened until that exact writer moves/closes and a fresh complete path-overlap inventory proves the intended residual surface free. The body-level 123 other non-default refs count is historical/stale; no deletion, ref retirement, source change, documentation change, or evidence transfer is authorized by this correction.