Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 30 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -270,3 +270,33 @@ add CODEOWNERS-based merge gates until multiple independent maintainers exist.
- Update README, architecture, ADR, operator documentation, doctoring, and
CHANGELOG whenever migration ordering, locking, evidence, failure, or
compatibility semantics change.

## Checkpoint audit export pagination contract

- Keep pagination opt-in and preserve `list_audit_events()` behavior. Do not
replace the existing bounded convenience read with an unbounded iterator.
- Use `checkpoint_audit_event_id` as a strict positive signed-PostgreSQL-`BIGINT`
keyset cursor. Do not use `OFFSET` for multi-page audit traversal.
- Query at most the validated public limit plus one lookahead row and expose at
most the requested 1..1,000 events. A driver returning more than the SQL bound
fails closed.
- Continuations must use `checkpoint_audit_event_id < before_audit_event_id` in
exact newest-first order. When more rows exist, the next cursor is exactly the
final returned event identity.
- Revalidate every database row through `CheckpointAuditEvent`, compare its
tenant/consumer/endpoint/batch key to the trusted request, and require strict
descending identities before exposing a page.
- Treat the cursor as navigation state only. It is not proof of completeness,
chronology, authenticity, delivery, or non-repudiation; identity allocation
can contain gaps and commit ordering can differ from allocation ordering.
- Package-owned page calls do not promise a single historic snapshot. Hosts
requiring one export snapshot must begin their own PostgreSQL `REPEATABLE READ`
or stricter transaction before the first query and repeatedly call
`list_audit_event_page_in_transaction()` on the same transaction.
- Keep export destinations, credentials, retention policy, immutable/WORM
storage, delivery receipts, cryptographic manifests, and reconciliation in the
host/operator boundary. This package primitive adds no network exporter or
write authority.
- Maintain 100% production statement, branch, and public-docstring coverage with
strict cursor, page-shape, lookahead, keyset SQL, ordering, row-key,
malformed-driver, concurrency-semantics, and transaction-ownership tests.
30 changes: 30 additions & 0 deletions ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -426,3 +426,33 @@ concurrent lock waiting, body-free JSON, unchanged `init-db`, and 100% productio
statement, branch, and public-docstring coverage. Final merge evidence must be
regenerated against the integrated base; successful stacked-base runs are not
reusable release evidence.

## Bounded checkpoint-audit export boundary

`CheckpointAuditPage`, `list_audit_event_page()`, and
`list_audit_event_page_in_transaction()` add an opt-in bounded traversal layer
without changing the existing one-page audit read or database schema. Pagination
uses `checkpoint_audit_event_id` as a strict positive signed PostgreSQL `BIGINT`
keyset cursor, never `OFFSET`.

Each query requests at most the validated public limit plus one lookahead row and
exposes at most 1,000 events. Continuations use
`checkpoint_audit_event_id < before_audit_event_id` in newest-first order. Every
row is revalidated through `CheckpointAuditEvent`, compared with the exact trusted
tenant/consumer/endpoint/batch key, and required to remain strictly descending.
Malformed collections, impossible driver overruns, cross-key rows, duplicate or
ascending identities, and cursor-domain violations fail closed before exposure.

Keyset traversal prevents later higher-identity inserts from shifting an older
continuation window, but package-owned calls do not provide one multi-page
historic snapshot. A host that requires snapshot-stable export must begin a
caller-owned PostgreSQL `REPEATABLE READ` or stricter transaction before the first
query and repeatedly call the in-transaction method on that same transaction.

Audit identities are navigation keys, not cryptographic chronology or completeness
proof. Sequence gaps and allocation/commit reordering are valid. External
immutable/WORM retention, delivery receipts, cryptographic manifests,
reconciliation, legal hold, and disposal remain host/operator responsibilities.
The primitive stays independently deployable and can be embedded into CWL MSA
workflows without requiring `contextual-orchestrator`, `naruon`, or a network
export service.
12 changes: 12 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,18 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

### Added

- Opt-in bounded checkpoint-audit export pagination through immutable
`CheckpointAuditPage`, `list_audit_event_page()`, and
`list_audit_event_page_in_transaction()`. Pagination uses a strict positive
PostgreSQL `BIGINT` keyset cursor, newest-first `<` continuation, and one-row
bounded lookahead rather than `OFFSET`; returned rows are revalidated against
the exact trusted tenant/consumer/endpoint/batch key and strict descending
identity order. A live PostgreSQL regression proves that a newer committed row
between pages cannot drift into the older continuation window. Package-owned
calls do not claim one multi-page snapshot; hosts needing that guarantee own a
`REPEATABLE READ` or stricter transaction. External immutable/WORM retention,
receipts, cryptographic manifests, and reconciliation remain host controls. No
migration, version bump, or release is included.
- Explicit opt-in `init-checkpoint-storage` operator for existing PostgreSQL
volumes. It bounded-reads, validates, and SHA-256 identifies
`0007_result_stream_checkpoints` and
Expand Down
28 changes: 28 additions & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -246,3 +246,31 @@
strict red-green-refactor tests for bounded input, exact order, one lock, one
transaction, one commit, rollback, concurrency, compatibility, CLI output,
documentation, and body-free diagnostics.

## Checkpoint audit export pagination invariants

- Keep `list_audit_events()` source-compatible and make multi-page traversal
opt-in through `CheckpointAuditPage` and the `list_audit_event_page*` methods.
- Validate `before_audit_event_id` as `None` or a strict positive signed
PostgreSQL `BIGINT`; reject booleans, coercible strings/floats, zero, negative,
and out-of-range identities before database access.
- Use primary-key keyset pagination only: newest-first ordering and strict
`checkpoint_audit_event_id < before_audit_event_id` continuation. Never use
`OFFSET` for retained audit traversal.
- Fetch no more than the validated limit plus one lookahead row and expose no
more than 1,000 events. Treat a driver overrun as an integrity failure.
- Revalidate every database row through `CheckpointAuditEvent`, require the
exact trusted tenant/consumer/endpoint/batch key, and require strictly
descending unique identities before exposing a page.
- The continuation cursor is navigation evidence only. It is not a completeness,
chronology, delivery, authenticity, or non-repudiation attestation; identity
gaps and allocation/commit reordering remain valid PostgreSQL behavior.
- Package-owned page calls do not provide one cross-page snapshot. Hosts needing
snapshot-stable export must begin a caller-owned `REPEATABLE READ` or stricter
transaction before the first query and reuse the in-transaction method.
- Keep destination credentials, immutable/WORM storage, retention, legal hold,
export receipts, cryptographic manifests, and reconciliation outside package
write authority.
- Maintain 100% production statement, branch, and public-docstring coverage with
strict cursor, immutable-page, lookahead, keyset SQL, trusted-key, ordering,
malformed-driver, no-commit, and live concurrent-insert pagination tests.
125 changes: 125 additions & 0 deletions docs/adr/0011-bounded-checkpoint-audit-export-pagination.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,125 @@
# ADR 0011: Bounded checkpoint-audit export pagination

- **Status:** Accepted for the stacked implementation
- **Date:** 2026-08-07
- **Decision owners:** ContextualWisdomLab
- **Depends on:** ADR 0009 and the append-only checkpoint audit trail
- **Stack order:** Follows ADR 0010, the atomic checkpoint schema operator

## Context

The package-owned audit store previously exposed only one newest-first bounded
read. That is appropriate for an operator console, but not sufficient for a
host that must move a longer retained audit history to separately governed
storage. Leaving multi-page traversal to every embedding product would produce
incompatible cursor semantics and would encourage `OFFSET` pagination, which can
skip or duplicate rows when the visible set changes between requests.

The existing audit identity is a PostgreSQL `BIGINT GENERATED ALWAYS AS
IDENTITY` primary key. The compound checkpoint-key index already ends in
`checkpoint_audit_event_id DESC`, so the table can support bounded keyset
pagination without a schema migration or a second ordering column.

## Decision

Add an opt-in `CheckpointAuditPage` contract and two audit-store methods:

- `list_audit_event_page()` for package-owned page reads; and
- `list_audit_event_page_in_transaction()` for hosts that own the surrounding
PostgreSQL transaction.

Each request validates a strict 1..1,000 page size and an optional positive
signed-`BIGINT` `before_audit_event_id`. The first page orders by
`checkpoint_audit_event_id DESC`. Continuations add the strict predicate
`checkpoint_audit_event_id < before_audit_event_id` and preserve the same order.
The SQL query requests only `limit + 1` rows. At most `limit` rows are returned;
the lookahead row only proves that an older continuation exists. When a
continuation exists, `next_before_audit_event_id` is exactly the identity of the
last returned event.

Every returned row is revalidated as a `CheckpointAuditEvent`, rechecked against
the trusted tenant/consumer/endpoint/batch key, and required to be strictly
descending. A database adapter that returns a non-sequence or more than the
bounded query size fails closed.

## Concurrency boundary

Keyset traversal solves offset drift, not snapshot isolation. Rows committed
after page one with identities greater than the cursor cannot move or duplicate
older rows already traversed. However, package-owned calls execute as separate
transactions and therefore do not promise one historic database snapshot.

A host that requires one snapshot for an export pass must own a PostgreSQL
`REPEATABLE READ` or stricter transaction and call the in-transaction method on
one cursor. The package does not silently alter transaction isolation because
that would change caller-owned database semantics.

Identity allocation order is not a cryptographic chronology and may contain
gaps. A transaction can allocate an identity before another transaction and
commit later. The cursor is therefore a database navigation key, not a statement
that all real-world events before or after a wall-clock instant have been
captured.

## Security and acquisition boundary

The page remains tenant-qualified and inherits forced RLS and ordinary-role
append-only controls from ADR 0009. It does not create a new database object,
credential, network destination, export worker, background process, or write
permission. It never accepts provider or model output as tenant or consumer
identity.

The feature is intended to make bounded export to separately governed immutable
or write-once retention storage practical. It does not itself provide immutable
external storage, cryptographic non-repudiation, signed completeness evidence,
administrator-proof tamper detection, or delivery acknowledgement. Those remain
host/operator controls. A later cryptographically protected export manifest can
be layered on this stable pagination contract without changing database
navigation semantics.

## Alternatives rejected

### SQL OFFSET/LIMIT

Rejected because additions or visibility changes before an offset can shift the
subsequent window, producing duplicates or omissions during a long-running
export.

### Unbounded iterator or fetch-all export

Rejected because acquisition-grade audit volume is not bounded by a single
operator interaction. Materializing arbitrary retained history would weaken the
package's existing memory-safety policy.

### Implicit REPEATABLE READ in the package-owned API

Rejected because changing isolation belongs to the owner of the transaction and
must occur before the first transaction query. The in-transaction API gives a
host an explicit way to obtain one snapshot without the package surprising
other database work.

### Timestamp cursor

Rejected because timestamps are not unique and database wall-clock values are
not a total order. The existing primary-key identity is already indexed and
provides an unambiguous strict continuation boundary.

## Verification

Permanent deterministic tests require strict cursor validation, immutable page
shape, strictly descending identities, one-row lookahead, no `OFFSET`, exact
`<` keyset continuation, trusted-key row revalidation, bounded driver output,
and no package commit during owned page reads. The live PostgreSQL regression
reads page one, commits a newer event in a separate transaction, and proves that
page two continues toward older identities without replaying page-one rows or
admitting the newer row. Integration/release gates remain required after this
stack is reconciled onto protected `main`.

## Consequences

Operators gain a package-owned bounded traversal primitive suitable for durable
audit export and acquisition diligence. Existing `list_audit_events()` behavior
is unchanged. No migration, release version, or publication authority is added.

The operational tradeoff is explicit: callers choosing independent page calls
accept normal transaction-to-transaction visibility changes; callers requiring a
single snapshot must own and configure that transaction deliberately.
56 changes: 51 additions & 5 deletions docs/checkpoint-audit.md
Original file line number Diff line number Diff line change
Expand Up @@ -89,6 +89,50 @@ action, a database-generated event identity, and an insert-time database
wall-clock timestamp. It does not include provider bodies, prompts, model output,
credentials, DSNs, transport headers, or exception text.

## Export longer retained history

Use the opt-in keyset page API instead of `OFFSET` or an unbounded fetch:

```python
page = store.list_audit_event_page(
"invoice-worker",
"batch-123",
"default",
limit=250,
)

while True:
persist_to_governed_retention(page.events)
if page.next_before_audit_event_id is None:
break
page = store.list_audit_event_page(
"invoice-worker",
"batch-123",
"default",
before_audit_event_id=page.next_before_audit_event_id,
limit=250,
)
```

`before_audit_event_id` is `None` or a strict positive signed PostgreSQL `BIGINT`.
Each SQL request reads at most `limit + 1` rows and returns at most `limit` events.
Continuation uses `checkpoint_audit_event_id < before_audit_event_id` in strict
newest-first order. Returned rows are revalidated against the exact trusted
tenant, consumer, endpoint, and batch key before exposure.

A committed event with a larger identity between page calls cannot shift the
older continuation window. Separate package-owned calls are still separate
transactions, however, so they do not form one historic database snapshot. For
an export that must observe one PostgreSQL snapshot, begin a caller-owned
`REPEATABLE READ` or stricter transaction **before the first query** and reuse
`list_audit_event_page_in_transaction()` on that transaction's cursor.

The cursor is navigation state, not completeness, chronology, delivery,
authenticity, or non-repudiation evidence. PostgreSQL identity sequences may have
gaps and allocation order can differ from commit order. Destination credentials,
immutable/WORM storage, retention, legal hold, delivery receipts, cryptographic
manifests, and reconciliation remain host/operator responsibilities.

## Retention and rollback

The database blocks ordinary `UPDATE`, `DELETE`, and `TRUNCATE` against the audit
Expand All @@ -105,8 +149,10 @@ signed/hash-chained evidence system.

## Standards and evidence

See [ADR 0009](adr/0009-append-only-checkpoint-audit-trail.md) for the decision
boundary and
[the assurance record](doctoring/checkpoint-audit-trail.md) for threat model,
verification evidence, and APA 7 references to NIST SP 800-53 Rev. 5 AU-3, the
OWASP Logging Cheat Sheet, and PostgreSQL 18 trigger and current-time semantics.
See [ADR 0009](adr/0009-append-only-checkpoint-audit-trail.md) for the accepted-save
audit boundary, [ADR 0011](adr/0011-bounded-checkpoint-audit-export-pagination.md)
for the pagination decision, [the audit assurance record](doctoring/checkpoint-audit-trail.md)
for audit threat-model evidence, and
[the export-pagination assurance record](doctoring/checkpoint-audit-export-pagination.md)
for keyset, isolation, operator, and APA 7 evidence. The latter records NIST SP
800-53 Rev. 5 AU-9 and PostgreSQL 18 transaction-isolation/concurrency guidance.
Loading
Loading