Prevent JSONB NUL payloads from dropping observability data

[sjawhar]

Summary

On 2026-09-09, PostgreSQL JSONB writes dropped conversation events and request logs when captured content contained U+0000. Those losses remove the affected session tail from history, activity monitoring, and judge transcripts.

Production evidence

The /ecs/luthien-proxy log group recorded asyncpg.exceptions.UntranslatableCharacterError('unsupported Unicode escape sequence') from conversation_events.payload writes. The event writer dropped 8 events at 11:43Z, about 135 from 13:24Z to 13:27Z, and about 100 from 19:13Z to 21:01Z. Its in-process counter read 111 at 21:01Z. Datadog APM recorded 28 matching INSERT error spans in the preceding hour and seven luthien_proxy.utils.db.DatabaseWriteError spans.

Mechanism and fix

json.dumps escapes U+0000 as \u0000, which PostgreSQL JSONB rejects. A shared write-boundary serializer now converts JSON-compatible values, replaces every NUL in keys and values with U+FFFD, and adds _sanitized: {"nul_replaced": N} to dict payloads. Both EventEmitter and RequestLogRecorder use it immediately before JSON serialization. Database-drop warnings now identify the call ID, session ID, and event type or request-log direction.

Verification

• Unit regression coverage verifies nested keys and values, non-string leaves, the stored marker, SQLite event storage, request-log headers and bodies, and attributed warnings: 78 passed.

• A PostgreSQL integration test persisted an event containing NUL characters and read back U+FFFD plus _sanitized: 1 passed.

• Red check with the sanitizer bypassed produced the original failed-write behavior:

  FAILED test_unsanitized_event_payload_is_dropped_by_postgres
  WARNING Failed to write event to database (...): UntranslatableCharacterError('unsupported Unicode escape sequence')
  asyncpg.exceptions.UntranslatableCharacterError: unsupported Unicode escape sequence
  DETAIL:  \u0000 cannot be converted to text.
  

• PYTEST_ADDOPTS='--ignore=tests/luthien_proxy/e2e_tests' ./scripts/dev_checks.sh completed with Ruff and Pyright clean. The default test selection already excludes the e2e tier; this also prevents its collection-time shared .env loader from changing unit-test configuration.

[sjawhar]

Production recurrence data for this fix (Trajectory Labs deployment, 2026-09-10):

• 07:45-08:30 UTC: 72 asyncpg.exceptions.UntranslatableCharacterError spans on INSERT (unsupported Unicode escape sequence — DETAIL: \u0000 cannot be converted to text) and 18 luthien_proxy.utils.db.DatabaseWriteError in 45 minutes; the APM error-rate monitor fired twice (08:07, 08:16).
• The same pair fired at 01:52-01:53 UTC the same day (2 / 18 in that window), so the class is recurring and growing with traffic that carries NUL bytes in model I/O.

Each failure drops the observability event for that request, which is the data loss this PR prevents.