V37 Gate 10: Harden conversation redaction scanning

geraldarthurdavis · geraldarthurdavis · commit 344579fef160 · 2026-05-24T16:14:57.000-03:00
Replace unbounded private-key PEM regex redaction in conversation persistence and telemetry with bounded string scanning. Add closed and unclosed PEM coverage, update V37 promotion/security documentation, and refresh the promotion readiness artifact.\n\nChecks run: pnpm --filter @bitcode/api test -- --runTestsByPath src/conversations/__tests__/privacy.test.ts src/conversations/__tests__/telemetry.test.ts; pnpm --filter @bitcode/api build; pnpm run check:v37-gate7; pnpm run check:v37-gate8; pnpm run check:v37-gate10; pnpm run check:v37-promotion-readiness; git diff --check; git diff --cached --check.
diff --git a/.bitcode/v37-promotion-readiness-report.json b/.bitcode/v37-promotion-readiness-report.json
@@ -346,7 +346,7 @@
     {
       "relativePath": "BITCODE_SPEC_V37.md",
       "present": true,
-      "digest": "sha256:865e5e929cc7a4d51e1009519910c31cc8b8402ceace11e76a609f93afc27251",
+      "digest": "sha256:43a927465914e8f435d65e195d53ef63a7f9e8ba11ee1733400d0a3c6b7bf3f3",
       "requiredTokens": [
         {
           "token": "V37 promotion readiness canon",
@@ -365,7 +365,7 @@
     {
       "relativePath": "BITCODE_SPEC_V37_DELTA.md",
       "present": true,
-      "digest": "sha256:2084f8e4fda4ae284fe41e680afc4af32051f3af4c0766d7733941bc00644887",
+      "digest": "sha256:5474db49b715401c294cf044d513e7d8e62763834289265e841c17d341a61873",
       "requiredTokens": [
         {
           "token": "Gate 10: V37 Promotion Readiness",
@@ -384,7 +384,7 @@
     {
       "relativePath": "BITCODE_SPEC_V37_NOTES.md",
       "present": true,
-      "digest": "sha256:9efc013a721a4f010f7f5328ffd673f6cef8860f6b2a1e0584156db831e0ef41",
+      "digest": "sha256:42bbb9d3e77377dd2ddda7935a29d0fa4f175942c2fcc790b3b607da295e0933",
       "requiredTokens": [
         {
           "token": "Gate 10: V37 Promotion Readiness",
@@ -403,7 +403,7 @@
     {
       "relativePath": "BITCODE_SPEC_V37_PARITY_MATRIX.md",
       "present": true,
-      "digest": "sha256:ee9848b1b851fe7ead17bd2267ffeb96f5150e40435d929698d7fe29f09c1959",
+      "digest": "sha256:07f237b864378366485334fadd62a21f227299e0baaa90dc74e39709a08f0a24",
       "requiredTokens": [
         {
           "token": "## Gate 10 Parity",
@@ -422,7 +422,7 @@
     {
       "relativePath": "SPECIFICATIONS_ROADMAP.md",
       "present": true,
-      "digest": "sha256:6cf0fa3085d6b3467fe2762a322b578f6b97d0532e7386d95060f023e32f0d93",
+      "digest": "sha256:b3a91a0c5c29fc125b8f66ef1f5920c46a14a8c8c61bd645cf7a0138c9f0d1d4",
       "requiredTokens": [
         {
           "token": "V37 Gate 10 closure anchor",
@@ -437,7 +437,7 @@
     {
       "relativePath": "README.md",
       "present": true,
-      "digest": "sha256:b755fce70d06ee22b4871135816a8fd9ea2632c0b08c7992ca215a0166ccecac",
+      "digest": "sha256:acad3ea210eed868af76747907274ef8a03206ac6dc6b9ea8c254a7fed3a8fdc",
       "requiredTokens": [
         {
           "token": "check:v37-gate10",
diff --git a/BITCODE_SPEC_V37.md b/BITCODE_SPEC_V37.md
@@ -214,6 +214,12 @@ operator private notes, unpaid AssetPack source, ledger write authority, and
 wallet signing authority before durable storage, telemetry, export, replay, or
 incident repair.
 
+Private-key PEM material is redacted by bounded string scanning before generic
+secret pattern replacement. Conversation persistence and telemetry redaction
+must not depend on unbounded multiline regular expressions for private-key
+blocks; unclosed PEM-shaped input is treated as secret material through the end
+of the text.
+
 Gate 7 covers seven persistence operations: persist message, restore history,
 export history, delete history, retain history, replay history, and incident
 repair. Export may only include source-safe data visible to the requesting
@@ -321,6 +327,9 @@ source-unsafe, stale, or disconnected from source evidence; when workflow,
 promotion, spec-family, runtime, or proven-generator support is missing; or
 when any value-bearing mainnet, credential, protected source, raw protected
 prompt, unpaid AssetPack source, or wallet private material is admitted.
+Promotion security scanning is also a closing signal: CodeQL or equivalent
+static-analysis alerts against source-safe redaction, persistence, telemetry,
+or promotion paths must be repaired before the V37 version branch can promote.
 
 Gate 10 adds `generate:v37-promotion-readiness`,
 `check:v37-promotion-readiness`, `check:v37-gate10`, package tests for
diff --git a/BITCODE_SPEC_V37_DELTA.md b/BITCODE_SPEC_V37_DELTA.md
@@ -230,6 +230,7 @@ V37 / draft V38 posture preparation, package tests, workflow wiring, and
 Closure acceptance:
 
 - V37 promotion checks validate all Conversations artifacts, contracts, UI proof, telemetry/docs/runbook bindings, privacy/redaction evidence, handoff evidence, rehearsal proof, and generated proof appendix support;
+- V37 promotion security checks require persistence and telemetry redaction to use bounded private-key PEM scanning rather than unbounded multiline regular expressions, with closed and unclosed PEM-shaped payloads covered by API tests;
 - promotion scripts support V37 command planning, dry-run, generated proof output, and derived promotion commit body generation;
 - promotion rewrites runtime posture to active V37 / draft V38 only after validations pass.
 
diff --git a/BITCODE_SPEC_V37_NOTES.md b/BITCODE_SPEC_V37_NOTES.md
@@ -135,6 +135,9 @@ redacted. Export is source-safe, delete leaves only a tombstone proof,
 retention never escalates visibility, replay uses prompt template ids and
 parsed result shapes, and incident repair operates over proof roots and
 redaction verdicts.
+Private-key PEM material is handled by bounded string scanning in the shared
+conversation redaction path so malformed or unclosed private-key-shaped input
+is redacted without polynomial multiline regular-expression behavior.
 
 Gate 8 adds `ConversationTelemetryProofHooks` so conversation sessions,
 messages, streams, tools, source selectors, Terminal handoffs, retries,
@@ -148,6 +151,8 @@ panel ids, runbook ids, and redacted error classes only. Protected prompts,
 protected model responses, protected source payloads, provider tokens, wallet
 private material, settlement private payloads, unpaid AssetPack source,
 ledger write authority, and wallet signing authority remain forbidden.
+Telemetry uses the same bounded private-key redaction before truncation so
+large secret-shaped metadata cannot survive as a preview artifact.
 
 Gate 9 closure adds `ConversationRehearsal` so local and staging-testnet
 Conversations are rehearsed before promotion readiness. The package source
diff --git a/BITCODE_SPEC_V37_PARITY_MATRIX.md b/BITCODE_SPEC_V37_PARITY_MATRIX.md
@@ -289,6 +289,7 @@ Gate 9 exact rehearsal statement: local and staging-testnet rehearsals exercise
 | Promotion automation supports V37 | `scripts/promote-bitcode-canon.mjs`, `scripts/prepare-bitcode-spec-family-promotion.mjs`, `scripts/prepare-bitcode-runtime-canon-promotion.mjs`, and `.github/workflows/v37-canon-promotion.yml` | closed |
 | Runtime posture is promotable | V37 promotion rewrites V36 active / V37 draft to V37 active / draft V38 only after validation | closed |
 | Promotion payloads are source-safe | credentials, protected source, raw protected prompts, unpaid AssetPack source, wallet private material, ledger write authority, and wallet signing authority are forbidden | closed |
+| Static security findings block promotion | Conversation persistence and telemetry private-key redaction use bounded scanning with API tests for closed and unclosed PEM-shaped input | closed |
 | Workflow and package tests are wired | `scripts/check-v37-gate10-promotion-readiness.mjs`, `.github/workflows/bitcode-gate-quality.yml`, `.github/workflows/bitcode-canon-quality.yml`, and `.github/workflows/v37-canon-promotion.yml` | closed |
 
 ## Inherited V36 implementation matrix
diff --git a/README.md b/README.md
@@ -96,6 +96,9 @@ Conversation artifact, generated `BITCODE_SPEC_V37_PROVEN.md` support,
 preparation from V36 active / V37 draft to V37 active / draft V38 without
 serializing credentials, protected source, raw protected prompts, unpaid
 AssetPack source, or wallet private material.
+Promotion hardening also keeps Conversation persistence and telemetry redaction
+on bounded private-key PEM scanning with closed/unclosed PEM tests so static
+security findings block promotion instead of being waived.
 V36 Gate 2 anchors market-wide activity through the package-owned
 `ExchangeActivityBook` and the source-safe generated artifact
 `.bitcode/v36-exchange-activity-book.json`, including listing, bid, ask,
diff --git a/SPECIFICATIONS_ROADMAP.md b/SPECIFICATIONS_ROADMAP.md
@@ -48,6 +48,7 @@
 - V37 Gate 8 closure anchor: Website Conversations depth now owns package-backed `ConversationTelemetryProofHooks` source, deterministic `.bitcode/v37-conversation-telemetry-proof-hooks.json`, `source-safe-conversation-telemetry-proof-hooks-metadata`, session, message, stream, tool, source selector, Terminal handoff, retry, error, and completion telemetry families, dashboard panel ids, runbook ids, correlation ids, proof roots, telemetry roots, API stream proof hook metadata, source-safe UI telemetry preview, public docs, internal runbooks, package tests, API tests, UI tests, workflow wiring, and `check:v37-gate8`.
 - V37 Gate 9 closure anchor: Website Conversations depth now owns package-backed `ConversationRehearsal` source, deterministic `.bitcode/v37-conversation-rehearsal.json`, `source-safe-conversation-rehearsal-metadata`, local and staging-testnet rehearsals exercise chat, streaming, writing, source selector, Terminal handoff, restore, retry, redaction, and error flows, rehearsal logs/screenshots are source-safe, route/UI checks, telemetry roots, and value-bearing mainnet blocking are visible, fullscreen Rehearsal Proof UI, package tests, UI tests, workflow wiring, and `check:v37-gate9`.
 - V37 Gate 10 closure anchor: Website Conversations depth now owns package-backed `ConversationPromotionReadinessReport` source, deterministic `.bitcode/v37-promotion-readiness-report.json`, `source-safe-conversation-promotion-readiness-metadata`, source-safe coverage for all V37 Conversation artifacts, `BITCODE_SPEC_V37_PROVEN.md` generation support, promotion command dry-run support, `v37-canon-promotion.yml`, active V37 / draft V38 posture preparation, package tests, workflow wiring, and `check:v37-gate10`.
+- V37 promotion hardening anchor: Conversation persistence and telemetry redaction use bounded private-key PEM scanning with closed/unclosed PEM API tests so static security findings remain promotion-blocking rather than waived.
 - Purpose: concise running index of Bitcode/ENGI specification history, current work, and planned work.
 
 This roadmap is not an active system specification.
diff --git a/packages/api/src/conversations/__tests__/privacy.test.ts b/packages/api/src/conversations/__tests__/privacy.test.ts
@@ -44,6 +44,27 @@ describe('conversation persistence privacy redaction', () => {
     expect((redacted.value as Record<string, unknown>).token_type).toBe('source');
   });
 
+  it('redacts closed and unclosed private key PEM blocks without exposing key material', () => {
+    const closedKey = [
+      'before',
+      '-----BEGIN PRIVATE KEY-----',
+      'abc123',
+      '-----END PRIVATE KEY-----',
+      'after',
+    ].join('\n');
+    const unclosedKey = `${'-----BEGIN PRIVATE KEY-----'.repeat(200)}untrusted-tail`;
+
+    const closed = redactConversationPersistenceValue({ note: closedKey });
+    const unclosed = redactConversationPersistenceValue({ note: unclosedKey });
+
+    expect(JSON.stringify(closed.value)).toContain('[redacted:conversation-persistence-secret]');
+    expect(JSON.stringify(closed.value)).not.toContain('abc123');
+    expect(JSON.stringify(unclosed.value)).toContain('[redacted:conversation-persistence-secret]');
+    expect(JSON.stringify(unclosed.value)).not.toContain('untrusted-tail');
+    expect(closed.redactionApplied).toBe(true);
+    expect(unclosed.redactionApplied).toBe(true);
+  });
+
   it('admits only envelopes with the full forbidden disclosure boundary', () => {
     const envelope = buildConversationPersistenceEnvelope({
       operationId: 'export_history',
diff --git a/packages/api/src/conversations/__tests__/telemetry.test.ts b/packages/api/src/conversations/__tests__/telemetry.test.ts
@@ -41,6 +41,33 @@ describe('conversation telemetry proof hooks', () => {
     });
   });
 
+  it('redacts private key PEM-shaped telemetry metadata before truncation', () => {
+    const hook = buildConversationTelemetryProofHook({
+      eventFamily: 'session',
+      eventKind: 'conversation.session.info',
+      status: 'started',
+      metadata: {
+        pem: [
+          'prefix',
+          '-----BEGIN OPENSSH PRIVATE KEY-----',
+          'abcdefghijklmnopqrstuvwxyz1234567890',
+          '-----END OPENSSH PRIVATE KEY-----',
+          'suffix',
+        ].join('\n'),
+        unclosed: `${'-----BEGIN PRIVATE KEY-----'.repeat(200)}tail`,
+      },
+    });
+
+    const serialized = JSON.stringify(hook);
+    expect(serialized).toContain('[redacted:conversation-telemetry-secret]');
+    expect(serialized).not.toContain('abcdefghijklmnopqrstuvwxyz1234567890');
+    expect(serialized).not.toContain('tail');
+    expect(assertConversationTelemetryProofHookSourceSafe(hook)).toEqual({
+      admitted: true,
+      reason: 'source_safe_conversation_telemetry_proof_hook',
+    });
+  });
+
   it('attaches telemetry proof hooks to conversation stream execution rows', () => {
     const event = buildConversationStreamEvent({
       eventKind: 'completion_decision',
diff --git a/packages/api/src/conversations/privacy.ts b/packages/api/src/conversations/privacy.ts
@@ -1,5 +1,7 @@
 import * as crypto from 'crypto';
 
+import { redactPemPrivateKeyBlocks } from './secret-redaction';
+
 export type ConversationPersistenceVisibilityTier =
   | 'public'
   | 'user_visible'
@@ -48,7 +50,6 @@ const SECRET_VALUE_PATTERNS = [
   new RegExp('eyJ[A-Za-z0-9_-]{16,}\\.[A-Za-z0-9_-]{16,}\\.[A-Za-z0-9_-]{16,}', 'gu'),
   /Bearer\s+[A-Za-z0-9._-]{16,}/giu,
   /(?:password|secret|token|private_key)\s*[:=]\s*\S{8,}/giu,
-  /-----BEGIN [A-Z ]*PRIVATE KEY-----[\s\S]*?-----END [A-Z ]*PRIVATE KEY-----/gu,
 ];
 
 export const CONVERSATION_PERSISTENCE_MUST_NOT_EXPOSE = [
@@ -103,8 +104,9 @@ function shouldRedactKey(key: string) {
 }
 
 export function redactConversationPersistenceText(value: string): ConversationPersistenceRedactionResult<string> {
-  let redacted = value;
-  let redactionApplied = false;
+  const privateKeyRedaction = redactPemPrivateKeyBlocks(value, '[redacted:conversation-persistence-secret]');
+  let redacted = privateKeyRedaction.value;
+  let redactionApplied = privateKeyRedaction.redactionApplied;
 
   for (const pattern of SECRET_VALUE_PATTERNS) {
     redacted = redacted.replace(pattern, () => {
diff --git a/packages/api/src/conversations/secret-redaction.ts b/packages/api/src/conversations/secret-redaction.ts
@@ -0,0 +1,57 @@
+export type SecretTextRedactionResult = {
+  value: string;
+  redactionApplied: boolean;
+};
+
+const PEM_BEGIN_PREFIX = '-----BEGIN ';
+const PEM_END_PREFIX = '-----END ';
+const PEM_BOUNDARY_SUFFIX = '-----';
+const PRIVATE_KEY_LABEL_SUFFIX = 'PRIVATE KEY';
+const MAX_PEM_LABEL_LENGTH = 80;
+
+export function redactPemPrivateKeyBlocks(value: string, replacement: string): SecretTextRedactionResult {
+  let cursor = 0;
+  let redactionApplied = false;
+  let redacted = '';
+
+  while (cursor < value.length) {
+    const beginIndex = value.indexOf(PEM_BEGIN_PREFIX, cursor);
+    if (beginIndex === -1) {
+      redacted += value.slice(cursor);
+      break;
+    }
+
+    const labelStart = beginIndex + PEM_BEGIN_PREFIX.length;
+    const labelEnd = value.indexOf(PEM_BOUNDARY_SUFFIX, labelStart);
+    if (labelEnd === -1 || labelEnd - labelStart > MAX_PEM_LABEL_LENGTH) {
+      redacted += value.slice(cursor, labelStart);
+      cursor = labelStart;
+      continue;
+    }
+
+    const label = value.slice(labelStart, labelEnd);
+    if (!label.endsWith(PRIVATE_KEY_LABEL_SUFFIX)) {
+      redacted += value.slice(cursor, labelEnd + PEM_BOUNDARY_SUFFIX.length);
+      cursor = labelEnd + PEM_BOUNDARY_SUFFIX.length;
+      continue;
+    }
+
+    const footer = `${PEM_END_PREFIX}${label}${PEM_BOUNDARY_SUFFIX}`;
+    const footerIndex = value.indexOf(footer, labelEnd + PEM_BOUNDARY_SUFFIX.length);
+    redacted += value.slice(cursor, beginIndex);
+    redacted += replacement;
+    redactionApplied = true;
+
+    if (footerIndex === -1) {
+      cursor = value.length;
+      break;
+    }
+
+    cursor = footerIndex + footer.length;
+  }
+
+  return {
+    value: redacted,
+    redactionApplied,
+  };
+}
diff --git a/packages/api/src/conversations/telemetry.ts b/packages/api/src/conversations/telemetry.ts
@@ -1,6 +1,7 @@
 import * as crypto from 'crypto';
 
 import conversationTelemetryProofHooksArtifact from '../../../../.bitcode/v37-conversation-telemetry-proof-hooks.json';
+import { redactPemPrivateKeyBlocks } from './secret-redaction';
 
 export type ConversationTelemetryEventFamily =
   | 'session'
@@ -89,7 +90,6 @@ const SECRET_VALUE_PATTERNS = [
   new RegExp('eyJ[A-Za-z0-9_-]{16,}\\.[A-Za-z0-9_-]{16,}\\.[A-Za-z0-9_-]{16,}', 'gu'),
   /Bearer\s+[A-Za-z0-9._-]{16,}/giu,
   /(?:password|secret|token|private_key)\s*[:=]\s*\S{8,}/giu,
-  /-----BEGIN [A-Z ]*PRIVATE KEY-----[\s\S]*?-----END [A-Z ]*PRIVATE KEY-----/gu,
 ];
 
 const TELEMETRY_ROWS_BY_FAMILY = new Map(
@@ -119,13 +119,15 @@ function shouldRedactKey(key: string) {
 }
 
 function sanitizeText(value: string) {
-  let redacted = value.length > 500 ? `${value.slice(0, 497)}...` : value;
+  const privateKeyRedaction = redactPemPrivateKeyBlocks(value, '[redacted:conversation-telemetry-secret]');
+  let redacted = privateKeyRedaction.value;
 
   for (const pattern of SECRET_VALUE_PATTERNS) {
     redacted = redacted.replace(pattern, '[redacted:conversation-telemetry-secret]');
   }
 
-  return redacted.replace(PROTECTED_CONTEXT_RE, '[redacted:conversation-telemetry-protected-context]');
+  redacted = redacted.replace(PROTECTED_CONTEXT_RE, '[redacted:conversation-telemetry-protected-context]');
+  return redacted.length > 500 ? `${redacted.slice(0, 497)}...` : redacted;
 }
 
 function sanitizeMetadata(value: unknown): unknown {

Original file line number	Diff line number	Diff line change
`@@ -346,7 +346,7 @@`
`346`	`346`	`{`
`347`	`347`	`"relativePath": "BITCODE_SPEC_V37.md",`
`348`	`348`	`"present": true,`
`349`		`- "digest": "sha256:865e5e929cc7a4d51e1009519910c31cc8b8402ceace11e76a609f93afc27251",`
	`349`	`+ "digest": "sha256:43a927465914e8f435d65e195d53ef63a7f9e8ba11ee1733400d0a3c6b7bf3f3",`
`350`	`350`	`"requiredTokens": [`
`351`	`351`	`{`
`352`	`352`	`"token": "V37 promotion readiness canon",`
`@@ -365,7 +365,7 @@`
`365`	`365`	`{`
`366`	`366`	`"relativePath": "BITCODE_SPEC_V37_DELTA.md",`
`367`	`367`	`"present": true,`
`368`		`- "digest": "sha256:2084f8e4fda4ae284fe41e680afc4af32051f3af4c0766d7733941bc00644887",`
	`368`	`+ "digest": "sha256:5474db49b715401c294cf044d513e7d8e62763834289265e841c17d341a61873",`
`369`	`369`	`"requiredTokens": [`
`370`	`370`	`{`
`371`	`371`	`"token": "Gate 10: V37 Promotion Readiness",`
`@@ -384,7 +384,7 @@`
`384`	`384`	`{`
`385`	`385`	`"relativePath": "BITCODE_SPEC_V37_NOTES.md",`
`386`	`386`	`"present": true,`
`387`		`- "digest": "sha256:9efc013a721a4f010f7f5328ffd673f6cef8860f6b2a1e0584156db831e0ef41",`
	`387`	`+ "digest": "sha256:42bbb9d3e77377dd2ddda7935a29d0fa4f175942c2fcc790b3b607da295e0933",`
`388`	`388`	`"requiredTokens": [`
`389`	`389`	`{`
`390`	`390`	`"token": "Gate 10: V37 Promotion Readiness",`
`@@ -403,7 +403,7 @@`
`403`	`403`	`{`
`404`	`404`	`"relativePath": "BITCODE_SPEC_V37_PARITY_MATRIX.md",`
`405`	`405`	`"present": true,`
`406`		`- "digest": "sha256:ee9848b1b851fe7ead17bd2267ffeb96f5150e40435d929698d7fe29f09c1959",`
	`406`	`+ "digest": "sha256:07f237b864378366485334fadd62a21f227299e0baaa90dc74e39709a08f0a24",`
`407`	`407`	`"requiredTokens": [`
`408`	`408`	`{`
`409`	`409`	`"token": "## Gate 10 Parity",`
`@@ -422,7 +422,7 @@`
`422`	`422`	`{`
`423`	`423`	`"relativePath": "SPECIFICATIONS_ROADMAP.md",`
`424`	`424`	`"present": true,`
`425`		`- "digest": "sha256:6cf0fa3085d6b3467fe2762a322b578f6b97d0532e7386d95060f023e32f0d93",`
	`425`	`+ "digest": "sha256:b3a91a0c5c29fc125b8f66ef1f5920c46a14a8c8c61bd645cf7a0138c9f0d1d4",`
`426`	`426`	`"requiredTokens": [`
`427`	`427`	`{`
`428`	`428`	`"token": "V37 Gate 10 closure anchor",`
`@@ -437,7 +437,7 @@`
`437`	`437`	`{`
`438`	`438`	`"relativePath": "README.md",`
`439`	`439`	`"present": true,`
`440`		`- "digest": "sha256:b755fce70d06ee22b4871135816a8fd9ea2632c0b08c7992ca215a0166ccecac",`
	`440`	`+ "digest": "sha256:acad3ea210eed868af76747907274ef8a03206ac6dc6b9ea8c254a7fed3a8fdc",`
`441`	`441`	`"requiredTokens": [`
`442`	`442`	`{`
`443`	`443`	`"token": "check:v37-gate10",`