From e238dff51651208c800275ce0e15b0039bddc8ae Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 12 Aug 2026 15:16:04 +0000 Subject: [PATCH 1/3] =?UTF-8?q?docs(otel):=20OTel=E2=86=94OCSF=20audit=20c?= =?UTF-8?q?rosswalk=20=E2=80=94=20mapping,=20seams,=20derived=20test=20vec?= =?UTF-8?q?tors?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Answers atoulme's interest in the #2409 crosswalk offer with the artifact itself: field-level mapping between the apeirora audit data model (auditing@663d809) and OCSF 1.9.0 record_integrity/attestation, eight documented non-1:1 seams with transform rules, four one-attribute spec asks, and stdlib-only derived vectors that re-verify chain linkage across the transform (235/235 links on the full production export). Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01PEKfZALGj5NWxW2dTUd121 --- CHANGELOG.md | 1 + docs/otel-ocsf-audit-crosswalk/README.md | 337 ++++++++++++++++++ .../derive_otel_vectors.py | 228 ++++++++++++ .../otel-audit-records.sample.ndjson | 7 + 4 files changed, 573 insertions(+) create mode 100644 docs/otel-ocsf-audit-crosswalk/README.md create mode 100644 docs/otel-ocsf-audit-crosswalk/derive_otel_vectors.py create mode 100644 docs/otel-ocsf-audit-crosswalk/otel-audit-records.sample.ndjson diff --git a/CHANGELOG.md b/CHANGELOG.md index d6c5d67..f76dc68 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -12,6 +12,7 @@ Format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). - **Cisco Slack reply trimmed — Teryl leads the MCP-gap filing, Jeff co-authors.** Two facts landed after #439 merged: Teryl asked to take the upstream filing of the MCP-gap issue with Jeff as co-author (agreed, superseding the issues draft's "Lead: Fred" note), and Teryl is in CoSAI but not in the OCSF community Slack. The reply in `docs/ocsf-slack-mcp-reply-draft.md` is trimmed accordingly: the field-level sketch of the `tool` object and the "happy to share the full draft" offer are gone — proposal details are the author's to reveal, and a channel discussion the author can't join helps nobody. What remains is the connector role: confirm the production shape (6003 + `ai_operation` + reference bundle), name the gap in one paragraph, promise to ping the thread when the issue files so Cisco's use case lands on the record, and the #1724 cross-link. Header records the ownership decision. Docs only — no product impact. ### Added +- **OTel↔OCSF audit crosswalk drafted — the #2409 offer becomes an artifact before the reply promising it is even posted.** Antoine Toulme (OTel Collector maintainer, Splunk) answered the crosswalk offer in [open-telemetry/community#2409](https://github.com/open-telemetry/community/pull/2409) with "Interested in this," pointing at his stale-closed OCSF-connector draft (opentelemetry-collector-contrib#47461) — a connector that died for lack of exactly this mapping. `docs/otel-ocsf-audit-crosswalk/` is the deliverable: a field-level mapping (envelope + integrity constructs) between the apeirora audit data model (`auditing`@`663d809`, includes the `578b930` revisions our review prompted) and the OCSF 1.9.0 `record_integrity`/`attestation` shape, with requirement levels verified against both schemas — including the tell that OCSF's `prev_event.uid` is *required* while its `fingerprint` is optional, the exact inverse of OTel's hash-only chain pointer. Eight documented non-1:1 seams, each with a transform rule (canonicalization mandated-vs-declared is the deepest: a translated signature stays bound to its origin-shape bytes, so the transform must carry origin digest + declared scheme or the proof degrades to noise), and four one-attribute spec asks that would make the mapping lossless (three OTel, one already filed as ocsf-schema#1709). Backed by executable evidence, not prose: `derive_otel_vectors.py` (stdlib-only) derives OTel `AuditRecord`s from the production OCSF export and re-verifies chain linkage *across the transform* — 235/235 links on the full 236-event export, genesis by omission — and the committed 7-record sample chains against the OCSF bundle record-for-record, signatures verifiable against the same public JWKS. Docs only — no product impact. - **OCSF Slack: Cisco asked our exact question — reply drafted, and Issue 1 gets its demand signal.** Mitchell Wasson (Cisco) asked the OCSF channel how to represent AI agent logs (Codex, Claude) in OCSF and which event type fits MCP calls, guessing API Activity — which is precisely our production shape and precisely the gap Issue 1 of the CoSAI WS4 issues draft has been holding in REVIEW since July (the generic `tool` object with `primitive`/`type` axes and `mcp` sub-block, Teryl-aligned against CMF). `docs/ocsf-slack-mcp-reply-draft.md` is the paste-ready reply: confirms 6003 + `ai_operation` from production, points at the public reference bundle as a poke-able example, names the MCP-identity gap honestly (tool name smuggled into `api.operation`, server/resource/prompt unmapped), sketches the drafted `tool` object, cross-links ocsf-schema#1724 as the schema-side complement, and offers to share the full draft. Strategic note recorded in the header: a Cisco producer asking in public is the multi-producer justification for filing Issue 1 upstream — with AI Identity shipping it, IBM/CMF aligned, and Cisco asking, the proposal stops being one vendor's wish. Docs only — no product impact. (#436, #437, #438 are the same day's OCSF/OTel thread.) - **OTel Audit Logging initiative: supporter acceptance drafted — the standards footprint grows a second leg.** Hilmar Falkenberg (SAP) incorporated our two OCSF observations into the Audit Logging spec draft (apeirora/opentelemetry-specification@578b930: `audit.sequence.stream_id` chain scoping, `audit.integrity.signer` producer-vs-collector split) and invited us to join [open-telemetry/community#2409](https://github.com/open-telemetry/community/pull/2409) as a supporter. `docs/otel-2409-draft-comment.md` is the paste-ready acceptance, scoped deliberately to the OCSF↔OTel intersection: three production lessons verified absent from 578b930 (chain pointers need a resolvable record id, not only a hash; genesis omits the prev pointer — never a sentinel; signatures need a declared canonicalization or verification degrades to trusting the producer), the offer of a written OCSF↔OTel crosswalk (`stream_id`↔`chain_uid`, signer split↔`authority_uid`, chain pointer↔`prev_event`), and both public artifacts offered as test vectors (production reference bundle + stdlib-verifiable sample). Context recorded in the draft's header: SAP-led (sponsors SAP + Microsoft), developed in the `apeirora` org (SAP's IPCEI-CIS EU sovereign-cloud program) after the OTel GC declined formal SIG status Oct 2025 — supporter is the lightest-weight standards commitment available, and the crosswalk is the durable role. Docs only — no product impact. - **PR-shaped OCSF class draft for ocsf-schema#1724 — AI Agent Trust Inventory — ready to hand to Dave the moment the co-drafting offer is taken up.** `docs/ocsf-1724-class-draft/` lays the files out exactly as they would land in the ocsf-schema repo: a thin Discovery class (`events/discovery/ai_agent_trust_inventory.json`, uid 24 — first free above `cloud_resources_inventory_info`'s 23 — with a Log/Collect/Change activity enum whose `Change` caption carries the admission-control timing contract), five objects (`agent_config_declaration`, `agent_execution_params`, a single typed `agent_artifact` with a Model Weights/Adapter/Tool Schema/Policy Bundle/Charter enum instead of per-kind arrays, `ai_sampling_params`, `agent_credential` — references and scopes only, structurally incapable of carrying material), and a `dictionary-additions.json` fragment listing every new attribute, all collision-checked by name against ocsf-schema `main` 2026-08-11. Three verified conventions shaped the design: `record_integrity` is attached at `base_event`, so the per-emission chaining the issue asks for needs zero schema mechanics; `charter` already exists in the dictionary as a `file` object, so the charter digest rides `ai_agent.charter.hashes[]` natively; and `agent_artifact` introduces no new dictionary attributes at all. The worked sample (`docs/cosai-ws4-ocsf-mapping/trust-base-inventory-sample/`) is regenerated in the drafted shape — `declared_configuration`/`executed_parameters` with the typed `artifacts` array and credential `type_id`s — so the public sample Dave's issue thread links to and the class draft are one shape; `build_sample.py --verify` re-verifies the chain, digests, and event-3 divergence after the regen. The draft README carries the design rationale and six open questions for the WG (naming, enum-merge semantics, float attributes, the cross-object required-ness of loaded artifacts, TEE reference posture). Not metaschema-validated here — that needs the files inside a real ocsf-schema clone, which is step one when this graduates to an actual PR branch. Docs only — no product impact. diff --git a/docs/otel-ocsf-audit-crosswalk/README.md b/docs/otel-ocsf-audit-crosswalk/README.md new file mode 100644 index 0000000..18aded4 --- /dev/null +++ b/docs/otel-ocsf-audit-crosswalk/README.md @@ -0,0 +1,337 @@ +# OTel Audit ↔ OCSF `record_integrity` Crosswalk + +**Status:** draft for review — written for the OpenTelemetry Audit Logging +initiative ([open-telemetry/community#2409](https://github.com/open-telemetry/community/pull/2409)), +offered for the spec repo if the group wants it there. + +**Pinned inputs** (a crosswalk against moving targets is worthless — these are +the exact revisions mapped): + +| Side | Source | Revision | +|---|---|---| +| OTel | [`specification/audit/data-model.md`](https://github.com/apeirora/opentelemetry-specification/blob/auditing/specification/audit/data-model.md), `apeirora/opentelemetry-specification` branch `auditing` | `663d809` (includes `578b930`) | +| OCSF | `attestation` object + `record_integrity` profile ([ocsf/ocsf-schema#1661](https://github.com/ocsf/ocsf-schema/pull/1661)), API Activity 6003 + `ai_operation` | OCSF **1.9.0** (released 2026-08-03) | +| Fixture | [Production reference bundle](../cosai-ws4-ocsf-mapping/ocsf-log-reference-bundle/) — 236-event hash-chained export, per-event ECDSA-P256 signatures verifiable against a [public JWKS](https://api.ai-identity.co/.well-known/ai-identity-public-keys.json) | bundle of 2026-08-06 | + +**Why this exists:** both specs define tamper-evident audit records — hash +chain, per-record signature, stream identity. If the field mapping between +them is explicit and lossless, one producer can emit both shapes, and an +OTel-Collector-side translation (the +[opentelemetry-collector-contrib#47461](https://github.com/open-telemetry/opentelemetry-collector-contrib/pull/47461) +OCSF-connector idea) becomes a mechanical transform instead of a design +exercise. If it isn't written down, the two ecosystems grow divergent +integrity constructs and every producer that needs both pays the design tax +privately. This document is the mapping, the honest list of places it is +**not** 1:1 with what a transform must do there, and derived test vectors +that prove the integrity constructs survive the round trip. + +Everything here is checked against running code, not read off the two specs: +`derive_otel_vectors.py` applies the mapping to the production OCSF export +and re-verifies chain linkage across the transform (235/235 links on the +full export). + +--- + +## 1. The mapping + +Requirement levels are quoted from each side (`MUST`/`SHOULD`/`MAY` from the +OTel data model; `required`/`recommended`/`optional` from the OCSF schema). + +### 1.1 Envelope + +| OTel (`AuditRecord` on `LogRecord`) | Req | OCSF (API Activity 6003) | Req | Notes | +|---|---|---|---|---| +| `Timestamp` | MUST | `time` | required | OTel ns vs OCSF ms — precision loss OTel→OCSF (§2.8) | +| `ObservedTimestamp` | MUST | `metadata.logged_time` | optional | OCSF→OTel: when absent, set `= Timestamp` (the constraint floor) | +| `EventName` | MUST | `class_uid` + `activity_id` (`type_uid`) | required | open string vs closed enum (§2.8) | +| `Body` | MAY | `message` | recommended | | +| `Resource` (`service.*`) | MUST | `metadata.product` | required | | +| `TraceId` / `SpanId` | MAY | `metadata.correlation_uid` | optional | loose: correlation, not trace identity | +| `audit.record.id` | MUST | `metadata.uid` | optional | the stable per-record identifier; also the chain join key (§2.2) — a requirement-level mismatch worth noting: MUST vs optional | +| `audit.actor.id` / `audit.actor.type` | MUST | `actor.user.uid` / `actor.user.type_id` | `actor` required; `user` recommended | OCSF splits human vs agent actor (§2.5) | +| — | | `ai_agent.uid` / `ai_agent.name` (`ai_operation` profile) | optional | no `audit.*` home; carry as `gen_ai.agent.id` / `gen_ai.agent.name` (§2.5) | +| `audit.action` | MUST | `activity_id` (verb axis) | required | `CREATE`/`READ`/`UPDATE`/`DELETE` ↔ 1/2/3/4 | +| `audit.outcome` | MUST | `action_id` (`security_control` profile; + `status_id`) | recommended | **decision ≠ outcome** — see §2.6 | +| `audit.target.id` / `audit.target.type` | SHOULD | `api.operation` (+ `http_request.url.path`) | `api` required | the data model's own `http.endpoint` example | +| `audit.source.id` / `audit.source.type` | MAY | `src_endpoint.ip` / type | required | not exercised in the fixture | +| `audit.schema.version` | SHOULD | `metadata.version` | required | derived vectors carry `ocsf/1.9.0` to say *which* schema versioned the payload | + +### 1.2 Integrity constructs (the core) + +| OTel | Req | OCSF (`attestation` via `record_integrity`) | Req | Notes | +|---|---|---|---|---| +| `audit.sequence.stream_id` | MAY | `attestation.chain_uid` | recommended | **clean 1:1.** Same semantics: opaque id scoping one chain; demultiplexing key | +| `audit.sequence.number` | MAY | *(none — `unmapped.org_chain_seq` in the fixture)* | — | #1661 dropped the draft-era counter; see §2.4 | +| `audit.sequence.prev_hash` | MAY | `attestation.prev_event.fingerprint.value` | `prev_event` recommended; `fingerprint` within it optional | hash-only vs id+hash pointer; see §2.2, genesis in §2.3 | +| — | | `attestation.prev_event.uid` (+ `type_uid`) | **required** within `prev_event` | the resolvable half of the chain pointer; no OTel home (§2.2) | +| `audit.integrity.value` | MAY | `attestation.signatures[]` *describes* it; bytes have no OCSF field (ocsf-schema#1709) — fixture: `unmapped.signature_b64` | at_least_one | §2.1 governs what the proof is *over* | +| `audit.integrity.signer` (`producer` \| `collector`) | MAY | `attestation.authority_uid` | recommended | tier-role vs named authority; see §2.5b | +| *(multi-valued `.0`/`.1` pairs)* | MAY | `attestation_list[]` — one entry per attester | — | OCSF's array is the cleaner multi-attester shape | +| `audit.integrity.algorithm` (Resource, JWA) | MUST if value set | `signatures[].algorithm_id`/`algorithm` (per signature) | required | placement + naming differ; see §2.7 | +| `audit.integrity.certificate` (Resource) | MAY | *(no field — fixture: `unmapped.signature_key_id` = JWKS `kid`)* | — | OTel's Key-ID form fits exactly; OCSF gap is #1709 again | +| *(JCS is mandated, not declared)* | MUST | `fingerprint.serialization_id`/`serialization` | required | **the deepest divergence** — §2.1 | +| `AuditReceipt` (`RecordId`, `IntegrityHash`, `SinkTimestamp`) | MUST (API) | *(no counterpart)* | — | receipt is an emit-API artifact, not a record field; out of scope for a record-level transform | + +--- + +## 2. Where the mapping is not 1:1 — and what a transform must do + +In roughly descending order of how much they matter. + +### 2.1 Canonicalization: mandated (JCS) vs declared (`fingerprint.serialization`) + +OTel: the integrity proof is computed over the RFC 8785 (JCS) canonical form +of the record (minus `audit.integrity.*`), and implementations "MUST NOT use +any other serialization or canonicalization method." OCSF: the fingerprint +*declares* its serialization (`serialization_id` + free-text `serialization` +sibling), precisely so producers whose scheme isn't JCS can say so honestly. + +Consequence for any transform, in either direction: **a signature is bound to +the canonical bytes of its origin shape and cannot be re-derived after +translation without the signing key.** Translating the record does not — +cannot — re-sign it. So a transform MUST carry the origin-side digest and its +declared canonicalization along with the signature, or the proof degrades to +noise. The derived vectors do this with three attributes: +`ocsf.attestation.entry_hash`, `ocsf.attestation.entry_hash.algorithm`, +`ocsf.attestation.canonicalization`. Verification recipe in §3. + +Note what this means for strict conformance: a producer whose proof is over a +non-JCS canonicalization (this fixture's is a declared producer scheme) can +be *represented* in the OTel shape but not *conformant* to the MUST as +written. A small spec ask that would fix it is in §4. + +### 2.2 The chain pointer: hash-only vs id + hash + +OTel `audit.sequence.prev_hash` is a bare hash. OCSF `prev_event` carries +`uid` + `type_uid` + `fingerprint` — a *resolvable* reference plus the hash +binding it to content. The requirement levels are the tell: within +`prev_event`, `uid` is **required** and `fingerprint` is optional — OCSF made +the locator the mandatory half, the exact inverse of OTel's hash-only +pointer. A predecessor hash proves linkage but cannot locate +the predecessor across storage, sharding, or retention boundaries (raised as +#2409 review point; the id half is what makes a broken-chain investigation +actionable). OCSF→OTel is lossy here: the transform parks the id in +`ocsf.attestation.prev_event.uid`. OTel→OCSF must synthesize `prev_event.uid` +from the predecessor's `audit.record.id` — which works, because +`audit.record.id` is MUST-level. §4 has the one-attribute spec ask. + +### 2.3 Genesis: SHA-256 of empty string vs omission + +OTel: the first record of a stream "SHOULD set `audit.sequence.prev_hash` to +the SHA-256 hash of the empty string" (`e3b0c442…`). OCSF fixture behavior: +genesis **omits `prev_event` entirely** — there is no predecessor to point +at. The empty-string constant is schema-safe (unlike a `"GENESIS"` string +sentinel, which we shipped once and documented as an anti-pattern), but it is +indistinguishable from a genuine hash of empty content and it makes "has a +predecessor" a value comparison instead of a presence check. Transform rule +adopted by the vectors: genesis omits `audit.sequence.prev_hash`; a receiver +should treat the OTel constant and omission as equivalent genesis markers. +§4 asks to align on omission. + +### 2.4 Sequence number: OTel has one, final-#1661 OCSF does not + +`audit.sequence.number` (monotonic, gap = lost/deleted record, alert) has no +home in the merged attestation object — the draft-era `sequence` field was +dropped before merge. This fixture's producer keeps a per-chain counter in +`unmapped.org_chain_seq`, so OCSF→OTel maps cleanly *for this producer* but +not for OCSF producers in general: a generic transform MUST tolerate absent +sequence numbers and fall back to linkage-only continuity (deletion is still +detectable — the chain breaks — but "how many records are missing" is not). +This is a real capability OTel has that merged OCSF lacks; worth stating +plainly in both venues rather than papering over. + +### 2.5 Actor: one slot vs human + agent split + +OTel has a single mandatory `audit.actor.id`/`type` (guidance: use `user` +"even if performed by an AI agent on behalf of a user"). OCSF 6003 with +`ai_operation` carries both `actor.user` (the human/principal) **and** +`ai_agent` (the acting agent, merged via ocsf-schema#1641). For agentic +workloads that split is load-bearing — "which human authorized" and "which +agent acted" are different investigations. Transform rule: `audit.actor.*` ← +`actor.user`; the agent rides OTel's existing GenAI semconv +(`gen_ai.agent.id`, `gen_ai.agent.name`) rather than a private namespace, so +OTel tooling that already understands GenAI attributes gets the agent for +free. + +### 2.5b Signer: tier role vs named authority + +`audit.integrity.signer` is a closed tier enum — `producer` | `collector` — +answering *where in the pipeline* the proof was made. `attestation.authority_uid` +names *who* attested. These compose rather than conflict: role and identity. +OCSF→OTel: signer = `producer` when the attesting authority is the emitting +service (this fixture), `collector` for a custody-tier attestation — but that +classification requires out-of-band knowledge of which authority is which. +OTel→OCSF: `authority_uid` should carry a stable identity (service identity +or signing-key reference; `audit.integrity.certificate` is the natural +source), not the literal string "producer" — a tier is not an identity. §4 +proposes the missing attribute. + +### 2.6 Outcome vs decision + +`audit.outcome` (`success`/`failure`) records whether the action completed. +OCSF `action_id` (`Allowed`/`Denied`) records the **policy decision** — a +denied call is a *successfully denied* operation, and OCSF separately has +`status_id` for operational success. For an enforcement-point producer the +decision is the primary fact. Transform rule: `Allowed` → `success`, +`Denied` → `failure` (from the caller's perspective the action did not +complete), and the decision is preserved verbatim as `ocsf.action` / +`ocsf.action_id` so the distinction is never laundered away. Receivers doing +policy analytics should query the latter, not `audit.outcome`. + +### 2.7 Algorithm: placement and naming + +Placement: OTel pins `audit.integrity.algorithm` + `certificate` at +**Resource** scope — once per service instance, "identical for every +AuditRecord emitted." OCSF carries `algorithm` per signature. Resource scope +is the right economy for the common case but cannot represent a batch whose +records were signed by different keys (mid-batch key rotation, multi-tenant +exporters). A transform MUST verify the single-key invariant before claiming +Resource-level attributes — `derive_otel_vectors.py` hard-fails if the +export contains more than one key or algorithm rather than silently +misrepresenting it. + +Naming: OTel wants JWA identifiers (`ES256`); OCSF uses an enum + string +(`algorithm_id: 3`, `"ECDSA-P256-SHA256"`). Small, mechanical, but a +transform needs the table (`ES256` ↔ `ECDSA-P256-SHA256`, `EdDSA` ↔ +`Ed25519`, …) — it's in the script. + +### 2.8 Smaller frictions, recorded so nobody rediscovers them + +- **Timestamp precision:** OTel ns, OCSF ms. OCSF→OTel is exact (×10⁶); + OTel→OCSF truncates. Only matters if the timestamp is inside the signed + canonical form — which is another reason §2.1's declared-canonicalization + discipline matters. +- **`EventName` (open, MUST NOT be empty) vs `type_uid` (closed enum):** + the vectors derive `api.activity.create|read|update|delete`; the reverse + direction needs an EventName→class registry and will be lossy for names + outside it. +- **`SeverityNumber`:** OTel says SHOULD NOT set; OCSF `severity_id` is + required. Transform drops it OCSF→OTel (the fixture's severity encodes the + decision, already preserved); OTel→OCSF must synthesize (`Informational`). + +--- + +## 3. Test vectors + +| File | What | +|---|---| +| [`otel-audit-records.sample.ndjson`](otel-audit-records.sample.ndjson) | 7 OTel `AuditRecord`s derived from the bundle's annotated excerpt (one agent's lifecycle, chain seq 16–22) | +| [`derive_otel_vectors.py`](derive_otel_vectors.py) | stdlib-only derivation + post-transform chain verification; `--full` runs the 236-event production export | +| OCSF side | the [reference bundle](../cosai-ws4-ocsf-mapping/ocsf-log-reference-bundle/) — same records, origin shape, with its own verifier (`regenerate.py`) | + +Because both shapes are derived from the same production records, they chain +against **each other**: record N+1's `audit.sequence.prev_hash` (OTel shape) +equals record N's `attestation.fingerprint.value` (OCSF shape). Current run: +7/7 excerpt records, 6/6 internal links; full export 236 records, 235/235 +links, 1 genesis (prev omitted). + +**What verifies without any secret:** + +1. *Chain linkage, both shapes, and across shapes* — structural hash + comparison, no keys (`derive_otel_vectors.py` does it for the OTel shape). +2. *Every ECDSA signature* — `audit.integrity.value` is a DER ECDSA-P256 + signature over `bytes.fromhex(ocsf.attestation.entry_hash)`; the public + key is in the [JWKS](https://api.ai-identity.co/.well-known/ai-identity-public-keys.json) + under `kid = audit.integrity.certificate`. Note that per §2.1 the + signature is bound to the origin (OCSF) fingerprint carried in the + record — not to JCS of the OTel form. That is the honest state of a + translated proof, and exactly why §2.1/§4 matter. + +**What requires the org key:** recomputing `entry_hash` itself from record +content (the chain hash is keyed HMAC — key-holder verifiable, declared as +such in `ocsf.attestation.entry_hash.algorithm`). + +### Worked example — one record, both shapes + +The allowed inference call (chain seq 18; the same event annotated in the +[bundle README](../cosai-ws4-ocsf-mapping/ocsf-log-reference-bundle/README.md#anatomy-of-one-event-the-allowed-inference)). +OTel shape, as derived (long values truncated here; the ndjson carries full +values): + +```json +{ + "Resource": { + "service.name": "ai-identity-gateway", + "audit.integrity.algorithm": "ES256", + "audit.integrity.certificate": "projects/…/cryptoKeys/session-attestation/cryptoKeyVersions/1" + }, + "Timestamp": 1776094414825000000, + "ObservedTimestamp": 1776094414825000000, + "EventName": "api.activity.create", + "Body": null, + "Attributes": { + "audit.record.id": "99", + "audit.actor.id": "a33fb1e9-adac-4052-bdd6-e6d96292bbce", + "audit.actor.type": "user", + "audit.action": "CREATE", + "audit.outcome": "success", + "audit.target.id": "/v1/chat/completions", + "audit.target.type": "http.endpoint", + "audit.schema.version": "ocsf/1.9.0", + "audit.sequence.stream_id": "f3576cf6-87ff-4c07-b446-e6ac526236a5", + "audit.sequence.number": 18, + "audit.sequence.prev_hash": "90ba42f3b92586ff…", + "audit.integrity.value": "MEUCIQC7SNQRH0a8IEKO…", + "audit.integrity.signer": "producer", + "ocsf.attestation.entry_hash": "1d9548729d942e30…", + "ocsf.attestation.entry_hash.algorithm": "HMAC-SHA-256", + "ocsf.attestation.canonicalization": "AI-Identity audit chain v1 (sorted-compact JSON + prev hash)", + "ocsf.attestation.prev_event.uid": "98", + "gen_ai.agent.id": "32928870-56a1-4518-be76-7e99bfcdeac4", + "gen_ai.agent.name": "QA-eae97318", + "http.request.method": "POST", + "url.path": "/v1/chat/completions", + "ocsf.class_uid": 6003, + "ocsf.type_uid": 600301, + "ocsf.action": "Allowed", + "ocsf.action_id": 1, + "ocsf.duration_ms": 182, + "ocsf.policy_version": 10 + } +} +``` + +Read it against the OCSF anatomy and every §2 rule is visible in data: +`stream_id` = `chain_uid`, `prev_hash` = seq 17's fingerprint value, the +signature carried with its origin digest and declared canonicalization, the +human in `audit.actor.*` and the agent in `gen_ai.agent.*`, the Allowed +decision surviving next to the derived outcome. + +--- + +## 4. What would make the mapping lossless — four small spec asks + +Each of these is one attribute or one sentence, not a redesign. OTel-side +asks are for the audit data model; the OCSF ask is already filed. + +1. **`audit.integrity.canonicalization` (OTel, closes §2.1).** An optional + companion to `audit.integrity.value` naming the canonicalization the + proof is over — `jcs` (default, keeps today's MUST as the default path), + or a declared producer scheme. Mirrors OCSF's `fingerprint.serialization` + enum + sibling. Without it, "verify the signature" quietly becomes "trust + the producer" for every record that crossed a schema boundary. +2. **`audit.sequence.prev_record_id` (OTel, closes §2.2).** The resolvable + half of the chain pointer, `= audit.record.id` of the predecessor. + `prev_hash` binds content; the id locates it across storage, sharding, + and retention boundaries. OCSF's merged shape requires both halves for + the same reason. +3. **Genesis by omission (OTel, closes §2.3).** Change the SHA-256("") + SHOULD to: the first record of a stream omits `audit.sequence.prev_hash`. + Presence check beats value comparison, and no verifier has to special-case + a magic constant. +4. **Signature bytes + key reference (OCSF, closes the `unmapped` riders).** + Already filed as [ocsf-schema#1709](https://github.com/ocsf/ocsf-schema/issues/1709): + `digital_signature` describes a signature but cannot carry its bytes or + key id. When it lands, `audit.integrity.value` ↔ signature bytes and + `audit.integrity.certificate` ↔ key reference become clean 1:1 rows in + §1.2. + +With 1–3 in the OTel model and 4 in OCSF, every row in §1.2 is bidirectional +without an `ocsf.*` / `unmapped` escape hatch, and a signed record survives +OCSF → OTel → OCSF byte-identical in its integrity constructs. + +--- + +*Maintained in the AI Identity repo; regenerate the vectors with +`python3 derive_otel_vectors.py` after any bundle refresh. Questions / +corrections: the #2409 thread, or issues here.* diff --git a/docs/otel-ocsf-audit-crosswalk/derive_otel_vectors.py b/docs/otel-ocsf-audit-crosswalk/derive_otel_vectors.py new file mode 100644 index 0000000..0f29507 --- /dev/null +++ b/docs/otel-ocsf-audit-crosswalk/derive_otel_vectors.py @@ -0,0 +1,228 @@ +#!/usr/bin/env python3 +"""Derive OTel AuditRecord test vectors from the OCSF reference bundle. + +Applies the crosswalk in README.md mechanically: reads OCSF API Activity +events (class 6003, ``ai_operation`` + ``record_integrity`` profiles, the +final #1661 attestation shape) and emits one OTel ``AuditRecord`` per event, +shaped per the apeirora audit data model (``specification/audit/data-model.md``, +``auditing`` branch). Stdlib only — no OTel SDK, no crypto dependencies. + +Each output line is a self-contained JSON object carrying ``Resource`` + +LogRecord fields + ``Attributes``. In real OTLP the Resource would be the +batch envelope, not repeated per record; the per-line shape here is for +fixture ergonomics (NDJSON, SIEM-style). + +After deriving, the script re-verifies chain linkage *across the transform*: +for consecutive ``audit.sequence.number`` values in the same +``audit.sequence.stream_id``, record N+1's ``audit.sequence.prev_hash`` must +equal record N's ``ocsf.attestation.entry_hash``. A transform that breaks +this check has corrupted the integrity constructs it claims to preserve. + +ECDSA signature verification is deliberately out of scope here (not in the +stdlib): use the bundle's ``regenerate.py`` on the OCSF side, or follow the +recipe in README.md §3 — the signatures survive this transform byte-for-byte +and verify against the same public JWKS. + +Usage: + python3 derive_otel_vectors.py # excerpt -> sample + python3 derive_otel_vectors.py --full # full export + python3 derive_otel_vectors.py IN.ndjson -o OUT.ndjson +""" + +from __future__ import annotations + +import argparse +import json +import sys +from pathlib import Path + +BUNDLE_DIR = Path(__file__).resolve().parent.parent / "cosai-ws4-ocsf-mapping" / "ocsf-log-reference-bundle" +DEFAULT_INPUT = BUNDLE_DIR / "production-ocsf-excerpt.ocsf.ndjson" +FULL_INPUT = BUNDLE_DIR / "production-ocsf-full-export.ocsf.ndjson" +DEFAULT_OUTPUT = Path(__file__).resolve().parent / "otel-audit-records.sample.ndjson" + +# API Activity (6003) activity axis -> OTel EventName / audit.action verb. +ACTIVITY_VERBS = {1: "create", 2: "read", 3: "update", 4: "delete"} + +# OCSF actor.user.type_id -> audit.actor.type. +ACTOR_TYPES = {1: "user", 2: "user", 3: "service"} # 1 User, 2 Admin, 3 System/service + +# OCSF disposition/action axis -> audit.outcome. The policy decision itself +# is preserved separately as ocsf.action / ocsf.action_id (README §2.6). +OUTCOMES = {1: "success", 2: "failure"} + +# signatures[].algorithm (OCSF string) -> JWA identifier for +# audit.integrity.algorithm (README §2.7). +JWA_ALGORITHMS = { + "ECDSA-P256-SHA256": "ES256", + "ECDSA-P384-SHA384": "ES384", + "RSA-SHA256": "RS256", + "Ed25519": "EdDSA", +} + + +def pick_attestation(event: dict) -> dict | None: + """The record-integrity attestation entry (single-attester producer).""" + lst = event.get("attestation_list") or [] + return lst[0] if lst else None + + +def ocsf_to_otel(event: dict) -> dict: + """Map one OCSF 6003 event to one OTel AuditRecord (README §1).""" + att = pick_attestation(event) + unmapped = event.get("unmapped", {}) + activity = ACTIVITY_VERBS.get(event.get("activity_id"), "other") + actor_user = (event.get("actor") or {}).get("user") or {} + agent = event.get("ai_agent") or {} + http = event.get("http_request") or {} + + attrs: dict = { + # --- mandatory audit.* ------------------------------------------- + "audit.record.id": event["metadata"]["uid"], + "audit.actor.id": actor_user.get("uid", "unknown"), + "audit.actor.type": ACTOR_TYPES.get(actor_user.get("type_id"), "service"), + "audit.action": activity.upper(), + "audit.outcome": OUTCOMES.get(event.get("action_id"), "unknown"), + # --- optional audit.* -------------------------------------------- + "audit.target.id": event.get("api", {}).get("operation"), + "audit.target.type": "http.endpoint", + "audit.schema.version": f"ocsf/{event['metadata']['version']}", + } + + if att: + if att.get("chain_uid"): + attrs["audit.sequence.stream_id"] = att["chain_uid"] + if unmapped.get("org_chain_seq") is not None: + attrs["audit.sequence.number"] = unmapped["org_chain_seq"] + prev = att.get("prev_event") + if prev: + # Genesis records omit prev_event entirely on the OCSF side and + # therefore omit audit.sequence.prev_hash here — a deliberate, + # documented divergence from the data model's SHA-256("") + # genesis constant (README §2.3). + attrs["audit.sequence.prev_hash"] = prev["fingerprint"]["value"] + attrs["ocsf.attestation.prev_event.uid"] = prev["uid"] + + # The signature bytes ride unmapped on the OCSF side (digital_signature + # has no bytes field — ocsf-schema#1709); on the OTel side they ARE the + # integrity proof. + if unmapped.get("signature_b64"): + attrs["audit.integrity.value"] = unmapped["signature_b64"] + attrs["audit.integrity.signer"] = "producer" + + # Origin-shape integrity facts a verifier needs post-transform + # (README §2.1): the signature is computed over + # bytes.fromhex(entry_hash), NOT over JCS of this record. + fp = att.get("fingerprint") + if fp: + attrs["ocsf.attestation.entry_hash"] = fp["value"] + attrs["ocsf.attestation.entry_hash.algorithm"] = fp["algorithm"] + attrs["ocsf.attestation.canonicalization"] = fp["serialization"] + + # --- lossless remainder: fields OTel audit.* has no home for --------- + if agent.get("uid"): + attrs["gen_ai.agent.id"] = agent["uid"] + if agent.get("name"): + attrs["gen_ai.agent.name"] = agent["name"] + if http.get("http_method"): + attrs["http.request.method"] = http["http_method"] + if (http.get("url") or {}).get("path"): + attrs["url.path"] = http["url"]["path"] + attrs["ocsf.class_uid"] = event["class_uid"] + attrs["ocsf.type_uid"] = event["type_uid"] + attrs["ocsf.action"] = event.get("action") + attrs["ocsf.action_id"] = event.get("action_id") + if event.get("duration") is not None: + attrs["ocsf.duration_ms"] = event["duration"] + if unmapped.get("policy_version") is not None: + attrs["ocsf.policy_version"] = unmapped["policy_version"] + + record = { + "Timestamp": event["time"] * 1_000_000, # OCSF ms -> OTel ns + # The OCSF record carries no SDK-observation time (metadata.logged_time + # is not emitted by this producer); constraint says + # ObservedTimestamp >= Timestamp, so equality is the honest floor. + "ObservedTimestamp": event["time"] * 1_000_000, + "EventName": f"api.activity.{activity}", + "Body": None, + "Attributes": {k: v for k, v in attrs.items() if v is not None}, + } + return record + + +def derive(in_path: Path, out_path: Path) -> int: + events = [json.loads(line) for line in in_path.open() if line.strip()] + + # Resource-level integrity attributes (README §2.7): the data model pins + # algorithm + key reference at Resource scope, once per service instance. + # Verify the fixture actually satisfies that before claiming it. + kids = {e["unmapped"]["signature_key_id"] for e in events if e.get("unmapped", {}).get("signature_key_id")} + algs = {s["algorithm"] for e in events for a in e.get("attestation_list", []) for s in a.get("signatures", [])} + if len(kids) > 1 or len(algs) > 1: + sys.exit(f"ERROR: multiple signing keys ({len(kids)}) or algorithms ({algs}) in one export — " + "cannot be represented as Resource-level audit.integrity.* (README §2.7).") + resource = {"service.name": "ai-identity-gateway"} + if algs: + alg = algs.pop() + resource["audit.integrity.algorithm"] = JWA_ALGORITHMS.get(alg, alg) + if kids: + resource["audit.integrity.certificate"] = kids.pop() + + records = [] + for event in events: + rec = ocsf_to_otel(event) + rec = {"Resource": resource, **rec} + records.append(rec) + + with out_path.open("w") as fh: + for rec in records: + fh.write(json.dumps(rec, separators=(",", ":")) + "\n") + + # -- post-transform chain verification -------------------------------- + by_stream: dict[str, list[dict]] = {} + for rec in records: + sid = rec["Attributes"].get("audit.sequence.stream_id") + if sid: + by_stream.setdefault(sid, []).append(rec) + + links_ok = links_bad = links_skipped = 0 + for sid, recs in by_stream.items(): + recs.sort(key=lambda r: r["Attributes"].get("audit.sequence.number", 0)) + for prev_rec, rec in zip(recs, recs[1:]): + a, pa = rec["Attributes"], prev_rec["Attributes"] + if a.get("audit.sequence.number") != pa.get("audit.sequence.number", -2) + 1: + links_skipped += 1 # non-consecutive slice; predecessor not in this file + continue + if a.get("audit.sequence.prev_hash") == pa.get("ocsf.attestation.entry_hash"): + links_ok += 1 + else: + links_bad += 1 + print(f"LINK BROKEN: stream {sid} seq {a.get('audit.sequence.number')}", file=sys.stderr) + + genesis = sum(1 for r in records + if r["Attributes"].get("audit.sequence.number") == 1 + and "audit.sequence.prev_hash" not in r["Attributes"]) + print(f"{len(records)} AuditRecords -> {out_path.name}") + print(f"chain linkage across the transform: {links_ok} verified, {links_bad} broken, " + f"{links_skipped} skipped (predecessor outside this file)") + if genesis: + print(f"genesis records: {genesis} (prev_hash omitted — see README §2.3)") + return 1 if links_bad else 0 + + +def main() -> int: + ap = argparse.ArgumentParser(description=__doc__.splitlines()[0]) + ap.add_argument("input", nargs="?", type=Path, default=DEFAULT_INPUT) + ap.add_argument("-o", "--output", type=Path, default=None) + ap.add_argument("--full", action="store_true", + help="derive from the full 236-event production export instead of the excerpt") + args = ap.parse_args() + in_path = FULL_INPUT if args.full else args.input + out_path = args.output or ( + Path(__file__).resolve().parent / "otel-audit-records.full.ndjson" if args.full else DEFAULT_OUTPUT + ) + return derive(in_path, out_path) + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/docs/otel-ocsf-audit-crosswalk/otel-audit-records.sample.ndjson b/docs/otel-ocsf-audit-crosswalk/otel-audit-records.sample.ndjson new file mode 100644 index 0000000..02e6b15 --- /dev/null +++ b/docs/otel-ocsf-audit-crosswalk/otel-audit-records.sample.ndjson @@ -0,0 +1,7 @@ +{"Resource":{"service.name":"ai-identity-gateway","audit.integrity.algorithm":"ES256","audit.integrity.certificate":"projects/project-8bbb04f8-fda8-462e-bc2/locations/us-east1/keyRings/ai-identity-forensic/cryptoKeys/session-attestation/cryptoKeyVersions/1"},"Timestamp":1776094410673000000,"ObservedTimestamp":1776094410673000000,"EventName":"api.activity.create","Body":null,"Attributes":{"audit.record.id":"97","audit.actor.id":"a33fb1e9-adac-4052-bdd6-e6d96292bbce","audit.actor.type":"user","audit.action":"CREATE","audit.outcome":"success","audit.target.id":"/api/v1/agents","audit.target.type":"http.endpoint","audit.schema.version":"ocsf/1.9.0","audit.sequence.stream_id":"f3576cf6-87ff-4c07-b446-e6ac526236a5","audit.sequence.number":16,"audit.sequence.prev_hash":"3f63e46f14ec3fab02351abb89184cce7b628556944bfdd738dabecd50e91557","ocsf.attestation.prev_event.uid":"83","audit.integrity.value":"MEUCIQDTTWUsAcEF7Npjh8kTReoErh1y3WP/8OlcsPYKk3oTlwIgcN2M84tUgin3DiGr40qlkuRsP0lEZywEFjfA5w59++Y=","audit.integrity.signer":"producer","ocsf.attestation.entry_hash":"1e85b0036edcc2c2210c7611d3d78c6d71211f1fbc558be00856f547156b3a9f","ocsf.attestation.entry_hash.algorithm":"HMAC-SHA-256","ocsf.attestation.canonicalization":"AI-Identity audit chain v1 (sorted-compact JSON + prev hash)","gen_ai.agent.id":"32928870-56a1-4518-be76-7e99bfcdeac4","gen_ai.agent.name":"QA-eae97318","http.request.method":"POST","url.path":"/api/v1/agents","ocsf.class_uid":6003,"ocsf.type_uid":600301,"ocsf.action":"Allowed","ocsf.action_id":1}} +{"Resource":{"service.name":"ai-identity-gateway","audit.integrity.algorithm":"ES256","audit.integrity.certificate":"projects/project-8bbb04f8-fda8-462e-bc2/locations/us-east1/keyRings/ai-identity-forensic/cryptoKeys/session-attestation/cryptoKeyVersions/1"},"Timestamp":1776094412846000000,"ObservedTimestamp":1776094412846000000,"EventName":"api.activity.create","Body":null,"Attributes":{"audit.record.id":"98","audit.actor.id":"a33fb1e9-adac-4052-bdd6-e6d96292bbce","audit.actor.type":"user","audit.action":"CREATE","audit.outcome":"failure","audit.target.id":"/v1/chat/completions","audit.target.type":"http.endpoint","audit.schema.version":"ocsf/1.9.0","audit.sequence.stream_id":"f3576cf6-87ff-4c07-b446-e6ac526236a5","audit.sequence.number":17,"audit.sequence.prev_hash":"1e85b0036edcc2c2210c7611d3d78c6d71211f1fbc558be00856f547156b3a9f","ocsf.attestation.prev_event.uid":"97","audit.integrity.value":"MEYCIQC82yg+KsCSLSV0ARB4oJV0z6ZCkm2L8jpfO1eKaf3wVwIhAIixLSL44Rcq/Wg3M/J8YrCjRQ+nzgjWdylSIVKWcidA","audit.integrity.signer":"producer","ocsf.attestation.entry_hash":"90ba42f3b92586ffb034d85015fe5c6d0321a5c2242eed50b8d9301acaa1d3b0","ocsf.attestation.entry_hash.algorithm":"HMAC-SHA-256","ocsf.attestation.canonicalization":"AI-Identity audit chain v1 (sorted-compact JSON + prev hash)","gen_ai.agent.id":"32928870-56a1-4518-be76-7e99bfcdeac4","gen_ai.agent.name":"QA-eae97318","http.request.method":"POST","url.path":"/v1/chat/completions","ocsf.class_uid":6003,"ocsf.type_uid":600301,"ocsf.action":"Denied","ocsf.action_id":2,"ocsf.duration_ms":254}} +{"Resource":{"service.name":"ai-identity-gateway","audit.integrity.algorithm":"ES256","audit.integrity.certificate":"projects/project-8bbb04f8-fda8-462e-bc2/locations/us-east1/keyRings/ai-identity-forensic/cryptoKeys/session-attestation/cryptoKeyVersions/1"},"Timestamp":1776094414825000000,"ObservedTimestamp":1776094414825000000,"EventName":"api.activity.create","Body":null,"Attributes":{"audit.record.id":"99","audit.actor.id":"a33fb1e9-adac-4052-bdd6-e6d96292bbce","audit.actor.type":"user","audit.action":"CREATE","audit.outcome":"success","audit.target.id":"/v1/chat/completions","audit.target.type":"http.endpoint","audit.schema.version":"ocsf/1.9.0","audit.sequence.stream_id":"f3576cf6-87ff-4c07-b446-e6ac526236a5","audit.sequence.number":18,"audit.sequence.prev_hash":"90ba42f3b92586ffb034d85015fe5c6d0321a5c2242eed50b8d9301acaa1d3b0","ocsf.attestation.prev_event.uid":"98","audit.integrity.value":"MEUCIQC7SNQRH0a8IEKOQN3oMWr1XHOr6E6ZgOO8QGSgyYPLGAIgBKeUC8QsZVGN2IJQjAp2BAkhQhUyDbUaiQGgz190D/U=","audit.integrity.signer":"producer","ocsf.attestation.entry_hash":"1d9548729d942e307793babf5ab8efb33551dd4be5cd60d94522542cde4c3107","ocsf.attestation.entry_hash.algorithm":"HMAC-SHA-256","ocsf.attestation.canonicalization":"AI-Identity audit chain v1 (sorted-compact JSON + prev hash)","gen_ai.agent.id":"32928870-56a1-4518-be76-7e99bfcdeac4","gen_ai.agent.name":"QA-eae97318","http.request.method":"POST","url.path":"/v1/chat/completions","ocsf.class_uid":6003,"ocsf.type_uid":600301,"ocsf.action":"Allowed","ocsf.action_id":1,"ocsf.duration_ms":182,"ocsf.policy_version":10}} +{"Resource":{"service.name":"ai-identity-gateway","audit.integrity.algorithm":"ES256","audit.integrity.certificate":"projects/project-8bbb04f8-fda8-462e-bc2/locations/us-east1/keyRings/ai-identity-forensic/cryptoKeys/session-attestation/cryptoKeyVersions/1"},"Timestamp":1776094415978000000,"ObservedTimestamp":1776094415978000000,"EventName":"api.activity.delete","Body":null,"Attributes":{"audit.record.id":"100","audit.actor.id":"a33fb1e9-adac-4052-bdd6-e6d96292bbce","audit.actor.type":"user","audit.action":"DELETE","audit.outcome":"failure","audit.target.id":"/v1/admin/secrets","audit.target.type":"http.endpoint","audit.schema.version":"ocsf/1.9.0","audit.sequence.stream_id":"f3576cf6-87ff-4c07-b446-e6ac526236a5","audit.sequence.number":19,"audit.sequence.prev_hash":"1d9548729d942e307793babf5ab8efb33551dd4be5cd60d94522542cde4c3107","ocsf.attestation.prev_event.uid":"99","audit.integrity.value":"MEUCIQCygvcVxZSyFY4/wD6+NI5I5Hb/5gJp8TNrAE0woWH4PgIgf8RSE1XuJuE7TDXRvmiYaJV9P+tmfQ8z11/CppRs4l0=","audit.integrity.signer":"producer","ocsf.attestation.entry_hash":"17cf378f3e646101555e7375d446f5ad3ee407a94c4b4273e49f20643b58543f","ocsf.attestation.entry_hash.algorithm":"HMAC-SHA-256","ocsf.attestation.canonicalization":"AI-Identity audit chain v1 (sorted-compact JSON + prev hash)","gen_ai.agent.id":"32928870-56a1-4518-be76-7e99bfcdeac4","gen_ai.agent.name":"QA-eae97318","http.request.method":"DELETE","url.path":"/v1/admin/secrets","ocsf.class_uid":6003,"ocsf.type_uid":600304,"ocsf.action":"Denied","ocsf.action_id":2,"ocsf.duration_ms":180,"ocsf.policy_version":10}} +{"Resource":{"service.name":"ai-identity-gateway","audit.integrity.algorithm":"ES256","audit.integrity.certificate":"projects/project-8bbb04f8-fda8-462e-bc2/locations/us-east1/keyRings/ai-identity-forensic/cryptoKeys/session-attestation/cryptoKeyVersions/1"},"Timestamp":1776094418882000000,"ObservedTimestamp":1776094418882000000,"EventName":"api.activity.create","Body":null,"Attributes":{"audit.record.id":"101","audit.actor.id":"a33fb1e9-adac-4052-bdd6-e6d96292bbce","audit.actor.type":"user","audit.action":"CREATE","audit.outcome":"success","audit.target.id":"/api/v1/agents/32928870-56a1-4518-be76-7e99bfcdeac4/keys/rotate","audit.target.type":"http.endpoint","audit.schema.version":"ocsf/1.9.0","audit.sequence.stream_id":"f3576cf6-87ff-4c07-b446-e6ac526236a5","audit.sequence.number":20,"audit.sequence.prev_hash":"17cf378f3e646101555e7375d446f5ad3ee407a94c4b4273e49f20643b58543f","ocsf.attestation.prev_event.uid":"100","audit.integrity.value":"MEYCIQCPkEJ7e24xixoNeCLUff0tUHdIdGtV2LFvBYX9bpKEbQIhAKvSBig8iacXmEd6ySU6vKABN5wEqyK6D1DKAC4VABMc","audit.integrity.signer":"producer","ocsf.attestation.entry_hash":"3ba7d43a8e89e69f06ffb46886fbb963164818ba3a3b99aada0e143bf8191b59","ocsf.attestation.entry_hash.algorithm":"HMAC-SHA-256","ocsf.attestation.canonicalization":"AI-Identity audit chain v1 (sorted-compact JSON + prev hash)","gen_ai.agent.id":"32928870-56a1-4518-be76-7e99bfcdeac4","gen_ai.agent.name":"QA-eae97318","http.request.method":"POST","url.path":"/api/v1/agents/32928870-56a1-4518-be76-7e99bfcdeac4/keys/rotate","ocsf.class_uid":6003,"ocsf.type_uid":600301,"ocsf.action":"Allowed","ocsf.action_id":1}} +{"Resource":{"service.name":"ai-identity-gateway","audit.integrity.algorithm":"ES256","audit.integrity.certificate":"projects/project-8bbb04f8-fda8-462e-bc2/locations/us-east1/keyRings/ai-identity-forensic/cryptoKeys/session-attestation/cryptoKeyVersions/1"},"Timestamp":1776094420444000000,"ObservedTimestamp":1776094420444000000,"EventName":"api.activity.delete","Body":null,"Attributes":{"audit.record.id":"102","audit.actor.id":"a33fb1e9-adac-4052-bdd6-e6d96292bbce","audit.actor.type":"user","audit.action":"DELETE","audit.outcome":"success","audit.target.id":"/api/v1/agents/32928870-56a1-4518-be76-7e99bfcdeac4","audit.target.type":"http.endpoint","audit.schema.version":"ocsf/1.9.0","audit.sequence.stream_id":"f3576cf6-87ff-4c07-b446-e6ac526236a5","audit.sequence.number":21,"audit.sequence.prev_hash":"3ba7d43a8e89e69f06ffb46886fbb963164818ba3a3b99aada0e143bf8191b59","ocsf.attestation.prev_event.uid":"101","audit.integrity.value":"MEUCIQDLviv9BxAk0MsI4umlCVj2089I3GxetPyt8i/TR7ZlhwIgbvRIpVP/HI9t3JkLFkIw1o0Ag3sSO4xLuMmXZcsvZQo=","audit.integrity.signer":"producer","ocsf.attestation.entry_hash":"6957feb6878c645a514a240abcb3d82954417130d9cd6839629a2deebc98ce89","ocsf.attestation.entry_hash.algorithm":"HMAC-SHA-256","ocsf.attestation.canonicalization":"AI-Identity audit chain v1 (sorted-compact JSON + prev hash)","gen_ai.agent.id":"32928870-56a1-4518-be76-7e99bfcdeac4","gen_ai.agent.name":"QA-eae97318","http.request.method":"DELETE","url.path":"/api/v1/agents/32928870-56a1-4518-be76-7e99bfcdeac4","ocsf.class_uid":6003,"ocsf.type_uid":600304,"ocsf.action":"Allowed","ocsf.action_id":1}} +{"Resource":{"service.name":"ai-identity-gateway","audit.integrity.algorithm":"ES256","audit.integrity.certificate":"projects/project-8bbb04f8-fda8-462e-bc2/locations/us-east1/keyRings/ai-identity-forensic/cryptoKeys/session-attestation/cryptoKeyVersions/1"},"Timestamp":1776094421406000000,"ObservedTimestamp":1776094421406000000,"EventName":"api.activity.create","Body":null,"Attributes":{"audit.record.id":"103","audit.actor.id":"a33fb1e9-adac-4052-bdd6-e6d96292bbce","audit.actor.type":"user","audit.action":"CREATE","audit.outcome":"failure","audit.target.id":"/v1/chat/completions","audit.target.type":"http.endpoint","audit.schema.version":"ocsf/1.9.0","audit.sequence.stream_id":"f3576cf6-87ff-4c07-b446-e6ac526236a5","audit.sequence.number":22,"audit.sequence.prev_hash":"6957feb6878c645a514a240abcb3d82954417130d9cd6839629a2deebc98ce89","ocsf.attestation.prev_event.uid":"102","audit.integrity.value":"MEYCIQDMBPDOmpxKQimluf0DuO61GmqbgktJeLNxehU40T5GoAIhAM05dW/Nd1xgkwJlUdFIQDu2/mUS82rxKkFHnZ6blNaY","audit.integrity.signer":"producer","ocsf.attestation.entry_hash":"6262322924cf26b7d3a0c23eb5357b045f15dd3d35748763db06c209e70645f3","ocsf.attestation.entry_hash.algorithm":"HMAC-SHA-256","ocsf.attestation.canonicalization":"AI-Identity audit chain v1 (sorted-compact JSON + prev hash)","gen_ai.agent.id":"32928870-56a1-4518-be76-7e99bfcdeac4","gen_ai.agent.name":"QA-eae97318","http.request.method":"POST","url.path":"/v1/chat/completions","ocsf.class_uid":6003,"ocsf.type_uid":600301,"ocsf.action":"Denied","ocsf.action_id":2,"ocsf.duration_ms":120}} From 471b2c67aef9650317e5b7994af8fd8067bf5072 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 12 Aug 2026 15:18:25 +0000 Subject: [PATCH 2/3] fix(lint): explicit strict=False on pairwise zip in crosswalk verifier Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01PEKfZALGj5NWxW2dTUd121 --- docs/otel-ocsf-audit-crosswalk/derive_otel_vectors.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/otel-ocsf-audit-crosswalk/derive_otel_vectors.py b/docs/otel-ocsf-audit-crosswalk/derive_otel_vectors.py index 0f29507..95c8952 100644 --- a/docs/otel-ocsf-audit-crosswalk/derive_otel_vectors.py +++ b/docs/otel-ocsf-audit-crosswalk/derive_otel_vectors.py @@ -188,7 +188,7 @@ def derive(in_path: Path, out_path: Path) -> int: links_ok = links_bad = links_skipped = 0 for sid, recs in by_stream.items(): recs.sort(key=lambda r: r["Attributes"].get("audit.sequence.number", 0)) - for prev_rec, rec in zip(recs, recs[1:]): + for prev_rec, rec in zip(recs, recs[1:], strict=False): # pairwise; lengths differ by one a, pa = rec["Attributes"], prev_rec["Attributes"] if a.get("audit.sequence.number") != pa.get("audit.sequence.number", -2) + 1: links_skipped += 1 # non-consecutive slice; predecessor not in this file From d43d564f8d9b46b77d288ec6295b9c16d412d9af Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 12 Aug 2026 15:19:40 +0000 Subject: [PATCH 3/3] style: ruff format on crosswalk verifier Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01PEKfZALGj5NWxW2dTUd121 --- .../derive_otel_vectors.py | 56 ++++++++++++++----- 1 file changed, 41 insertions(+), 15 deletions(-) diff --git a/docs/otel-ocsf-audit-crosswalk/derive_otel_vectors.py b/docs/otel-ocsf-audit-crosswalk/derive_otel_vectors.py index 95c8952..a366d52 100644 --- a/docs/otel-ocsf-audit-crosswalk/derive_otel_vectors.py +++ b/docs/otel-ocsf-audit-crosswalk/derive_otel_vectors.py @@ -36,7 +36,9 @@ import sys from pathlib import Path -BUNDLE_DIR = Path(__file__).resolve().parent.parent / "cosai-ws4-ocsf-mapping" / "ocsf-log-reference-bundle" +BUNDLE_DIR = ( + Path(__file__).resolve().parent.parent / "cosai-ws4-ocsf-mapping" / "ocsf-log-reference-bundle" +) DEFAULT_INPUT = BUNDLE_DIR / "production-ocsf-excerpt.ocsf.ndjson" FULL_INPUT = BUNDLE_DIR / "production-ocsf-full-export.ocsf.ndjson" DEFAULT_OUTPUT = Path(__file__).resolve().parent / "otel-audit-records.sample.ndjson" @@ -156,11 +158,22 @@ def derive(in_path: Path, out_path: Path) -> int: # Resource-level integrity attributes (README §2.7): the data model pins # algorithm + key reference at Resource scope, once per service instance. # Verify the fixture actually satisfies that before claiming it. - kids = {e["unmapped"]["signature_key_id"] for e in events if e.get("unmapped", {}).get("signature_key_id")} - algs = {s["algorithm"] for e in events for a in e.get("attestation_list", []) for s in a.get("signatures", [])} + kids = { + e["unmapped"]["signature_key_id"] + for e in events + if e.get("unmapped", {}).get("signature_key_id") + } + algs = { + s["algorithm"] + for e in events + for a in e.get("attestation_list", []) + for s in a.get("signatures", []) + } if len(kids) > 1 or len(algs) > 1: - sys.exit(f"ERROR: multiple signing keys ({len(kids)}) or algorithms ({algs}) in one export — " - "cannot be represented as Resource-level audit.integrity.* (README §2.7).") + sys.exit( + f"ERROR: multiple signing keys ({len(kids)}) or algorithms ({algs}) in one export — " + "cannot be represented as Resource-level audit.integrity.* (README §2.7)." + ) resource = {"service.name": "ai-identity-gateway"} if algs: alg = algs.pop() @@ -197,14 +210,22 @@ def derive(in_path: Path, out_path: Path) -> int: links_ok += 1 else: links_bad += 1 - print(f"LINK BROKEN: stream {sid} seq {a.get('audit.sequence.number')}", file=sys.stderr) - - genesis = sum(1 for r in records - if r["Attributes"].get("audit.sequence.number") == 1 - and "audit.sequence.prev_hash" not in r["Attributes"]) + print( + f"LINK BROKEN: stream {sid} seq {a.get('audit.sequence.number')}", + file=sys.stderr, + ) + + genesis = sum( + 1 + for r in records + if r["Attributes"].get("audit.sequence.number") == 1 + and "audit.sequence.prev_hash" not in r["Attributes"] + ) print(f"{len(records)} AuditRecords -> {out_path.name}") - print(f"chain linkage across the transform: {links_ok} verified, {links_bad} broken, " - f"{links_skipped} skipped (predecessor outside this file)") + print( + f"chain linkage across the transform: {links_ok} verified, {links_bad} broken, " + f"{links_skipped} skipped (predecessor outside this file)" + ) if genesis: print(f"genesis records: {genesis} (prev_hash omitted — see README §2.3)") return 1 if links_bad else 0 @@ -214,12 +235,17 @@ def main() -> int: ap = argparse.ArgumentParser(description=__doc__.splitlines()[0]) ap.add_argument("input", nargs="?", type=Path, default=DEFAULT_INPUT) ap.add_argument("-o", "--output", type=Path, default=None) - ap.add_argument("--full", action="store_true", - help="derive from the full 236-event production export instead of the excerpt") + ap.add_argument( + "--full", + action="store_true", + help="derive from the full 236-event production export instead of the excerpt", + ) args = ap.parse_args() in_path = FULL_INPUT if args.full else args.input out_path = args.output or ( - Path(__file__).resolve().parent / "otel-audit-records.full.ndjson" if args.full else DEFAULT_OUTPUT + Path(__file__).resolve().parent / "otel-audit-records.full.ndjson" + if args.full + else DEFAULT_OUTPUT ) return derive(in_path, out_path)