Unify content hash strategy: align RID and anchor hashing

[glandua]

Summary

The claims engine uses two different hash algorithms for two different purposes, creating a coherence gap that blocks interop with the CLAMS CosmWasm contract and Regen Data Module.

Current State

Purpose	Algorithm	Fields Hashed	Location
RID generation	SHA256, truncated to 16 hex chars	claimant_uri, statement, claim_type, about_uri, metadata	claims_router.py:_claim_rid()
Content hash for anchoring	BLAKE2b-256, full 32 bytes	claim_rid, entity_uri, claimant_uri, statement, claim_type, verification, metadata, supersedes_rid	ledger_anchor.py:compute_content_hash()
Attestation RID	BLAKE2b, truncated to 16 hex chars	claim_rid, reviewer_uri	claims_router.py:_attestation_rid()

Problems

1. Two algorithms (SHA256 vs BLAKE2b-256): The RID uses SHA256 while the anchor hash uses BLAKE2b-256. The Regen Data Module uses BLAKE2b-256 exclusively. Standardizing on BLAKE2b-256 for both simplifies reasoning and aligns with on-chain conventions.

2. Anchor hash includes verification state: The content hash changes as the claim moves through verification stages. This is intentional (the anchor proves "this claim was at this state") but should be explicitly documented, and CLAMS contract should know whether to follow this pattern.

3. 16 hex char truncation = 8 bytes of collision resistance: The RID uses only 16 hex characters of SHA256, giving ~64 bits of collision resistance. For a content-addressed identifier that will be referenced across systems, this is thin. Consider increasing to 32 hex chars (128 bits) or full hash.

4. No path from RID to Data Module IRI: The orn:koi-net.claim:{hash} format cannot be converted to a valid Data Module IRI (regen:...). These are parallel addressing systems with no bridge.

Proposed Changes

1. Switch RID generation to BLAKE2b-256 (matching anchor hash and Data Module)
2. Increase RID hash prefix from 16 to 32 hex characters
3. Explicitly document that anchor hash is state-inclusive (a snapshot, not a stable identifier)
4. Add a data_iri field to the claims table that stores the Regen Data Module IRI derived from the content hash -- this becomes the bridge identifier between KOI and CLAMS

Migration

Existing claims retain their current RIDs (no retroactive changes). New claims generated after the change use the updated algorithm. The supersedes_rid mechanism handles version chains across the boundary.

Related

• Integrate regen-data-standards schemas: add @context, reference FWG taxonomy #11 (regen-data-standards schema integration)
• Add Claim and Attestation schemas to data standards regen-network/regen-data-standards#52

---

Implementation Spec (for Darren)

1. Switch _claim_rid() to BLAKE2b-256

In claims_router.py, replace:

h = hashlib.sha256()

with:

h = hashlib.blake2b(digest_size=32)

This aligns RID generation with the anchor hash algorithm and the Regen Data Module convention (BLAKE2b-256).

2. Increase hash prefix from 16 to 32 hex chars

In _claim_rid(), change:

return f"orn:koi-net.claim:{h.hexdigest()[:16]}"

to:

return f"orn:koi-net.claim:{h.hexdigest()[:32]}"

This increases collision resistance from ~64 bits to ~128 bits (birthday bound ~2^64 → ~2^128). 32 hex chars = 16 bytes = 128 bits.

3. Same for _attestation_rid()

Already uses BLAKE2b, just increase truncation:

return f"orn:koi-net.attestation:{h.hexdigest()[:32]}"

4. Add data_iri column

New Alembic migration:

"""Add data_iri column to claims table"""

def upgrade():
    op.add_column('claims', sa.Column('data_iri', sa.Text(), nullable=True))
    op.create_index('ix_claims_data_iri', 'claims', ['data_iri'], unique=True)

Populate during prepare-anchor flow in ledger_anchor.py:

# After computing content_hash:
data_iri = f"regen:{content_hash_base64url}.rdf"
# Store in claims table

The exact IRI format depends on Regen Data Module conventions — use regen:{base64url(blake2b_hash)}.rdf for graph content.

5. Document state-inclusive anchor hash

Add docstring to compute_content_hash():

def compute_content_hash(claim_data: dict) -> str:
    """Compute BLAKE2b-256 content hash for on-chain anchoring.
    
    IMPORTANT: This hash includes `verification` status, making it
    state-inclusive. The hash is a snapshot of the claim at a point in
    time, NOT a stable identifier. The claim RID (orn:koi-net.claim:...)
    is the stable identifier; the content hash changes as verification
    state evolves.
    
    This is intentional: anchoring proves "this claim was at this 
    verification state at this time." CLAMS contract should follow
    the same pattern for MsgAnchor operations.
    """

6. Migration strategy

• Existing 49+ dogfood claims: Keep their current SHA256 RIDs unchanged. No retroactive re-hashing.
• New claims after deployment: Get BLAKE2b-256 RIDs with 32 hex chars.
• Backward compat: All query endpoints accept both old (16-char) and new (32-char) RID formats. The orn:koi-net.claim: prefix remains the same.
• supersedes_rid: Version chains work across the boundary — old RIDs can be referenced in supersedes_rid fields of new claims.
• No data_iri for old claims: The data_iri column is nullable. Old claims get populated only if/when they are re-anchored.

Acceptance Criteria

• _claim_rid() uses hashlib.blake2b(digest_size=32) instead of hashlib.sha256()
• RID prefix length is 32 hex chars (both claims and attestations)
• compute_content_hash() has clear docstring about state-inclusive hashing
• New migration adds data_iri TEXT column with unique index
• prepare-anchor populates data_iri from content hash
• Existing dogfood claims still readable and queryable
• Both 16-char and 32-char RIDs accepted in query endpoints

[DarrenZal]

✅ Implemented — b38eecb7 on regen-prod

All acceptance criteria met:

• _claim_rid() uses hashlib.blake2b(digest_size=32) instead of hashlib.sha256()
• RID prefix length is 32 hex chars (both claims and attestations)
• New migration 070_data_iri.sql adds data_iri TEXT column with unique partial index
• prepare-anchor populates data_iri via derive_ledger_iri(content_hash)
• anchor_claim lazily backfills data_iri for pre-070 claims that have content_hash but no data_iri
• reconcile backfills data_iri from ledger_iri when NULL
• data_iri included in ClaimResponse, proof pack claim dict, and proof pack anchor dict
• Existing 50+ dogfood claims still readable and queryable (TEXT column, no format constraint)
• Cross-cutover idempotency guard: re-posting content that exists under a legacy 16-char SHA256 RID returns the existing claim instead of creating a duplicate

Semantic note on data_iri vs ledger_iri: data_iri is derived from content hash at prepare-anchor time (pre-broadcast); ledger_iri is populated after broadcast confirms on-chain. In practice they're equal once anchored. We use whatever derive_ledger_iri() produces (currently file_extension="json", not .rdf — if the spec expects .rdf that's a separate alignment).

Not changed: compute_content_hash() in ledger_anchor.py — the verification field was intentionally removed from canonical hashing per commit 4823bfa9 (dogfood-results.md:113). No state-inclusive hashing to document.

29 tests covering hash unification (legacy RID queryability, 32-char BLAKE2b RIDs, data_iri lifecycle, cross-cutover idempotency).