fix(nostr-pinner): move chain validation to Go sidecar to eliminate remaining Python memory leak

[vrogojin]

Problem

After moving /routing-get, DHT refresh, and /ws/ipns to the Go sidecar (PR #18 + upcoming PR), the Python pinner still leaks ~160 MB/min from the ipns-intercept write path:

• ~28 ipns-intercept stores/min trigger chain validation
• Each chain validation call fetches CID content from kubo via /api/v0/cat
• The HTTP response parsing (httpx response objects, CBOR/JSON decode) creates short-lived Python objects that fragment CPython's pymalloc arenas
• malloc_trim() cannot reclaim pymalloc-internal fragmentation

Current Memory Profile

Component	RSS	Growth	Status
Go sidecar	27 MB	flat	routing-get, DHT refresh, ws/ipns
Python pinner	~160 MB/min	leaking	ipns-intercept, chain validation, Nostr

The Python pinner self-restarts at 4 GB (~25 min cycle). This is stable but wastes resources.

Root Cause

The validate_version_chain() function in nostr_pinner.py fetches CID content from kubo:

response = await client.post(f"{IPFS_API_URL}/api/v0/cat", params={"arg": cid})
content = response.content  # bytes object allocated in pymalloc arena
data = cbor2.loads(content)  # dict/list objects allocated in pymalloc arena

Even at ~25 req/min, the accumulated allocations fragment pymalloc arenas faster than they can be reclaimed.

Proposed Fix

Option A: Move chain validation to Go sidecar (recommended)

Move the validate_version_chain logic to Go:

1. Go sidecar adds /internal/validate-chain endpoint
2. Accepts ipns_name, new_cid, current_cid, new_sequence, existing_sequence
3. Fetches CID content from kubo (/api/v0/cat)
4. Parses CBOR to extract _meta.lastCid and _meta.version
5. Validates chain link (lastCid must equal current CID)
6. Returns validation result as JSON

Python ipns-intercept calls Go for validation instead of doing it in-process. This eliminates all HTTP/CBOR allocations from Python.

Estimated impact: Reduces Python to Nostr relay connections + SQLite writes only. Expected steady-state: ~100-200 MB, flat.

Option B: Move entire ipns-intercept to Go

More aggressive: move the full ipns-intercept handler to Go, including:

• Rate limiting (per-IP, per-name, global)
• Signature verification
• Chain validation
• Record storage with optimistic locking
• Security audit logging

Python would only handle Nostr relay subscriptions and reannouncement.

Estimated impact: Python ~50-100 MB, flat. But significantly more Go code.

Option C: Accept current state

The 25-min self-restart cycle is stable and the Go sidecar handles all high-throughput paths. The remaining leak is bounded by MAX_RSS_MB (4 GB default).

This is acceptable if the operational overhead of the restart cycle is tolerable.

Context

• Issue fix(nostr-pinner): eliminate Python memory leak via architectural redesign #17: original memory leak analysis and architectural plan
• PR fix(nostr-pinner): reduce memory pressure via nginx cache and pre-serialized responses #18: Phase 1+2 (nginx cache, pre-serialized responses, regex extraction)
• Upcoming PR: Phase 3 (Go sidecar for routing-get, DHT refresh, ws/ipns)
• Production: 99k+ IPNS records, ~43 req/sec total, ~28 ipns-intercept writes/min