feat(nginx)(sphere-sdk-#370): expose /api/v0/dag/import + /api/v0/dag/export

[vrogojin]

Summary

Part 1 of unicity-sphere/sphere-sdk#370. Adds two location blocks to the nginx ACL on the Kubo API surface so sphere-sdk's CAR-batched fast paths can replace its per-block /dag/put loop and per-block /block/get BFS walk with single-shot CAR push and pull.

Background

The sphere-sdk currently pins UXF/Profile CARs block-by-block via parallel /api/v0/dag/put calls (DEFAULT_PIN_CONCURRENCY=10). A 250-block migration CAR emits ~250 HTTP requests against this gateway. The new SDK code (sphere-sdk PR #371) prefers a single /api/v0/dag/import POST for the whole CAR — dramatically fewer HTTP round-trips, no parallel-connection burst at the proxy layer.

This is the operator-side half: the nginx ACL in this repo gates the Kubo API surface to a small allowlist of routes (everything not explicitly listed falls through to the gateway on :8080 and returns 404). Both /dag/import and /dag/export are exposed by Kubo by default on :5001 — they're only blocked here at the nginx layer.

Backwards compatibility

The SDK side capability-probes each gateway at startup. Until this change is deployed, the probe sees 404 from both endpoints and transparently falls back to the existing per-block paths. Deploying this change is safe for every existing SDK build — the only behavioural change is that newer SDK builds will start using the fast path automatically.

Sizing rationale

Setting	/dag/import	/dag/export	Why
client_max_body_size	100M	(none)	Import body is the full CAR; export body is just ?arg=<cid>. Sized 2× /dag/put's 50M cap.
proxy_read_timeout	300s	300s	Kubo's internal pin of every block in a CAR takes seconds; headroom over typical ~1-2s.
proxy_send_timeout	300s	300s	Symmetric.
proxy_buffering	(default on)	off	Export responses are tens of MB and content-addressed (verified end-to-end by the receiver); buffering wastes worker memory under concurrent fetches.
CORS pass-through	yes	yes	Matches the existing /dag/put pattern — Kubo emits its own CORS headers and we pass them through.

What's NOT in this PR

• Kubo container env changes: none needed — both endpoints are exposed by Kubo by default; only the nginx ACL was blocking them.
• Rate-limit changes: the MAX_PINS_PER_SECOND=100 env var lives in nostr-pinner/nostr_pinner.py and only gates the Nostr-driven pin queue. Direct SDK /dag/put traffic was never throttled by it — the "throttling under burst" root cause cited in sphere-sdk #370 needs more investigation. Once /dag/import is live and soak A/B has run we'll have data to drive any limiter change.
• HAProxy config: confirmed pure hostname-routing for unicity-ipfs1.dyndns.org → ipfs-kubo:443. No URL-path ACL there.

Acceptance check (post-deploy)

# Both should return HTTP 400 (healthy Kubo on empty body) — NOT 404.
curl -X POST 'https://unicity-ipfs1.dyndns.org/api/v0/dag/import'
curl -X POST 'https://unicity-ipfs1.dyndns.org/api/v0/dag/export'

Once these return 400, sphere-sdk PR #371's capability probe will start returning dagImport: true / dagExport: true on the next process start, and the fast paths will activate automatically.

Test plan

• Diff reviewed — only adds two location blocks, no changes to existing routes
• Sized via comparison with the existing /dag/put block + Kubo HTTP API docs (https://docs.ipfs.tech/reference/kubo/rpc/#api-v0-dag-import, https://docs.ipfs.tech/reference/kubo/rpc/#api-v0-dag-export)
• Build new image, deploy to unicity-ipfs1.dyndns.org, verify the curl acceptance checks pass
• Coordinate with sphere-sdk PR #371: once both are live, run soak A/B per docs/uxf/SOAK-RUNBOOK-370.md

Refs

• perf(profile/ipfs): enable CAR-batching at Unicity Kubo gateway + switch SDK to /dag/import + /dag/export unicity-sphere/sphere-sdk#370 (the SDK + operator coordination issue)
• perf(profile/ipfs)(issue-370): single-shot CAR /dag/import + /dag/export with capability probe and legacy fallback unicity-sphere/sphere-sdk#371 (the SDK-side PR — already merged-pending, capability probe + fast path + legacy fallback + 21 new tests)
• https://docs.ipfs.tech/reference/kubo/rpc/#api-v0-dag-import
• https://docs.ipfs.tech/reference/kubo/rpc/#api-v0-dag-export

[gemini-code-assist]

Code Review

This pull request adds Nginx location blocks for /api/v0/dag/import and /api/v0/dag/export to support CAR-batched UXF Profile pushes and fetches. The feedback suggests optimizing these endpoints for streaming large CAR files by explicitly configuring proxy_http_version 1.1 and disabling request buffering on imports.

[gemini-code-assist]

By default, Nginx buffers the entire request body in memory or on disk before forwarding it to the upstream server. For large CAR file uploads (up to 100MB), this introduces significant latency because Kubo cannot start processing/pinning the blocks until the upload to Nginx is fully complete. It also causes unnecessary disk I/O overhead on the proxy layer.

Disabling request buffering with proxy_request_buffering off; and enabling proxy_http_version 1.1; allows Nginx to stream the request body directly to Kubo as it is received from the client.

        location /api/v0/dag/import {
            client_max_body_size 100M;
            proxy_http_version 1.1;
            proxy_request_buffering off;
            proxy_pass http://127.0.0.1:5001;
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_read_timeout 300s;
            proxy_send_timeout 300s;
            proxy_pass_header Access-Control-Allow-Origin;
            proxy_pass_header Access-Control-Allow-Methods;
            proxy_pass_header Access-Control-Allow-Headers;
            proxy_pass_header Access-Control-Expose-Headers;
        }

[gemini-code-assist]

For streaming large CAR files via /api/v0/dag/export, it is highly recommended to explicitly set proxy_http_version 1.1;. This ensures native support for HTTP/1.1 chunked transfer encoding, which is standard for streaming responses of unknown length.

        location /api/v0/dag/export {
            proxy_pass http://127.0.0.1:5001;
            proxy_http_version 1.1;
            proxy_buffering off;
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_read_timeout 300s;
            proxy_send_timeout 300s;
            proxy_pass_header Access-Control-Allow-Origin;
            proxy_pass_header Access-Control-Allow-Methods;
            proxy_pass_header Access-Control-Allow-Headers;
            proxy_pass_header Access-Control-Expose-Headers;
        }

[vrogojin]

Deployed via hot-update — soak readiness confirmed.

Per coordination on sphere-sdk PR #371, I hot-updated the production container to unblock the issue #370 A/B soak:

1. docker cp config/nginx.conf.template ipfs-kubo:/etc/nginx/nginx.conf.template
2. Re-rendered via the same envsubst invocation configure_nginx() uses (DOMAIN=ipfs.unicity.network, SSL_CERT_PATH/SSL_KEY_PATH from /etc/letsencrypt/live/ipfs.unicity.network/)
3. Re-merged /tmp/.ssl-alias-nginx.conf per the entrypoint flow
4. nginx -t clean, nginx -s reload

Verification (both domain aliases):

=== https://unicity-ipfs1.dyndns.org ===
  /dag/import     = 400
  /dag/export     = 400
  /dag/put        = 400
  /version        = 200

=== https://ipfs.unicity.network ===
  /dag/import     = 400
  /dag/export     = 400
  /dag/put        = 400
  /version        = 200

400 (healthy Kubo on empty body) is the expected positive signal — sphere-sdk's capability probe accepts any 4xx ≠ 404/405 as "endpoint exposed".

The hot-update is ephemeral. If the container restarts before this PR merges + the image is rebuilt, the change is lost (next entrypoint render reads the old template from the image layer). Backups:

• /etc/nginx/nginx.conf.template.bak-pre-370 (old template)
• /etc/nginx/nginx.conf.bak-pre-370 (old rendered config)

Rollback if needed:

docker exec ipfs-kubo cp /etc/nginx/nginx.conf.bak-pre-370 /etc/nginx/nginx.conf
docker exec ipfs-kubo nginx -s reload

Recommend: merge this PR, rebuild + redeploy the image at the next regular maintenance window for durable persistence. Soak A/B can run against the hot-updated gateway in the meantime.