Skip to content

feat(nginx)(sphere-sdk-#370): expose /api/v0/dag/import + /api/v0/dag/export #15

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
vrogojin wants to merge 2 commits into
base: main
Choose a base branch
from
Open

feat(nginx)(sphere-sdk-#370): expose /api/v0/dag/import + /api/v0/dag/export #15

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
dac9cf7
feat(nginx)(sphere-sdk-#370): expose /api/v0/dag/import + /api/v0/dag…
vrogojin May 31, 2026
ae05807
perf(ipfs): tighten ConnMgr/ResourceMgr + RelayService off + StorageM…
vrogojin Jun 1, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
58 changes: 58 additions & 0 deletions config/nginx.conf.template
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -239,6 +239,64 @@ http {
proxy_pass_header Access-Control-Expose-Headers;
}

# DAG import endpoint for CAR-batched UXF Profile pushes
# (sphere-sdk #370 Part 1). Replaces the per-block /dag/put loop
# with a SINGLE multipart POST of the full CAR — eliminates the
# burst of N parallel /dag/put requests during §D.1 pre-clear
# snapshots. Capability-probed once per process by the SDK at
# startup; absence here (404 or 405) makes the SDK transparently
# fall back to the per-block /dag/put path above.
#
# client_max_body_size: 100M — CAR-batched bundles carry the
# full pin payload in a single body (vs ~50KB-1MB per individual
# block on /dag/put). Sized generously above the typical Profile
# snapshot (~few MB) to avoid 413 truncation on large wallets.
#
# Timeouts: 300s — Kubo's internal pin of every block in a
# large CAR takes several seconds end-to-end; allow headroom
# over the typical ~1-2s import time so a slow inner CAR walk
# doesn't surface as a spurious gateway-side timeout that
# triggers the SDK's legacy fallback unnecessarily.
location /api/v0/dag/import {
client_max_body_size 100M;
proxy_pass http://127.0.0.1:5001;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_read_timeout 300s;
proxy_send_timeout 300s;
proxy_pass_header Access-Control-Allow-Origin;
proxy_pass_header Access-Control-Allow-Methods;
proxy_pass_header Access-Control-Allow-Headers;
proxy_pass_header Access-Control-Expose-Headers;
}
Comment on lines +260 to +271

@gemini-code-assist gemini-code-assist Bot May 31, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

By default, Nginx buffers the entire request body in memory or on disk before forwarding it to the upstream server. For large CAR file uploads (up to 100MB), this introduces significant latency because Kubo cannot start processing/pinning the blocks until the upload to Nginx is fully complete. It also causes unnecessary disk I/O overhead on the proxy layer.

Disabling request buffering with proxy_request_buffering off; and enabling proxy_http_version 1.1; allows Nginx to stream the request body directly to Kubo as it is received from the client.

        location /api/v0/dag/import {
            client_max_body_size 100M;
            proxy_http_version 1.1;
            proxy_request_buffering off;
            proxy_pass http://127.0.0.1:5001;
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_read_timeout 300s;
            proxy_send_timeout 300s;
            proxy_pass_header Access-Control-Allow-Origin;
            proxy_pass_header Access-Control-Allow-Methods;
            proxy_pass_header Access-Control-Allow-Headers;
            proxy_pass_header Access-Control-Expose-Headers;
        }


# DAG export endpoint for CAR-batched UXF Profile fetches
# (sphere-sdk #370 Part 1). Replaces the per-block BFS walk
# (one /api/v0/block/get per block) with a single CAR stream
# rooted at the requested CID. Capability-probed by the SDK;
# absence here transparently falls the SDK back to the BFS
# path above.
#
# proxy_buffering off: response CARs can be tens of MB and
# are content-addressed (verified end-to-end by the receiver).
# Buffering provides no value and would pile bytes up in
# nginx's worker memory under concurrent fetches.
#
# No client_max_body_size needed — the request body is the
# query string (`?arg=<cid>`); the bytes flow the other way.
location /api/v0/dag/export {
proxy_pass http://127.0.0.1:5001;
proxy_buffering off;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_read_timeout 300s;
proxy_send_timeout 300s;
proxy_pass_header Access-Control-Allow-Origin;
proxy_pass_header Access-Control-Allow-Methods;
proxy_pass_header Access-Control-Allow-Headers;
proxy_pass_header Access-Control-Expose-Headers;
}
Comment on lines +287 to +298

@gemini-code-assist gemini-code-assist Bot May 31, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

For streaming large CAR files via /api/v0/dag/export, it is highly recommended to explicitly set proxy_http_version 1.1;. This ensures native support for HTTP/1.1 chunked transfer encoding, which is standard for streaming responses of unknown length.

        location /api/v0/dag/export {
            proxy_pass http://127.0.0.1:5001;
            proxy_http_version 1.1;
            proxy_buffering off;
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_read_timeout 300s;
            proxy_send_timeout 300s;
            proxy_pass_header Access-Control-Allow-Origin;
            proxy_pass_header Access-Control-Allow-Methods;
            proxy_pass_header Access-Control-Allow-Headers;
            proxy_pass_header Access-Control-Expose-Headers;
        }


# Safe API proxy for content preloading (refs triggers fetch via bitswap)
location /api/v0/refs {
client_max_body_size 10M;
Expand Down
57 changes: 33 additions & 24 deletions scripts/configure-ipfs.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,26 +42,26 @@ su -s /bin/sh ipfs -c "IPFS_PATH=/data/ipfs $IPFS_CMD config Addresses.API /ip4/
# Bind Gateway to all interfaces (within container)
su -s /bin/sh ipfs -c "IPFS_PATH=/data/ipfs $IPFS_CMD config Addresses.Gateway /ip4/0.0.0.0/tcp/8080"

# Enable relay client and service with resource caps
# Relay: keep CLIENT (used in case we're behind NAT), DISABLE SERVICE (we don't
# forward other peers' traffic — that's pure overhead for a wallet gateway).
su -s /bin/sh ipfs -c "IPFS_PATH=/data/ipfs $IPFS_CMD config --json Swarm.RelayClient.Enabled true"
su -s /bin/sh ipfs -c "IPFS_PATH=/data/ipfs $IPFS_CMD config --json Swarm.RelayService '{
\"Enabled\": true,
\"ConnectionDurationLimit\": \"2m0s\",
\"ConnectionDataLimit\": 131072,
\"ReservationTTL\": \"60m\",
\"MaxReservations\": 64,
\"MaxCircuits\": 16,
\"BufferSize\": 4096,
\"MaxReservationsPerIP\": 8,
\"MaxReservationsPerASN\": 32
}'"

# Connection manager: cap connections with graceful trimming
# LowWater/HighWater ratio kept gentle (75%) to avoid connection storm churn
su -s /bin/sh ipfs -c "IPFS_PATH=/data/ipfs $IPFS_CMD config --json Swarm.RelayService.Enabled false"

# Connection manager: aggressive cap for "API-first, light p2p" profile.
# LowWater 16 / HighWater 64 = node maintains a small mesh for occasional
# cross-peer bitswap, doesn't try to be a public IPFS supernode.
# GracePeriod 30s = ConnMgr prunes new peers 4× faster than the default 2m.
su -s /bin/sh ipfs -c "IPFS_PATH=/data/ipfs $IPFS_CMD config --json Swarm.ConnMgr.Type '\"basic\"'"
su -s /bin/sh ipfs -c "IPFS_PATH=/data/ipfs $IPFS_CMD config --json Swarm.ConnMgr.LowWater 96"
su -s /bin/sh ipfs -c "IPFS_PATH=/data/ipfs $IPFS_CMD config --json Swarm.ConnMgr.HighWater 128"
su -s /bin/sh ipfs -c "IPFS_PATH=/data/ipfs $IPFS_CMD config --json Swarm.ConnMgr.GracePeriod '\"2m\"'"
su -s /bin/sh ipfs -c "IPFS_PATH=/data/ipfs $IPFS_CMD config --json Swarm.ConnMgr.LowWater 16"
su -s /bin/sh ipfs -c "IPFS_PATH=/data/ipfs $IPFS_CMD config --json Swarm.ConnMgr.HighWater 64"
su -s /bin/sh ipfs -c "IPFS_PATH=/data/ipfs $IPFS_CMD config --json Swarm.ConnMgr.GracePeriod '\"30s\"'"

# DHT: leave as "auto" (Kubo modern default: server when publicly reachable,
# client otherwise). Earlier experiments with "dhtclient" broke the public
# `/ipfs/<CID>` gateway route — content provider discovery for non-local
# CIDs timed out because client mode degrades the findProviders walk. The
# auto-mode load is small enough on this host (~5-15% CPU steady state).
su -s /bin/sh ipfs -c "IPFS_PATH=/data/ipfs $IPFS_CMD config Routing.Type auto"

# Add Unicity bootstrap peers
echo "[IPFS] Adding Unicity bootstrap peers..."
Expand All @@ -70,6 +70,15 @@ su -s /bin/sh ipfs -c "IPFS_PATH=/data/ipfs $IPFS_CMD bootstrap add /dns4/unicit
su -s /bin/sh ipfs -c "IPFS_PATH=/data/ipfs $IPFS_CMD bootstrap add /dns4/unicity-ipfs3.dyndns.org/tcp/4001/p2p/12D3KooWQ4aujVE4ShLjdusNZBdffq3TbzrwT2DuWZY9H1Gxhwn6" || true
su -s /bin/sh ipfs -c "IPFS_PATH=/data/ipfs $IPFS_CMD bootstrap add /dns4/unicity-ipfs3.dyndns.org/tcp/4003/wss/p2p/12D3KooWQ4aujVE4ShLjdusNZBdffq3TbzrwT2DuWZY9H1Gxhwn6" || true

# === DATASTORE: storage ceiling ===
# Kubo's default StorageMax is 10 GB — far below the working set of a real
# gateway. Once RepoSize exceeds StorageMax, kubo enters a continuous GC
# reclaim loop that fights every pin write (and can cause "silent pin drops"
# where the API reports success but bytes are GC'd before the next reader
# can fetch them). 750 GB matches the operator's disk headroom while
# keeping the cap as a defensive ceiling against runaway growth.
su -s /bin/sh ipfs -c "IPFS_PATH=/data/ipfs $IPFS_CMD config Datastore.StorageMax 750GB"

# === RESOURCE MANAGER: cap system-wide resources ===
# Kubo 0.19+ removed Swarm.ResourceMgr.Limits — use libp2p-resource-limit-overrides.json instead
echo "[IPFS] Configuring resource manager limits..."
Expand All @@ -78,12 +87,12 @@ su -s /bin/sh ipfs -c "IPFS_PATH=/data/ipfs $IPFS_CMD config --json Swarm.Resour
cat > /data/ipfs/libp2p-resource-limit-overrides.json << 'RESLIMITS'
{
"System": {
"Conns": 512,
"ConnsInbound": 256,
"ConnsOutbound": 256,
"Streams": 2048,
"StreamsInbound": 1024,
"StreamsOutbound": 1024,
"Conns": 192,
"ConnsInbound": 128,
"ConnsOutbound": 64,
"Streams": 1024,
"StreamsInbound": 512,
"StreamsOutbound": 512,
"FD": 4096,
"Memory": 1610612736
}
Expand Down