Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 43 additions & 0 deletions .github/workflows/build-image.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
name: Build container image

on:
push:
branches: [main, deploy/*]
pull_request:
branches: [main]

permissions:
contents: read
packages: write

jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4

- uses: docker/login-action@v3
if: github.event_name == 'push'
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}

- name: Docker meta
id: meta
uses: docker/metadata-action@v5
with:
images: ghcr.io/${{ github.repository }}
tags: |
type=sha,prefix=
type=raw,value=latest,enable={{is_default_branch}}
type=ref,event=branch

- name: Build and push
uses: docker/build-push-action@v6
with:
context: .
push: ${{ github.event_name == 'push' }}
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
platforms: linux/amd64
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ gwcfg
kuri
settings.env
.env
secrets.env
*~
.idea/

Expand Down
129 changes: 129 additions & 0 deletions deploy/hetzner-test/QUICKSTART.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,129 @@
# filecoin-gateway on Hetzner (FCOS + k3s)

## provision

```bash
scp deploy/hetzner-test/{config.bu,install.sh} root@RESCUE_IP:/tmp/
ssh root@RESCUE_IP

curl -sLO https://github.com/coreos/butane/releases/latest/download/butane-x86_64-unknown-linux-gnu
mv butane-x86_64-unknown-linux-gnu /usr/local/bin/butane && chmod +x /usr/local/bin/butane

export SSH_PUBKEY="ssh-ed25519 AAAA... you@host"
envsubst '$SSH_PUBKEY' < /tmp/config.bu | butane --strict > /tmp/config.ign
bash /tmp/install.sh /tmp/config.ign
reboot
```

## first boot

FCOS boots, layers packages (tmux, btop, rclone, bc), reboots once,
then installs k3s, clones repo, builds image, imports into k3s containerd.

```bash
ssh core@SERVER_IP

# check first-boot services (may need to wait for the package reboot)
journalctl -u rpm-ostree-overlay.service
journalctl -u install-k3s.service
journalctl -u setup-fgw.service
kubectl get nodes
```

## configure

```bash
# create wallet + CIDGravity account + staging config
podman run --rm -it \
-v /var/mnt/fgw/config:/config:Z \
-v /var/mnt/fgw/wallet:/root/.ribswallet:Z \
localhost/fgw:local ./gwcfg -f /config/settings.env

# create wallet secret from generated keys
kubectl create secret generic wallet-keys \
--from-file=/var/mnt/fgw/wallet/ \
-n filecoin-gateway

# create secrets.env from template
cd /opt/filecoin-gateway
cp deploy/hetzner-test/k3s/secrets.env.example deploy/hetzner-test/k3s/secrets.env
# edit with CIDGravity token from gwcfg output
$EDITOR deploy/hetzner-test/k3s/secrets.env

# edit configmap if needed (EXTERNAL_LOCALWEB_URL, fallback providers, etc)
$EDITOR deploy/hetzner-test/k3s/configmap.yaml
```

## deploy

```bash
kubectl apply -k deploy/hetzner-test/k3s/

kubectl -n filecoin-gateway get pods -w
kubectl -n filecoin-gateway logs -f deployment/kuri
```

## DNS (required for deal-making)

kuri's builtin autocert provisions a Let's Encrypt cert on port 443,
which SPs use to pull CAR data. point a DNS A record at the server IP
with no proxy (no cloudflare orange cloud):

```
fgw-test.yourdomain.com → A → SERVER_IP
```

set `EXTERNAL_LOCALWEB_URL=https://fgw-test.yourdomain.com` in the configmap.

## test

```bash
# S3 is accessible via localhost port-forward
kubectl -n filecoin-gateway port-forward deployment/kuri 8078:8078 &

curl -s -o /dev/null -w '%{http_code}' http://localhost:8078/healthz

echo "hello" | rclone rcat fgw:test/hello.txt
rclone cat fgw:test/hello.txt

# upload test workload
./test-workload-gen.sh /var/mnt/testdata/upload
rclone copy /var/mnt/testdata/upload fgw:testbucket/ --transfers 8 --progress
```

## operations

```bash
# logs
kubectl -n filecoin-gateway logs deployment/kuri --tail=50
kubectl -n filecoin-gateway logs statefulset/yugabyte --tail=50

# dashboard (ssh tunnel)
ssh -L 9010:localhost:9010 core@SERVER_IP
# then http://localhost:9010

# restart kuri
kubectl -n filecoin-gateway rollout restart deployment/kuri

# update config
$EDITOR deploy/hetzner-test/k3s/configmap.yaml
kubectl apply -k deploy/hetzner-test/k3s/
kubectl -n filecoin-gateway rollout restart deployment/kuri

# update image (when maintainers ship a release, pull from ghcr instead)
cd /opt/filecoin-gateway && git pull
podman build . -t fgw:local
podman save localhost/fgw:local | sudo k3s ctr images import -
kubectl -n filecoin-gateway rollout restart deployment/kuri
```

## storage / redundancy

this deployment uses k3s `local-path` provisioner -- PVCs are
directories on the host NVMe with no replication or snapshots.

**storage redundancy is the operator's responsibility.** consider:
- RAID1 for OS disk, separate data disk(s) for block groups
- external block storage with snapshots (e.g. hetzner volumes)
- YugabyteDB is single-replica here; production should be RF=3

Loading