Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions .dockerignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
.venv*
__pycache__/
*.pyc
.git/
.github/
tests/
.env
.env.example
data/
*.md
docker-compose.yml
5 changes: 5 additions & 0 deletions .github/dependabot.yml
Original file line number Diff line number Diff line change
Expand Up @@ -9,3 +9,8 @@ updates:
directory: "/"
schedule:
interval: "weekly"

- package-ecosystem: "docker"
directory: "/"
schedule:
interval: "weekly"
96 changes: 96 additions & 0 deletions .github/workflows/deploy.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,96 @@
name: Deploy

# Builds the api image and deploys it to Datum Cloud Compute on every push to
# main (i.e. after a PR merges). See DEPLOY.md for the one-time setup this
# depends on (Datum project, service account, Secrets, DATABASE_URL).
on:
push:
branches: [main]
workflow_dispatch: {}

concurrency:
group: deploy-api
cancel-in-progress: false

env:
IMAGE: ghcr.io/altsmanager/api

jobs:
build-and-push:
runs-on: ubuntu-latest
permissions:
contents: read
packages: write
outputs:
image: ${{ steps.image.outputs.image }}
steps:
- uses: actions/checkout@v4

- name: Log in to GHCR
uses: docker/login-action@v3
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}

- uses: docker/setup-buildx-action@v3

- name: Build and push
id: push
uses: docker/build-push-action@v6
with:
context: .
push: true
tags: ${{ env.IMAGE }}:${{ github.sha }}
cache-from: type=gha
cache-to: type=gha,mode=max

# Deploy by digest, not tag -- a `datumctl compute deploy` re-run
# against the same tag didn't reliably pick up a newly-pushed image
# during manual testing (see DEPLOY.md "Gotchas" #5).
- name: Set deploy image reference
id: image
run: echo "image=${IMAGE}@${{ steps.push.outputs.digest }}" >> "$GITHUB_OUTPUT"

deploy:
needs: build-and-push
runs-on: ubuntu-latest
env:
DATUM_PROJECT: altsmanager
steps:
- uses: actions/setup-go@v5
with:
go-version: "1.23"

# datumctl is a published Go module (go.datum.net/datumctl); there's no
# separate install script as of this writing.
- name: Install datumctl
run: go install go.datum.net/datumctl@latest

# Service-account login per https://www.datum.net/docs/datumctl/auth/logging-in.
# The service account must be scoped to exactly this one project --
# datum-cloud/datumctl#324 is an open bug where --credentials login fails
# non-interactively if the account can see more than one context.
- name: Authenticate to Datum Cloud
env:
DATUM_SERVICE_ACCOUNT_KEY: ${{ secrets.DATUM_SERVICE_ACCOUNT_KEY }}
run: |
umask 077
printf '%s' "$DATUM_SERVICE_ACCOUNT_KEY" > "$RUNNER_TEMP/datum-key.json"
datumctl login --credentials "$RUNNER_TEMP/datum-key.json"
rm -f "$RUNNER_TEMP/datum-key.json"

# --runtime-class is required and immutable: the platform default
# (unikernel) can't run a plain `docker build` image. See DEPLOY.md
# "Gotchas" for this and the other flags below -- all verified against
# a real deployment.
- name: Deploy to Datum Compute
run: |
datumctl compute deploy altsmanager-api \
--image="${{ needs.build-and-push.outputs.image }}" \
--runtime-class=general-purpose \
--city=DFW \
--min=1 \
--http-port=8000 \
--env-from-secret=altsmanager-api-secrets \
--env=ENVIRONMENT=production
5 changes: 5 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -214,3 +214,8 @@ __marimo__/

# Locally-stored uploaded documents (STORAGE_BACKEND=local)
/data/

# Live-state snapshot saved locally by `datumctl compute deploy` -- not clean
# source (resourceVersion/managedFields/status churn on every deploy); the
# deploy pipeline manages the workload declaratively via deploy.yml's flags.
/workload.yaml
103 changes: 103 additions & 0 deletions DEPLOY.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,103 @@
# Deploying to Datum Cloud

This repo deploys to [Datum Cloud](https://datum.net) Compute. `api` is live
at **https://grain-noodle-m23qc.datumproxy.net** (health: `/health`, docs:
`/docs`, admin: `/admin`). `.github/workflows/deploy.yml` automates the same
`datumctl compute deploy` flow on every push to `main`, once its one-time
setup (below) is done.

## Status

- [x] Datum Cloud project `altsmanager` (org: Humans of the Internet /
`joe-company-inc-3a1tyj`), Compute enabled
- [x] Postgres: Supabase (session pooler, port 5432 -- not the transaction
pooler on 6543, which doesn't play well with SQLAlchemy's plain pool /
alembic), schema migrated, decided over self-hosting since Datum Compute
has no persistent disk
- [x] Document storage: Wasabi bucket `altsmanager-documents`
(`https://s3.wasabisys.com`, `us-east-1`, versioned), same reasoning as
Postgres
- [x] `altsmanager-api-secrets` Secret created (`DATABASE_URL`, `SECRET_KEY`,
`STORAGE_BACKEND`, `S3_*`) via `datumctl apply -f` -- native `Secret`
objects aren't reachable through the datum-mcp `resource` tool, only
`datumctl`/the console
- [x] First manual deploy done and verified end-to-end (see "Gotchas" below
for what it took to get there)
- [ ] Datum service account for CI, scoped to only the `altsmanager` project
(manual deploys so far used a user session, not a service account)
- [ ] GitHub repo secrets/variables (list below) -- `deploy.yml` itself is
still unexercised by actual CI

## Architecture

```
GitHub (push to main)
-> build image, push to ghcr.io/altsmanager/api (must be a PUBLIC package --
Datum Compute has no documented imagePullSecrets flag on `compute deploy`)
-> datumctl compute deploy altsmanager-api --runtime-class=general-purpose --http-port=8000

Datum Compute workload "altsmanager-api" (general-purpose runtime class)
-> DATABASE_URL secret -> Supabase Postgres (external, session pooler)
-> S3_* secrets -> Wasabi bucket (external)
-> public HTTPS hostname via --http-port (Datum-generated *.datumproxy.net;
api.altsmanager.org not yet attached as a custom hostname)
```

Postgres and document storage are **not** Datum Compute workloads -- both need
persistent state Datum Compute's container runtime doesn't offer, so they're
external managed services the api workload connects to.

## Gotchas found getting this running (read before touching the Dockerfile/deploy flags)

1. **Runtime class must be `general-purpose`, not the platform-default
`unikernel`.** `unikernel` (Unikraft) only runs specially-built
position-independent binaries; a plain `docker build` image won't boot
under it (manifests as `ImageUnavailable`-looking failures, not an obvious
"wrong runtime" error). **Runtime class is immutable once a workload
exists** -- getting it wrong means delete + recreate. Always pass
`--runtime-class=general-purpose` explicitly.
2. **The container must bind to `::` (IPv6 wildcard), never `0.0.0.0`.**
Datum Compute's network is IPv6-only -- an instance has no IPv4 address at
all. Binding to `0.0.0.0` looks completely healthy in the app's own logs
(and even in `compute_workload_diagnose`) while the ALB gets instant
connection-refused on every request. The Dockerfile's `CMD` passes
`--host ::` to uvicorn for this reason.
3. **Trust the edge's proxy headers**, or redirects break. Datum's edge
terminates TLS and forwards plain HTTP to the container, but uvicorn
doesn't know that by default, so it generates `Location: http://...`
redirects -- which a browser correctly refuses to follow from an HTTPS
page. Fixed with `--proxy-headers --forwarded-allow-ips='*'` (safe here:
the container has no other public ingress path than Datum's edge).
4. **GHCR packages are private by default.** Datum Compute can't pull a
private image anonymously, and `datumctl compute deploy` has no
`imagePullSecrets`-equivalent flag. Make the package public: org ->
Packages -> the package -> Package settings -> Change visibility.
5. **Prefer deploying by image digest** (`ghcr.io/owner/image@sha256:...`)
over a mutable tag. A `datumctl compute deploy` re-run with the *same tag*
did not reliably pick up a newly-pushed image during testing here; pinning
the digest forced a correct pull every time.
6. **After a redeploy, give the edge real time to settle before concluding
something is broken.** Even with everything else correct, the ALB
reported `serving: true` / `confidence: unverified` while still returning
503s with `UF` (upstream connection failure) response flags for over a
minute after a clean rollout -- `alb_traffic_summary` and
`compute_workload_diagnose` are the right tools to check (they'll show the
app/workload as genuinely healthy while the edge path is still catching
up), not a sign the deploy itself failed.

## One-time setup (remaining)

1. **Create a Datum service account** scoped to only the `altsmanager`
project (see note in `deploy.yml` about datum-cloud/datumctl#324), download
its credentials JSON, and set it as the `DATUM_SERVICE_ACCOUNT_KEY` GitHub
Actions secret in this repo.
2. **Run the `Deploy` workflow** (push to `main`, or `workflow_dispatch`) and
confirm it reaches the same state the manual deploys already did.

## Required GitHub secrets/variables

| Name | Type | Purpose |
|---|---|---|
| `DATUM_SERVICE_ACCOUNT_KEY` | secret | Service account credentials JSON for `datumctl login --credentials` |

(`GITHUB_TOKEN` for GHCR push is automatic, no setup needed.)
19 changes: 19 additions & 0 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
FROM python:3.12-slim

WORKDIR /app
ENV PYTHONDONTWRITEBYTECODE=1 \
PYTHONUNBUFFERED=1

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY alembic.ini .
COPY alembic ./alembic
COPY app ./app
COPY scripts ./scripts

RUN useradd --create-home --uid 1000 appuser && chown -R appuser:appuser /app
USER appuser

EXPOSE 8000
CMD ["sh", "-c", "alembic upgrade head && exec uvicorn app.main:app --host :: --port 8000 --proxy-headers --forwarded-allow-ips='*'"]
35 changes: 34 additions & 1 deletion app/admin.py
Original file line number Diff line number Diff line change
@@ -1,12 +1,17 @@
import os
import uuid
from typing import Any

from fastapi import FastAPI
from sqladmin import Admin, ModelView
from sqladmin.authentication import AuthenticationBackend
from starlette.requests import Request
from wtforms import PasswordField
from wtforms.form import Form as WTForm
from wtforms.validators import Optional as OptionalField

from app.core.config import get_settings
from app.core.security import verify_password
from app.core.security import hash_password, verify_password
from app.crud.user import get_user_by_email
from app.db.session import SessionLocal, engine
from app.models.counterparty import Counterparty
Expand Down Expand Up @@ -55,11 +60,15 @@ async def authenticate(self, request: Request) -> bool:


class EntityAdmin(ModelView, model=Entity):
name = "Entity"
name_plural = "Entities"
column_list = [Entity.id, Entity.name, Entity.entity_type, Entity.jurisdiction]
form_excluded_columns = [Entity.created_at, Entity.updated_at]


class CounterpartyAdmin(ModelView, model=Counterparty):
name = "Counterparty"
name_plural = "Counterparties"
column_list = [
Counterparty.id,
Counterparty.name,
Expand All @@ -82,6 +91,8 @@ class InvestmentAdmin(ModelView, model=Investment):


class InvestmentCounterpartyAdmin(ModelView, model=InvestmentCounterparty):
name = "Investment Counterparty"
name_plural = "Investment Counterparties"
column_list = [
InvestmentCounterparty.id,
InvestmentCounterparty.investment_id,
Expand Down Expand Up @@ -130,6 +141,27 @@ class UserAdmin(ModelView, model=User):
form_excluded_columns = [User.hashed_password, User.created_at, User.updated_at]
can_delete = False

async def scaffold_form(self, rules: list[str] | None = None) -> type[WTForm]:
form_class = await super().scaffold_form(rules)
form_class.password = PasswordField(
"Password",
validators=[OptionalField()],
description="Required when creating a user. Leave blank on edit to keep the current password.",
)
return form_class

async def on_model_change(
self, data: dict[str, Any], model: Any, is_created: bool, request: Request
) -> None:
if isinstance(data.get("email"), str):
data["email"] = data["email"].strip()

password = data.pop("password", None)
if password:
data["hashed_password"] = hash_password(password)
elif is_created:
raise ValueError("Password is required when creating a user.")


def mount_admin(app: FastAPI) -> Admin:
settings = get_settings()
Expand All @@ -139,6 +171,7 @@ def mount_admin(app: FastAPI) -> Admin:
authentication_backend=AdminAuth(secret_key=settings.secret_key),
base_url="/admin",
title="AltsManager Admin",
templates_dir=os.path.join(os.path.dirname(__file__), "admin_templates"),
)
for view in (
EntityAdmin,
Expand Down
53 changes: 53 additions & 0 deletions app/admin_templates/sqladmin/base.html
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
{% extends "sqladmin_original/base.html" %}
{% block head_tail %}
<style>
:root {
--tblr-primary: #13263f;
--tblr-primary-rgb: 19, 38, 63;
}

body {
background-color: #f4f5f7;
}

.navbar.navbar-vertical {
background: linear-gradient(180deg, #0d1b2e 0%, #13263f 100%) !important;
}

.navbar-brand-autodark h3 {
color: #c9a227;
font-weight: 700;
letter-spacing: 0.06em;
text-transform: uppercase;
font-size: 1.1rem;
margin: 0;
}

.navbar-vertical .nav-link.active {
background-color: rgba(201, 162, 39, 0.15);
border-left: 3px solid #c9a227;
color: #fff !important;
}

.btn-primary {
background-color: #13263f;
border-color: #13263f;
}

.btn-primary:hover,
.btn-primary:focus {
background-color: #0d1b2e;
border-color: #0d1b2e;
}

.page-title {
font-weight: 600;
color: #13263f;
}

.card {
border: 1px solid #e6e8eb;
box-shadow: 0 1px 3px rgba(16, 24, 40, 0.06);
}
</style>
{% endblock %}
Loading
Loading