Kubernetes Terraform Modules

Reusable Terraform modules for a private, single-node Kubernetes platform on a Netcup RS 1000 G12 running Debian 13 (Trixie), k3s, and k3s-packaged Traefik.

This repository is a module library. It deliberately has no root Terraform configuration, backend, state, environment values, or apply/destroy scripts. Those belong in a consumer repository. Pin every module used by one environment to the same release tag rather than a moving branch.

Module Catalog

Module	Purpose
setup_twingate_connector	Creates the Twingate remote network, host connector, and bootstrap tokens.
provision_server	Discovers an existing Netcup RS 1000 G12 through the SCP API, installs Debian 13, configures its firewall and Twingate connector, and waits for SSH.
setup_twingate_access	Creates private Twingate resources for SSH and the k3s API.
setup_cluster	Configures Debian, installs pinned k3s, publishes OIDC metadata, configures Entra authentication and workload identity, and exports credentials.
setup_ingress_controller	Configures k3s-packaged Traefik, Gateway API, Cloudflare DNS/security, and origin TLS.
setup_monitoring_crds	Installs monitoring CRDs before charts that create monitoring resources.
create_postgres_database	Installs PostgreSQL with a retained host-path volume and ServiceMonitor.
setup_postgres_schema	Provisions a dedicated login role and owned schema for one application.
setup_redis	Installs shared Redis with a retained host-path volume.
register_web	Registers confidential Web clients for server-side ForwardAuth, with OIDC consent, optional API access and required user assignments.
register_spa	Registers legacy browser-OAuth SPA clients.
setup_app_forwardauth	Provisions per-app stock oauth2-proxy sessions/Secrets and exports the proxy reference for application charts.
setup_victoria_metrics	Installs VictoriaMetrics, Grafana, exporters, and Grafana ingress.
setup_victoria_logs	Installs Alloy pod-log/Faro collection and Faro ingress.
setup_app_base	Provisions an application's namespace, vault, deployment identity and dashboard descriptor.
setup_static_website	Provisions static client document/asset storage and scoped deployment access.

The registration and application-base modules provide reusable Entra, Key Vault, namespace and deployment resources. setup_app_base exposes a non-secret dashboard_app inventory descriptor consumed by observatory-app.

Application-specific infrastructure belongs in consumer repositories. The former Hello, Cooking, Library, Learn Language, Training Log, Expense Tracker, Party, Observatory, Backup and bank email worker modules have been removed from this library. Consumers must migrate their definitions and Terraform resource addresses before upgrading to a release containing these removals. Published older tags remain available. Application workloads continue to be installed by their own repositories.

PostgreSQL Credentials

Each application's setup_postgres_schema has one password generator and one apply-time terraform_data action. A Bash helper uses the operator's existing SSH agent to run sudo -n k3s kubectl exec into PostgreSQL. The application password travels through encrypted stdin, not command arguments, temporary files, or Kubernetes objects. Administrator credentials stay inside the database container. The SQL creates/hardens the login and schema transactionally; credential outputs and app Key Vault publication wait for it to succeed.

The runner needs Bash, SSH, GNU timeout, the loaded cluster SSH key, and operator Twingate access. No local PostgreSQL client or provisioning identity is required. SSH uses StrictHostKeyChecking=accept-new: it records new hosts in the runner's known-hosts file and rejects changed keys. Preload a verified host key to avoid first-use trust; after a server rebuild, verify its new key before updating that entry. Agent forwarding is disabled. Provisioning has a ten-minute timeout and a failed apply must be retried.

Pass the same non-secret database descriptor to each database-backed app and Grafana. The deployment output orders provisioning after PostgreSQL's Helm release; the volume instance ID reruns it when storage/cluster resources are recreated:

locals {
  database = {
    host        = module.create_database.host
    port        = module.create_database.port
    name        = module.create_database.name
    jdbc_url    = module.create_database.jdbc_url
    namespace   = module.create_database.namespace
    deployment  = module.create_database.deployment
    instance_id = module.create_database.instance_id
    ssh = {
      host     = module.provision_server.ipv4_address
      port     = module.provision_server.ssh_port
      username = module.provision_server.username
    }
  }
}

The action also reruns when the password, target, SQL, or helper changes. Normal plans do not execute SQL or detect database-level drift. For a database reset outside Terraform, explicitly replace the affected action, for example terraform apply -replace=module.app_schema.terraform_data.init, using the consumer's actual schema module address. There is no destroy-time SQL: deleting Terraform resources does not drop schemas.

Upgrading removes provisioning Jobs, ConfigMaps, workload identities, and the Grafana vault introduced only for those Jobs. Password generator and application vault secret addresses remain unchanged. Replace admin_secret_name in consumer database descriptors with the fields above and pin all modules to the release containing this change together. Grafana no longer needs the four additional vault/workload-identity inputs introduced for Job provisioning.

Grafana's runtime grafana-database Secret, database administrator/exporter Secrets, backup vault credentials, and passwords in Terraform state remain. Applications must reload credentials after future password rotations.

Netcup Prerequisites

Netcup's SCP REST API manages existing contracts but cannot order a server. Order the RS 1000 G12 first, then obtain the following values from the SCP API:

• netcup_server_id from GET /api/v1/servers.
• netcup_user_id from the authenticated SCP account.
• A Debian 13 netcup_image_flavour_id from GET /api/v1/servers/{serverId}/imageflavours.
• netcup_disk_name from GET /api/v1/servers/{serverId}/disks.
• A long-lived OpenID Connect refresh token stored in a local file.

The API base URL defaults to https://www.servercontrolpanel.de/scp-core. Authentication uses Netcup's OAuth2 device-code flow, not the CCP API key, webservice password, or customer number. Request and authorize a device code:

umask 077
curl -sS -X POST \
  'https://www.servercontrolpanel.de/realms/scp/protocol/openid-connect/auth/device' \
  -d 'client_id=scp' \
  -d 'scope=offline_access openid' | tee device-response.json | jq

Open the returned verification_uri_complete, sign in to SCP, and approve the request. Exchange the authorized device code:

curl -sS -X POST \
  'https://www.servercontrolpanel.de/realms/scp/protocol/openid-connect/token' \
  -d 'grant_type=urn:ietf:params:oauth:grant-type:device_code' \
  -d 'client_id=scp' \
  --data-urlencode "device_code=$(jq -r .device_code device-response.json)" \
  | tee token-response.json | jq

REFRESH_TOKEN=$(jq -r .refresh_token token-response.json)
# Persist REFRESH_TOKEN in the caller's secret store, such as Azure Key Vault.
rm device-response.json token-response.json

Pass the current secret-store value through the sensitive netcup_refresh_token input. The module exchanges it for short-lived access tokens and uses any rotated refresh token for the remainder of the current run. Refresh-token changes are authentication data only and do not replace the firewall or server installation resources.

The Terraform runner needs curl, jq, ssh-agent, and a connected Twingate client. Netcup's current authentication instructions are available in SCP under API > REST API Docs > Authentication.

The HashiCorp HTTP provider has data sources but no managed resources. The server module uses it for authenticated discovery and validation. Apply-only terraform_data provisioners call the same REST API for SSH-key creation, image installation, task polling, firewall policy updates, and interface assignment. Destructive POSTs therefore never run during refresh or plan.

reinstall_generation is mandatory. Its first value approves erasing the selected disk; changing it intentionally reinstalls and erases that disk again. Protect Terraform state because it contains bootstrap secrets, OAuth tokens, and HTTP request metadata.

Consumer Composition

Configure Kubernetes-facing providers from setup_cluster outputs:

provider "kubernetes" {
  host                   = module.setup_cluster.k8s_host
  client_certificate     = module.setup_cluster.k8s_client_certificate
  client_key             = module.setup_cluster.k8s_client_key
  cluster_ca_certificate = module.setup_cluster.k8s_cluster_ca_certificate
}

provider "helm" {
  kubernetes = {
    host                   = module.setup_cluster.k8s_host
    client_certificate     = module.setup_cluster.k8s_client_certificate
    client_key             = module.setup_cluster.k8s_client_key
    cluster_ca_certificate = module.setup_cluster.k8s_cluster_ca_certificate
  }
}

provider "kubectl" {
  host                   = module.setup_cluster.k8s_host
  client_certificate     = module.setup_cluster.k8s_client_certificate
  client_key             = module.setup_cluster.k8s_client_key
  cluster_ca_certificate = module.setup_cluster.k8s_cluster_ca_certificate
  load_config_file       = false
}

The host and access modules deliberately form a resource-level lifecycle back-edge:

module "setup_twingate_connector" {
  source = "git::https://github.com/mucsi96/k8s-modules.git//modules/setup_twingate_connector?ref=<release-tag>"

  environment_name = var.environment_name
}

module "provision_server" {
  source = "git::https://github.com/mucsi96/k8s-modules.git//modules/provision_server?ref=<release-tag>"

  server_name             = var.server_name
  netcup_server_id        = var.netcup_server_id
  netcup_user_id          = var.netcup_user_id
  netcup_refresh_token    = var.netcup_refresh_token
  netcup_image_flavour_id = var.netcup_image_flavour_id
  netcup_disk_name        = var.netcup_disk_name
  reinstall_generation   = var.reinstall_generation
  https_source_ips        = concat(var.cloudflare_ipv4_cidrs, var.cloudflare_ipv6_cidrs)
  twingate_network        = var.twingate_network
  twingate_access_token   = module.setup_twingate_connector.access_token
  twingate_refresh_token  = module.setup_twingate_connector.refresh_token
  ssh_ready_wait_for = join(",", [
    module.setup_twingate_access.ssh_resource_id,
    module.setup_twingate_access.k8s_resource_id,
  ])
}

module "setup_twingate_access" {
  source = "git::https://github.com/mucsi96/k8s-modules.git//modules/setup_twingate_access?ref=<release-tag>"

  environment_name  = var.environment_name
  remote_network_id = module.setup_twingate_connector.remote_network_id
  k8s_host          = module.provision_server.ipv4_address
  k8s_port          = 6443
  ssh_address       = module.provision_server.ipv4_address
  ssh_port          = module.provision_server.ssh_port
}

module "setup_cluster" {
  source = "git::https://github.com/mucsi96/k8s-modules.git//modules/setup_cluster?ref=<release-tag>"

  host     = module.provision_server.ipv4_address
  ssh_port = module.provision_server.ssh_port
  username = module.provision_server.username
  wait_for = module.provision_server.ssh_ready

  server_maintenance_github_repository       = "example-infrastructure"
  server_maintenance_github_repository_owner = "example-owner"
  server_maintenance_secret_scopes = {
    host            = azurerm_key_vault_secret.host.resource_versionless_id
    ssh_port        = azurerm_key_vault_secret.ssh_port.resource_versionless_id
    ssh_private_key = azurerm_key_vault_secret.ssh_private_key.resource_versionless_id
    ssh_user_name   = azurerm_key_vault_secret.ssh_user_name.resource_versionless_id
  }
  server_maintenance_twingate_service_key = module.setup_twingate_access.service_key

  # Other Azure inputs omitted.
}

Do not replace this with module-level depends_on between provision_server and setup_twingate_access; that creates a cycle. On destroy, the graph is reversed so private access remains available while Kubernetes resources are removed.

Pass the shared Gateway contract to modules that create HTTPRoutes:

module "setup_ingress_controller" {
  # Other inputs omitted.
  k8s_config = module.setup_cluster.k8s_config
}

module "setup_victoria_metrics" {
  # Other inputs omitted.
  gateway_parent_ref  = module.setup_ingress_controller.gateway_parent_ref
  wait_for            = module.setup_ingress_controller.ingress_controller_ready
}

module "setup_victoria_logs" {
  # Other inputs omitted.
  gateway_parent_ref = module.setup_ingress_controller.gateway_parent_ref
  wait_for            = module.setup_victoria_metrics.victoria_metrics_k8s_stack_ready
}

The full creation chain is:

Twingate connector
  -> Netcup firewall and Debian installation
  -> Twingate SSH and Kubernetes API resources
  -> authenticated SSH readiness
  -> k3s cluster and OIDC publication
  -> bundled Traefik configuration and programmed Gateway
  -> monitoring CRDs
  -> PostgreSQL, Redis, VictoriaMetrics, VictoriaLogs, and applications

Platform Details

Client-owned SPA routes and build-time configuration

The client chart always owns SPA routing and document auth; API charts own API auth and header policies. setup_app_forwardauth exports a small proxy reference (proxyService, proxyPort, mode) as deployment_config.auth. Hello deployment passes that same object from app AKV forward-auth-config to each chart's auth value. Hello, Cooking and Observatory compile their public Faro app name and same-origin /_telemetry/collect path into their bundles. No client-runtime-config, policy ConfigMap or filter lists are passed. create_app_namespace grants app deployers namespace-scoped Middleware management. All app hosts use the fixed host-only __Host-session cookie with per-app secrets. Clients are hosted at /; their nested paths rewrite to /index.html.

These applications omit chart runtimeConfig and reset Helm values, so their document routes have no response-body rewriting middleware. Origins serve real files; SPA fallback remains in client-chart Gateway resources, with no-store and credential-stripping policies. enable_index_config remains available only for older consumers; it can be disabled once all live index-config middleware is gone. Deploy all three build-config clients before applying this module release: it removes their obsolete configuration secrets and secret-scoped read grants.

Set setup_victoria_logs.faro_client_hostnames to the participating application hosts. The module owns exact POST/OPTIONS collection routes, strips app credentials and rewrites to /collect. The three built clients use the fixed /_telemetry/collect path on their app host, avoiding cross-origin browser preflight. Collector origin/rate limits remain. Legacy client_log_url module inputs remain compatible for other consumers; they no longer configure these UIs.

The module pins published proxy chart 8.0.0. See the proxy module contract.

setup_cluster.k3s_version defines the minimum version installed during provisioning; the default is v1.36.4+k3s1. The server's update-server maintenance helper advances k3s to newer releases from the stable channel and prunes unused container images when invoked. Debian package maintenance also explicitly updates the host-level twingate-connector package before the remaining distribution upgrade. ServiceLB is disabled because Traefik binds host port 443 directly.

setup_cluster also creates the maintenance workflow's Entra identity, GitHub OIDC credential, secret-scoped Key Vault access, and repository Actions secrets. The caller supplies the repository identity, Twingate service key, and versionless resource IDs of the four SSH connection secrets.

The maintenance helper runs as update-server.service. Its journal is persisted on the host and the logging Alloy DaemonSet forwards only that unit to VictoriaLogs. In Grafana, use {app="update-server"} to inspect package, k3s, image-pruning, and reboot scheduling output. Each line starts with its UTC timestamp in DD/Mon/YYYY:HH:MM:SS format.

The Traefik controller and its HelmChartConfig run in kube-system. The shared Gateway and Cloudflare Origin CA secret remain in traefik. The ingress readiness token is the UID of a Gateway that has reported both Accepted=True and Programmed=True.

The Netcup firewall accepts TCP 443 from Cloudflare IPv4 and IPv6 ranges. Netcup's mandatory copied policies are preserved for DNS and installation traffic. Netcup changes its implicit ingress action to DROP when ingress rules are present. With no egress rule, implicit egress remains ACCEPT. Port 80, SSH, and the k3s API are not publicly accepted; SSH and port 6443 use Twingate. This restrictive upstream policy makes Twingate use its TCP relay path rather than direct UDP peer-to-peer connectivity.

Persistence and Reinstallation

Use database-consistent backups and verify each static host-path PV/PVC binds to its intended path before permitting writes after a restore. Changing reinstall_generation after resources exist requires destroying the in-cluster resources first and then rebuilding them after the disk erase; a single apply cannot refresh resources before and after replacing their cluster.

Consumer automation must load provision_server.ssh_private_key into its ssh-agent on every run. The module loads it during initial creation, but a new runner process cannot inherit that earlier agent state.

Verification

kubectl get nodes
kubectl get pods --all-namespaces
kubectl get gatewayclass
kubectl get gateway,httproute --all-namespaces
kubectl top node
helm list --all-namespaces

Confirm Gateway and HTTPRoute conditions include Accepted=True, Programmed=True where applicable, and ResolvedRefs=True. Confirm exactly one Traefik and one Metrics Server deployment, no ServiceLB pods, human kubelogin, workload identity, Grafana metrics/logs, Faro ingestion, and application data. From outside Twingate verify that ports 22, the randomized SSH port, 6443, and 80 are closed and that direct port 443 connections outside Cloudflare are blocked for both IPv4 and IPv6.

Safe Destruction

Use a full destroy plan from the consumer. Do not routinely target the server, cluster, firewall, access resources, or individual prerequisites.

terraform plan -destroy -out=destroy.tfplan
terraform show destroy.tfplan
terraform apply destroy.tfplan

The SCP API cannot cancel the server contract. The server module intentionally does not detach or delete its firewall policy on Terraform destroy, because doing so would expose an externally-owned server. Remove the policy manually only after the contract is intentionally retired.

Development

CI runs formatting, Terraform validation, Bash/Ansible syntax checks, and the focused Netcup authentication helper test.

terraform fmt -check -diff -recursive modules

for dir in modules/*/; do
  terraform -chdir="$dir" init -backend=false
  terraform -chdir="$dir" validate
done

bash -n modules/provision_server/netcup_api.sh modules/provision_server/netcup_auth.sh modules/setup_postgres_schema/provision.sh
bash modules/provision_server/test_netcup_auth.sh

ansible-galaxy collection install -r requirements.yml
ansible-playbook --syntax-check -i 'localhost,' modules/setup_cluster/system_update.yaml
ansible-playbook --syntax-check -i 'localhost,' modules/setup_cluster/configure_dns.yaml
ansible-playbook --syntax-check -i 'localhost,' modules/setup_cluster/install_k3s.yaml
ansible-playbook --syntax-check -i 'localhost,' modules/setup_cluster/publish_k3s_oidc.yaml

Install Python dependencies from requirements.txt. The OIDC publication controller also requires the azwi executable.

Static client origins

setup_static_website provisions public build-resource hosting on a dedicated Azure Storage account with container-scoped Entra upload permissions. Consumers supply per-app storage account names, reusing the same website/Blob hosting and scoped deployment-secret contract. Enable setup_ingress_controller.enable_static_origins and deploy the dedicated document-only static-client-app 2.0.0 chart with staticOrigin. Documents use the website endpoint; browser assets use the public assets Blob container with application-origin CORS. Both application charts are published. See the module contract.