Skip to content

Latest commit

 

History

History
368 lines (264 loc) · 23.3 KB

README.md

File metadata and controls

368 lines (264 loc) · 23.3 KB

🚀 My Home Operations Repository 🚧

... managed with Flux, Renovate, and GitHub Actions 🤖

Discord   Talos   Kubernetes   Flux   Renovate

Cluster   Services

Age-Days   Uptime-Days   Node-Count   Pod-Count   CPU-Usage   Memory-Usage   Power-Usage   Alerts


💡 Overview

This is a mono repository for my home infrastructure and Kubernetes cluster. I try to adhere to Infrastructure as Code (IaC) and GitOps practices using tools like Kubernetes, Talos Linux, Flux, Renovate, and GitHub Actions.


🌱 Kubernetes

My Kubernetes cluster is deployed with Talos. All persistent storage is provided by an external Ceph cluster running on the Proxmox hosts, connected to Kubernetes via Rook in external-cluster mode.

Storage Architecture:

  • Block (RBD): ceph-rbd (default StorageClass) for databases and single-pod workloads (RWO)
  • Shared filesystem (CephFS): cephfs-shared / cephfs-static / cephfs-backups for multi-pod and media workloads (RWX)
  • Backups: VolSync (Restic) snapshots of every stateful PVC
  • Ceph cluster: Ceph 18.2.x (Reef) on Proxmox, dedicated 40Gb storage network (VLAN 200)
dapper-cluster storage architecture: external Ceph on Proxmox, consumed via Rook CSI

There is a template over at onedr0p/cluster-template if you want to try and follow along with some of the practices used here.

Core Components

Networking:

  • cilium: eBPF-based CNI with kube-proxy replacement, L2 announcements, and LB-IPAM.
  • multus-cni: Multiple network interfaces per pod for IoT and legacy network integration.
  • envoy-gateway: Gateway API implementation; internal, external, and media Gateways route traffic via HTTPRoutes.
  • external-dns: Syncs public DNS records to Cloudflare.
  • k8s-gateway: Authoritative internal DNS for cluster hostnames.
  • cloudflared: Secure Cloudflare tunnels for external access.

Storage:

  • rook-ceph: Connects Kubernetes to the external Proxmox Ceph cluster and provides the RBD + CephFS CSI drivers.
  • volsync: PVC backup and recovery (Restic).

Security & Secrets:

Observability:

  • kube-prometheus-stack: Prometheus, Grafana, and Alertmanager.
  • victoria-logs: Log aggregation and query.
  • fluent-bit: Log shipper into VictoriaLogs.
  • gatus: Service health monitoring and status page.
  • ntfy: Self-hosted push notifications for Gatus and Alertmanager alerts.
  • network-ups-tools: UPS power monitoring (NUT), surfaced via PeaNUT and a Prometheus exporter.

GPU & Hardware:

GitOps & Automation:

GitOps

Flux watches the clusters in my kubernetes folder (see Directories below) and makes the changes to my clusters based on the state of my Git repository.

The way Flux works for me here is it will recursively search the kubernetes/apps folder until it finds the most top level kustomization.yaml per directory and then apply all the resources listed in it. That aforementioned kustomization.yaml will generally only have a namespace resource and one or many Flux kustomizations (ks.yaml). Under the control of those Flux kustomizations there will be a HelmRelease or other resources related to the application which will be applied.

Renovate watches my entire repository looking for dependency updates, when they are found a PR is automatically created. When some PRs are merged Flux applies the changes to my cluster.

Directories

This Git repository contains the following directories under Kubernetes.

📁 kubernetes
├── 📁 apps           # applications
├── 📁 bootstrap      # bootstrap procedures
└── 📁 flux           # flux system config + reusable components

Flux Workflow

This is a high-level look how Flux deploys my applications with dependencies. In most cases a HelmRelease will depend on other HelmRelease's, in other cases a Kustomization will depend on other Kustomization's, and in rare situations an app can depend on a HelmRelease and a Kustomization. The example below shows that plex won't be deployed or upgrade until the storage dependencies are installed and in a healthy state.

graph TD
    %% Styling
    classDef kustomization fill:#2f73d8,stroke:#fff,stroke-width:2px,color:#fff
    classDef helmRelease fill:#389826,stroke:#fff,stroke-width:2px,color:#fff

    %% Nodes
    A>Kustomization: rook-ceph-operator]:::kustomization
    B[HelmRelease: rook-ceph-operator]:::helmRelease
    C[Kustomization: rook-ceph-cluster]:::kustomization
    D>Kustomization: plex]:::kustomization
    E[HelmRelease: plex]:::helmRelease

    %% Relationships with styled edges
    A -->|Creates| B
    A -->|Creates| C
    C -->|Depends on| B
    D -->|Creates| E
    E -->|Depends on| C

    %% Link styling
    linkStyle default stroke:#666,stroke-width:2px
Loading

Networking

My network spans two physical locations connected via a 60GHz wireless bridge, featuring a multi-tier switching architecture optimized for high-performance storage and compute workloads.

ChelonianLabs home network and fabric bandwidth: two-building multi-vendor fabric with 40/80G LACP

Key Features:

  • Dual locations connected via 60GHz wireless (1Gbps)
  • Multi-tier switching: Core (Brocade), Distribution (Arista), Access (Aruba)
  • Dedicated 40Gb storage network on Arista
  • LACP bonding on server links for redundancy
Kubernetes Cluster
graph TB
    %% Styling
    classDef control fill:#2f73d8,stroke:#fff,stroke-width:2px,color:#fff
    classDef worker fill:#389826,stroke:#fff,stroke-width:2px,color:#fff
    classDef gpu fill:#e74c3c,stroke:#fff,stroke-width:2px,color:#fff
    classDef vip fill:#f39c12,stroke:#fff,stroke-width:3px,color:#000

    VIP["Kubernetes API VIP<br/>10.100.0.40:6443"]:::vip

    subgraph ControlPlane["Control Plane Nodes"]
        CP1["talos-control-1<br/>10.100.0.50<br/>4 CPU / 16GB"]:::control
        CP2["talos-control-2<br/>10.100.0.51<br/>4 CPU / 16GB"]:::control
        CP3["talos-control-3<br/>10.100.0.52<br/>4 CPU / 16GB"]:::control
    end

    subgraph Workers["Worker Nodes"]
        GPU["talos-node-gpu-1<br/>10.100.0.53<br/>16 CPU / 128GB<br/>2x Tesla P100"]:::gpu
        W1["talos-node-large-1<br/>10.100.0.54<br/>16 CPU / 128GB"]:::worker
        W2["talos-node-large-2<br/>10.100.0.55<br/>16 CPU / 128GB"]:::worker
        W3["talos-node-large-3<br/>10.100.0.56<br/>16 CPU / 128GB"]:::worker
        S1["talos-node-small-1<br/>10.100.0.60<br/>6 CPU / 16GB"]:::worker
        S2["talos-node-small-2<br/>10.100.0.61<br/>6 CPU / 16GB"]:::worker
        S3["talos-node-small-3<br/>10.100.0.62<br/>6 CPU / 16GB"]:::worker
        S4["talos-node-small-4<br/>10.100.0.63<br/>6 CPU / 16GB"]:::worker
    end

    VIP -.-> CP1
    VIP -.-> CP2
    VIP -.-> CP3

    style ControlPlane fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
    style Workers fill:#e8f5e9,stroke:#388e3c,stroke-width:2px
Loading

Cluster Configuration:

  • Total Nodes: 11 (3 control plane, 8 workers)
  • Workers: 1x GPU (2x Tesla P100), 3x large (16 CPU / 128GB), 4x small (6 CPU / 16GB)
  • OS: Talos Linux (Kubernetes v1.36)
  • CNI: Cilium with eBPF (10.69.0.0/16 pod CIDR)
  • API VIP: 10.100.0.40 (shared across control plane)
VLAN & Network Segmentation
graph LR
    %% Styling
    classDef mgmt fill:#95a5a6,stroke:#fff,stroke-width:2px,color:#fff
    classDef servers fill:#3498db,stroke:#fff,stroke-width:2px,color:#fff
    classDef storage fill:#e74c3c,stroke:#fff,stroke-width:2px,color:#fff
    classDef k8s fill:#2ecc71,stroke:#fff,stroke-width:2px,color:#fff

    subgraph Physical["Physical Networks"]
        V1["VLAN 1<br/>192.168.1.0/24<br/>Management<br/>MTU 1500"]:::mgmt
        V100["VLAN 100<br/>10.100.0.0/24<br/>Servers/VMs<br/>MTU 1500"]:::servers
        V150["VLAN 150<br/>10.150.0.0/24<br/>Storage Public<br/>MTU 9000"]:::storage
        V200["VLAN 200<br/>10.200.0.0/24<br/>Storage Cluster<br/>MTU 9000"]:::storage
    end

    subgraph Kubernetes["Kubernetes Networks"]
        POD["Pod Network<br/>10.69.0.0/16<br/>Cilium CNI"]:::k8s
        SVC["Service Network<br/>10.96.0.0/16<br/>ClusterIP"]:::k8s
    end

    V100 -.Talos VMs.-> POD
    V100 -.Talos VMs.-> SVC
    V150 -.CSI Drivers.-> POD

    style Physical fill:#ecf0f1,stroke:#34495e,stroke-width:2px
    style Kubernetes fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px
Loading

Network Details:

  • VLAN 1: Switch management, IPMI, gateway: OPNsense
  • VLAN 100: Kubernetes nodes, gateway: OPNsense, internet access
  • VLAN 150: Storage client connections, jumbo frames, L2 only
  • VLAN 200: Storage cluster traffic (40Gb dedicated), jumbo frames, L2 only
  • Pod Network: Cilium eBPF-based CNI with native routing
  • Service Network: Standard Kubernetes ClusterIP services

For detailed network documentation including switch configurations, see Network Architecture.


😶 Cloud Dependencies

While most of my infrastructure and workloads are self-hosted I do rely upon the cloud for certain key parts of my setup. This saves me from having to worry about three things. (1) Dealing with chicken/egg scenarios, (2) services I critically need whether my cluster is online or not and (3) The "hit by a bus factor" - what happens to critical apps (e.g. Email, Password Manager, Photos) that my family relies on when I no longer around.

Alternative solutions to the first two of these problems would be to host a Kubernetes cluster in the cloud and deploy applications like HCVault, Vaultwarden, ntfy, and Gatus; however, maintaining another cluster and monitoring another group of workloads would be more work and probably be more or equal out to the same costs as described below.

Service Use Cost
Infisical Secrets with External Secrets Free
Cloudflare Domain and S3 Free
GCP Voice interactions with Home Assistant over Google Assistant Free
GitHub Hosting this repository and continuous integration/deployments Free
Migadu Email hosting ~$20/yr
Pushover Kubernetes Alerts and application notifications $5 OTP
Healthchecks.io Dead-man's-switch for the Alertmanager heartbeat Free
Total: ~$2/mo

🌎 DNS

Internal name resolution is handled by k8s-gateway, which is authoritative for my private domains and answers with the internal Envoy Gateway's load-balancer IP. My OPNsense resolver forwards those domains to it. Public records are synced to Cloudflare by a single ExternalDNS instance, and external traffic reaches the cluster through a Cloudflare Tunnel. Whether a service is internal- or external-facing is determined by which Gateway its HTTPRoute attaches to.


⚙ Hardware

Compute Infrastructure

Device CPU RAM Storage Network Function
Proxmox-01 Intel Xeon E3-1230 V2
(4 cores @ 3.30GHz)
16GB 24x 4TB HDD 1Gb IPMI
2x 1Gb
2x 10Gb
2x 40Gb
Ceph OSDs
Proxmox-02 2x Intel Xeon X5680
(24 cores @ 3.33GHz)
196GB 2x 120GB ZFS mirror 1Gb IPMI
2x 1Gb
2x 10Gb
2x 40Gb
Kubernetes + Ceph
Proxmox-03 2x Intel Xeon E5-2697A v4
(64 cores @ 2.60GHz)
516GB 1x 3.92TB SSD + 1x 800GB
2x Tesla P100 16GB
1Gb IPMI
2x 1Gb
2x 10Gb
2x 40Gb
Kubernetes + Ceph
GPU Passthrough
Proxmox-04 2x Intel Xeon X5680
(24 cores @ 3.33GHz)
196GB 8x 10TB + 1x 3.84TB + 1x 800GB 1Gb IPMI
2x 1Gb
2x 10Gb
2x 40Gb
Kubernetes + Ceph

Total Cluster Resources:

  • CPU: 116 cores total
  • RAM: 924GB total
  • Storage: 476.96TB raw (76 drives across all hosts + JBOD), pooled into the external Ceph cluster
  • GPU: 2x NVIDIA Tesla P100 16GB (32GB VRAM total)
  • Network: Multi-tier switching with 40Gb Ceph network

Kubernetes VMs (Talos Linux)

VM Name vCPU RAM Host Role Notes
talos-control-1 4 16GB Proxmox-03 Control Plane
talos-control-2 4 16GB Proxmox-04 Control Plane
talos-control-3 4 16GB Proxmox-02 Control Plane
talos-node-gpu-1 16 128GB Proxmox-03 Worker 2x P100 GPU passthrough
talos-node-large-1 16 128GB Proxmox-03 Worker
talos-node-large-2 16 128GB Proxmox-03 Worker
talos-node-large-3 16 128GB Proxmox-03 Worker
talos-node-small-1 6 16GB Proxmox cluster Worker
talos-node-small-2 6 16GB Proxmox cluster Worker
talos-node-small-3 6 16GB Proxmox cluster Worker
talos-node-small-4 6 16GB Proxmox cluster Worker

Kubernetes Cluster Totals: 100 vCPU, 624GB RAM, 2x Tesla P100 GPUs

Network Equipment

Device Model Location Role Specs
OPNsense Router Custom (i3-4130T) House Gateway/Firewall 2C/4T @ 2.90GHz, 16GB RAM, 2.5Gb ATT Fiber
Brocade ICX6610 Enterprise Switch Garage Core L3 Switch 48x 1/10Gb ports, 4x 40Gb QSFP+, VLAN routing
Arista 7050 Data Center Switch Garage Distribution 48x 10Gb SFP+, 4x 40Gb QSFP+
Aruba S2500-48p Access Switch House PoE Access 48x 1Gb PoE+ ports
Mikrotik NRay60 60GHz Radio (x2) Both Wireless Bridge 1Gbps point-to-point link

Storage

Storage Distribution (Ceph OSDs across Proxmox hosts):

  • Proxmox-01: 24x 4TB HDD (96TB)
  • Proxmox-03: 3x 4TB + 7x 10TB + 2x 12TB (106TB)
  • Proxmox-04: 8x 10TB + 1x 3.84TB + 1x 800GB (84.64TB)
  • JBOD Shelf: 18x 10TB + 1x 3.84TB + 1x 800GB (184.64TB)
  • Total Raw Capacity: 476.96TB across 76 drives
  • Network: Dedicated 40Gb network for Ceph traffic (VLAN 200)

🌟 Stargazers


🙏 Gratitude and Thanks

Thanks to all the people who donate their time to the Home Operations Discord community. Be sure to check out kubesearch.dev for ideas on how to deploy applications or get ideas on what you could deploy.