Skip to content

fix: clean up Traefik network connections on tenant deletion #135

Description

@dennisonbertram

Problem

When a tenant is deleted, their containers are stopped and removed, but the tenant's Docker network remains connected to the Traefik container. Over time (~100+ tenants), accumulated network connections slow down the Docker API and can cause Traefik instability. This is a known scaling wall identified in the investigation at docs/investigations/issue-15-traefik-network-fanout.md.

Proposed Solution

  1. On tenant deletion: After stopping all tenant containers, call NetworkDisconnect() to detach the tenant's network from Traefik before removing the network.

  2. Network cap monitoring: Add a traefik_networks count to the GET /v1/system/health/detailed response. Warn when count exceeds 150, degrade health status at 200.

  3. GC sweep: In the existing garbage collector loop, detect and clean up orphaned networks (networks with no running containers that are still connected to Traefik).

Implementation Notes

  • internal/services/services.go — add NetworkDisconnect call in the tenant deletion flow
  • internal/docker/client.go — may need to add NetworkDisconnect to the Docker client interface
  • internal/gc/gc.go — add orphaned network detection step
  • internal/api/health.go — add network count to detailed health response

Acceptance Criteria

  • Tenant deletion disconnects the tenant network from Traefik before removal
  • GC detects and cleans up orphaned Traefik-connected networks
  • Detailed health endpoint reports traefik_networks count
  • Health degrades when network count exceeds threshold
  • Unit tests for disconnect and GC logic
  • No regression in existing tenant deletion flow

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions