Skip to content

Scale the honeypot-ci runner: one instance serializes the whole all-homeserver CI fan-out #2572

Description

@Xore

#2568 routes every compute workflow homeserver-first. The box has exactly ONE registered honeypot-ci runner instance, so jobs that used to fan out across GitHub-hosted runners now serialize behind a single executor.

Why it matters now

  • quality.yml alone contributes up to ~20 concurrent jobs when the router proves the box live (paired families + the scripts-and-compose matrix); containers.yml adds 18 image builds; CodeQL adds 3 language jobs.
  • On one runner instance these run one-at-a-time: a push that hits the box can hold PR feedback for hours (the ES suites, Keycloak/OIDC suites and image builds are minutes each, not seconds).
  • A wedged pickup blocks everything behind it; the per-row timeout-minutes ceilings (45–120) only fire on a truly wedged job, and the router's heartbeat canary can still read "live" while a long queue drains.

Candidate shapes

  1. N runner instances on the homeserver. Register 2–4 additional honeypot-ci instances (systemd units with distinct _work dirs and names) via install-ci-runner.sh --name. Caveats to solve first:
    • Port contention between suites (the ES/Keycloak/OIDC suites bind fixed ports; two same-suite rows scheduled on different instances would collide). Either pin collision-prone matrix rows to a dedicated label (honeypot-ci-exclusive), or make the suites take ports from an env window.
    • Docker builds inside containers.yml are the heaviest fan-out winners (18 rows, buildx-local caches per runner instance — consider a shared buildx cache or accept cold caches on the extra instances).
  2. Split labels by weight: keep serial-sensitive rows on honeypot-ci and give the heavy fan-out (containers matrix, CodeQL) a second instance/label so they never queue behind the browser matrix.

Non-goal

  • No second machine in scope; this is about the existing box's concurrency.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

enhancementNew feature or requestopsDeployment, runners, observability, host access

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions