Skip to content

Emudhra ca remediation - #854

Merged
kacper-wolkiewicz-reef merged 9 commits into
masterfrom
emudhra-ca-remediation
Oct 23, 2026
Merged

kacper-wolkiewicz-reef merged 9 commits into
masterfrom
emudhra-ca-remediation

Conversation

@kacper-wolkiewicz-reef

Copy link
Copy Markdown
Contributor

No description provided.

certifi releases before 2026.10.19 trust the eMudhra/emSign intermediate
CAs whose private keys were compromised, which allows TLS interception
of every requests/httpx client that relies on certifi.

Only certifi was upgraded. The scripts/stress lock was out of sync with
compute_horde's dependencies, so relocking it also adds those packages.

Impacts: lib, sdk, validator, miner, executor, facilitator
- uv 0.5 -> 0.13.4, the first uv release whose bundled root store
  no longer trusts the compromised eMudhra CAs.
- pip 26.2.2 in the final images: python:3.11-slim ships pip 24.0 with
  a vendored certifi 2023.07.22, and pip always trusts its vendored
  bundle. The wheel is installed by URL with a pinned sha256 because the
  old pip performs the download.
- ca-certificates is installed explicitly, so a stale base image still
  picks up the Debian update that removes the compromised roots.

Impacts: validator, miner, executor, facilitator
- runners: docker:26-cli (EOL) -> docker:29-cli
- postgres:14.0-alpine -> postgres:14-alpine (same major, rebuilt);
  the docuum keep rule follows the new tag
- watchtower 1.20.3 -> 1.23.2 (built with Go 1.27.3, which rejects
  the compromised keys) in the install scripts and runner READMEs
- cadvisor v0.40.0 -> ghcr.io/google/cadvisor:v0.60.5; it serves
  inbound scrapes only and has no fixed release yet, but v0.40.0 is
  unsupported and does not see containers under Docker 29

Impacts: validator, validator-runner, miner, miner-runner, facilitator
uv < 0.13.4 and pip < 26.2.2 trust the compromised eMudhra CAs, so CI
installs and the library release could be intercepted. The library
release upgrades pip from a hash-pinned wheel before installing pdm.

Impacts: none
Promtail is end-of-life and gets no rebuild with the fixed root store,
yet it sends the Loki credentials over TLS. The Alloy config keeps the
promtail scrape setup (Docker discovery, container/logstream labels,
client and client_server_group external labels) and pushes to
/loki/api/v1/push instead of the legacy /api/prom/push endpoint.

Impacts: facilitator
Registry tags can be re-pushed, so a compromised publisher could swap
the image behind a tag. The base, builder and service images bumped for
the eMudhra CA compromise (python, uv, docker CLI, postgres, cadvisor,
alloy, watchtower) are now pinned to their multi-arch index digest; the
tag names the exact version behind it.

An image pulled by digest is untagged in the containerd image store and
docuum sees it as postgres:<none>, so the docuum keep rule now matches
any postgres image.

Impacts: validator, validator-runner, miner, miner-runner, executor, facilitator
Renaming the promtail service to alloy leaves its container orphaned.
Since Compose v2.24, `docker compose ps --services` also lists orphaned
services, so deploy.sh passed promtail to `docker compose stop`, which
fails with "no such service" and aborts the deploy. The services to
stop now come from `docker compose config --services`, and
`docker compose up -d --remove-orphans` removes the old container.

Impacts: facilitator
setup-uv@v6 verifies only releases listed in its built-in checksum
table, which ends before uv 0.13.4, so the pinned uv was installed
without an integrity check. A shared composite action now pins the uv
version together with the checksum of its x86_64 Linux archive, and
every workflow installs uv through it.

Impacts: none
The lockfiles alone fix only environments built from this repository.
With the minimum version in the published dependencies, upgrading
compute-horde or compute-horde-sdk also upgrades a certifi that still
trusts the compromised eMudhra/emSign CAs.

Impacts: lib, sdk, validator, miner, executor, facilitator
@github-actions

Copy link
Copy Markdown

Commit message lint report

gitlint --commits 8f6328ac77131b68dab2052a79064b5c6745bf8b..182f35e16acd3116fcf78eb114d71fed92ef0978

Status: ✅ No issues found.

@kacper-wolkiewicz-reef
kacper-wolkiewicz-reef merged commit 5dbd80b into master Oct 23, 2026
31 of 32 checks passed
@kacper-wolkiewicz-reef
kacper-wolkiewicz-reef deleted the emudhra-ca-remediation branch October 23, 2026 08:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant