Skip to content

A reproducible Linux end-to-end lab, and the three defects building it found - #6

Merged
BenjiTrapp merged 22 commits into
mainfrom
e2e-docker-lab
Nov 17, 2026
Merged

BenjiTrapp merged 22 commits into
mainfrom
e2e-docker-lab

Conversation

@BenjiTrapp

Copy link
Copy Markdown
Owner

Four defects shipped in the DNS rescue channel (#4, #5) and a 3538-test suite
found none of them. Two were platform defects that cannot surface on
macOS. The VM that did show one lost its network mid-run.

This makes the Linux end-to-end run reproducible, cheap enough for CI, and
extensible to the transports that come later.

Design: docs/superpowers/specs/2026-11-15-e2e-docker-lab-design.md
Plan: docs/superpowers/plans/2026-11-16-e2e-docker-lab.md
The 56 rulings this was built on: docs/superpowers/plans/2026-11-16-e2e-docker-lab-rulings.md

The lab

Severance is by topology: the beacon sits on a network the team server is
not on, so HTTPS, SMB and QUIC fail because they do not route. dnsmasq is a
real recursive resolver in front of the DNS listener, and _DNS_PORT is
untouched — so the hop the rescue channel rests on is exercised, not bypassed.

e2e_lab on arm64, Darwin, compose: docker-compose
registered as raccoon-bandit (implant id bac1446c40b63f5e)
severing raccoon-lab_c2-net
  severed: the team server's beacon socket        closed
  severed: the QUIC address                       closed
  severed: the operator socket                    closed

WHAT THE REAL RESOLVER LOGGED:
  [tasking] TXT  ojqwgy3pn5xc2ytbnzsgs5a.b.c2.lab     32 chars, longest label 23
  [RESCUE ] TXT  435tw77dgnpsxt2pa6vexv2ylf5smfqo...  130 chars, longest label 48
  [tasking] TXT  ojqwgy3pn5xc2ytbnzsgs5a.b.c2.lab     32 chars
  [tasking] TXT  ojqwgy3pn5xc2ytbnzsgs5a.b.c2.lab     32 chars
  [tasking] TXT  ojqwgy3pn5xc2ytbnzsgs5a.b.c2.lab     32 chars
  4 tasking queries, 1 rescue

  Stuck: HTTPS: unreachable; DNS: answered; SMB: not tried; QUIC: not tried

The beacon registers for real first, because the implant assigns its agent
id only in the HTTPS registration path — a beacon that never registered
fingerprints its implant id and matches no fleet key. So the life-cycle is
work, then be cut off, which is also the only one that matters operationally.

Evidence comes from the components: dnsmasq's own query log, the operator API,
the beacon container's output. Never the driver's bookkeeping.

Three scenarios, each falsifiable. rescue; stale-token, which the VM
died before reaching and which is the only one that proves the leg-recording
fix on Linux (SMB: unreachable; QUIC: unreachable, not "not tried"); and
smb-up, which keeps the stale token on purpose, because disabling DNS would
make "no rescue was sent" vacuous.

Mutation-proved, not asserted: revert the rescue trigger to tasking is None → 0 rescues, exit 1. Remove the SMB listener → three independent
assertions fire. Remove actions/setup-node → the CI guard fires.

The three Linux defects building it found

Defect Cause
The Unix one-liner could not run on Linux at all 135,492 bytes against MAX_ARG_STRLEN of 131,072, measured at the boundary. No quoting avoids it, and the POSIX stager's eval hits the same wall. The payload now travels on stdin through a shell builtin.
stop() left the relay's port accepting on Linux Closing a descriptor another thread blocks in accept() on is undefined in POSIX. macOS wakes it; Linux does not. shutdown() before close() — and the two try blocks are load-bearing, because macOS raises ENOTCONN on a listening socket.
An unattributed rescue was invisible to every operator unattributed_rescues was appended to, trimmed, and never served. Now: a log event, an authenticated route, a separate key on the agent tool, and one batched notification.

CI, where this repository had none

Two jobs on ubuntu-latest. Each apt package carries its measured
consequence: without git, ffmpeg and procps the suite reports 19 errors
and 6 failures that say nothing about the code.

node is not optional, and the claim is measured. With node hidden from
PATH and the rest left in place, three modules run 294 tests and report OK
either way — 0 skips with node, 17 without, seven of them the operator
console's notification tests. Green both times. That is the silent-green this
workflow exists to prevent.

Notes for review

  • Five of the nine tasks had no independent reviewer. Seven dispatches were
    refused by the provider, so Tasks 6–9 were implemented and reviewed by the
    same context, declared as self-reviews in the rulings document. A fresh
    reader found this feature's two worst defects earlier; self-review did not.
  • Eleven rulings record a defect of my own, five of them the same class — a
    test asserting an invented claim rather than a property of the code. The
    worst: three CI guards asserted against the workflow's prose, so deleting
    setup-node left all ten green. Each was caught by mutation, not by reading.
  • A recorded limit: stale-token reports unreachable, not timed_out.
    After severance the name is unresolvable, so the leg raises — the silent
    Linux timeout path is not exercised. Closing it needs a blackhole
    address on beacon-net.
  • software/c2/docker-compose.yml is untouched. The listener container is the
    shipped artefact, built from the shipped Dockerfile, and a test compares its
    flags against the shipped compose so the lab cannot drift into proving a
    shape nobody deploys.

3538 tests green; all three scenarios exit 0.

Awaiting approval; no implementation yet.

Four defects shipped in the DNS rescue channel and the 3257-test suite
found none of them. Two were platform defects that cannot show on macOS,
and the VM that did show one lost its network mid-run. This describes a
lab where the Linux end-to-end run is reproducible and cheap enough for
CI.

Severance is by topology: the beacon sits on a network the team server is
not on, so HTTPS, SMB and QUIC fail because they do not route. Every
constraint in the document was measured on this machine rather than
assumed, including that container-to-container UDP works while published
UDP on macOS does not -- which is why the beacon is a container.

A builder check found three things the first draft got wrong, all now
fixed in the document: a statically severed beacon can never be
attributed (the implant assigns its agent id only in the HTTPS
registration path), 'nothing published' would have left the driver unable
to read the record it asserts on, and the smb-up scenario as drafted
could not pass. It also surfaced a product gap the lab does not fix:
unattributed_rescues is never served over HTTP, so a beacon that never
registered is invisible to every operator.
Measured before writing the workflow, by running the suite in a Linux
container on python 3.9. 3438 tests ran there against 3258 on macOS --
180 that had never executed anywhere.

Without git, ffmpeg and procps in the image: 19 errors, 6 failures, none
of which say anything about the code. With them, three remain, and all
three are real defects on the platform the implant targets.

The Unix one-liner cannot be executed on Linux at all: the stage is
135,492 bytes and the kernel refuses any argv entry over 131,072
(measured exactly at the boundary). No quoting avoids it, and the POSIX
stager's eval path hits the same wall. docs/TUNNEL.md calls that path the
common case on Linux, which is false.

And stop() on the NTLM relay closes a socket another thread is blocked
in accept() on, which is undefined in POSIX: macOS wakes it, Linux leaves
the port bound and accepting. The operator believes the relay is down
while the target still has an open port.

In scope by the operator's decision, sequenced first so they can be
split out if the two worktrees working in that code conflict.
The operator brought the unattributed_rescues gap into scope.

Four surfaces, because each reaches a different reader: a log event on
arrival so it is never silent, an authenticated operator route, a
separate key on the agent tool (separate because there is no beacon id
to task and a model must not mistake one for something actionable), and
a bell notification, because visible in a drawer nobody opens is not
addressed.

The notification is the only part that edits _OPERATOR_HTML, so its
acceptance is a browser rather than a regex. A source-level assertion
that a string is present is exactly what let escHtml(info.stuck.summary)
ship and destroy the whole agent panel.
Nine tasks. The two platform fixes come first, in their own commits, so
they can be split out if the two worktrees working in tunnel and implant
code conflict. Then the four surfaces that make an unattributed rescue
visible, then the lab's containers, its driver and the three scenarios,
then CI.

Each scenario must fail when its subject is reverted, and the plan names
which commit to revert for two of them; the third is exempt because it
guards a property of the lab rather than a product change, which is
stated so its absence is not read as an oversight.

The notification task's acceptance is a browser, not a regex. A
source-level assertion that a string is present is what let
escHtml(info.stuck.summary) ship and destroy the whole agent panel.
Task 2's test monkeypatched relay._server.shutdown. Measured: socket
objects use __slots__ and reject attribute assignment, so the test could
not have been written. Replaced with a recording proxy on the relay's
attribute, which also pins the call order -- shutdown after close would
fix nothing, and the monkeypatch left that to chance.

Task 6 said to parse the compose file with yaml 'if available, otherwise
a small reader'. PyYAML 6.0.3 is installed and software/requirements.txt
already requires pyyaml>=6.0, so the reader is now forbidden.

And a variable in Task 1 named 'stage' while holding the one-liner,
which matters because the new form legitimately contains a quote and a
pipe.
Both were wrong in my document, not in the code, and an implementer
caught them by reading the design against the source.

The image table said the beacon container installs requests. It installs
nothing: beacon.py is standard-library only -- its HTTP session is
_HttpSession over urllib and says so -- and an unused dependency would
have contradicted the stdlib-only constraint this same document states.

And the smb-up scenario spoke of giving the SMB listener a valid token.
Only the DNS listener has that concept: listeners/dns.py:79 is the one
constructor taking listener_token, and the shipped compose gives the
variable to the DNS service alone.
The lab runs. On arm64 driving linux/aarch64 containers, with the compose
form detected rather than assumed: the beacon registers for real while
attached to c2-net, docker network disconnect severs it, and the driver
confirms from INSIDE the beacon container that the team server's beacon
socket, the QUIC address and the operator socket are all closed. So the
legs fail because they do not route, not because a stub returned None.

Evidence comes from the components. dnsmasq's own query log shows four
tasking queries and exactly one rescue across four stuck cycles, 130
characters with a longest label of 48 -- both inside the DNS limits,
checked against what actually crossed the resolver. The operator record
carries the diagnosis.

And it fails when it should: with the trigger reverted to `tasking is
None`, zero rescues cross and the operator sees nothing. That is the
defect a hand-kept VM found last week before losing its network; it is
now one command.

Four defects of my own, each found only by running it: _compose passed no
env, so the generated key never reached compose and the server exited
with its own correct complaint; the resolver's queries are in a file, as
resolver.conf's comment says, not on stdout; the fleet key is agent_id
and not id, which is the same distinction the rescue's attribution turns
on; and python 3.9's argparse rejects the empty list nargs="*" produces
when choices is set.
stale-token gives the DNS listener a token the server no longer accepts.
Tasking over DNS needs it; the rescue does not, because it goes through
the beacon handler. So the ladder runs all the way down, SMB and QUIC are
really attempted, and the operator record reads SMB: unreachable; QUIC:
unreachable instead of not tried. That is the fix from 3dad39e proved end
to end on Linux, in the scenario a hand-kept VM lost its network before
reaching.

smb-up keeps that stale token on purpose. Disabling DNS would make 'no
rescue was sent' vacuous: _maybe_rescue returns immediately when DNS is
off, so the latch would never be armed and the absence would prove
nothing. Here the resolver answers, the latch is armed and could fire,
and it does not -- because an upstream leg delivered.

Each scenario now declares what it claims, and a test refuses one that
claims nothing. The expectations differ, which matters: the Task 7 check
would have reported smb-up's correct 'no diagnosis' as a failure.

One defect of my own, worth naming. The first smb-up check compared
last_channel to 'smb'. A listener relays to the beacon handler over HTTP,
so an SMB delivery lands as last_channel: http and that field cannot
distinguish SMB from HTTPS at all -- the check printed 'expected smb',
saw 'http', and passed anyway. A central claim proved by nothing, which
is the same defect class I flagged three times in other people's work
this week.

It now rests on an invariant of the implant instead: a leg reaches
_leg_status only when it FAILS, and the ladder is ordered. If SMB
delivered, smb is absent from every cycle and quic was never reached; if
SMB failed, both are there. Both directions are demonstrated -- removing
the listener makes three independent assertions fire.
This repository had no CI at all. The reason it has some now is a
measurement: four defects shipped in the DNS rescue channel and a
3528-test suite found none of them, and two of the four could not surface
on macOS -- a dead QUIC endpoint raises on Darwin and merely times out on
Linux, and Linux refuses an argv entry over 131072 bytes where Darwin has
no such limit. The suite had never run on Linux.

Two jobs, because they fail for different reasons and a reader looking at
a red run should be able to tell which broke.

Each apt package carries its measured consequence: without git, ffmpeg
and procps the suite reports 19 errors and 6 failures that say nothing
about the code. Three packages with no reason beside them are three
packages the next person deletes.

node is installed deliberately, not incidentally. Several tests are
skipUnless(which('node')), including the operator console's notification
dedupe and batching, which run the page's real JavaScript -- without node
they skip SILENTLY and the job reports green over logic it never ran,
which is the exact failure mode this workflow exists to prevent. A
separate step proves the tools are present so a missing one is a loud
setup failure rather than a quiet gap. util-linux and bash-completion are
there for the same reason: on Linux the suite runs MORE than on a
developer's Mac, and one of the local skips says so itself.

Two defects of my own. CiTests.setUpClass raised SkipTest when the
workflow was absent, so deleting CI would have removed its own guard --
the same silent green it checks for. And the secret test forbade the
string 'secrets.', which the comment explaining why there are none then
failed; it now forbids the reference syntax, verified by adding a real
one and watching it fire.
Found by the final review, and the fifth defect of this class on this
branch. The tests for node, the apt packages and the python version
asserted against the workflow file's raw TEXT -- which contains the
comments explaining why each matters. The words node, ffmpeg and 3.9 are
all in that prose, so the tests stayed green when the step they describe
was deleted.

Measured: removing actions/setup-node left all ten CI tests passing.

They now assert against the parsed structure: the setup-node step, the
actual apt-get command line, the real python-version value. Two
counter-checks confirm the guard fires -- removing the step reports 'no
setup-node step', and removing ffmpeg from the apt line while keeping its
comment reports 'named in prose but never installed'. That distinction is
what this branch failed on four times before.

And the workflow's central claim is now measured rather than argued. With
node hidden from PATH and ffmpeg, git and ps left in place, three modules
run 294 tests and report OK either way: 0 skips with node, 17 without,
seven of them the operator console's notification tests. Green both
times, which is exactly the silent-green this workflow exists to prevent.
The numbers are in the file.
Every decision I took on the operator's behalf, in order, with what each
costs if it is wrong. Committed rather than left in the git-ignored
ledger, and rather than pasted into a chat window that cannot hold them:
a ruling that reaches nobody was a decision made in secret.

Eleven of them record a defect of my own. Five record the same class --
a test that asserts a claim I invented rather than a property of the code
-- which is the class I flagged four times in other people's work this
week. Each was caught by mutating the subject and watching the test, not
by reading it.
@BenjiTrapp
BenjiTrapp merged commit ae44160 into main Nov 17, 2026
0 of 4 checks passed
@BenjiTrapp
BenjiTrapp deleted the e2e-docker-lab branch November 17, 2026 22:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant