From 49de6fc15e97df2ec810b98016239633406290fe Mon Sep 17 00:00:00 2001 From: Antawari Date: Tue, 28 Jul 2026 22:11:57 -0600 Subject: [PATCH] Prove every marker the release box classifies by, or the coverage test goes red When a pip step in the release-gate box fails, classify_pip_failure decides whether the verdict blames the BOX's network (box_network_unreachable) or the ARTIFACT under test (artifact_install_failed). Both are a FAIL; only the claim differs, and that claim is load-bearing. On 2026-07-27 a run whose link had collapsed to 13 KiB/s aborted twice blaming the artifact, for a wheel that installs in 1.4 seconds on a healthy link. A verdict naming the artifact when the artifact was never reached is a false accusation against a released wheel, and it points the next operator at the wrong repository. The decision rests on two regex alternation lists: 10 artifact markers and 17 transport markers. Measured before this change: 6 of 17 transport alternatives appeared in any test, and ZERO were load-bearing, because each of the three network fixtures matched two or three alternatives at once -- deleting any single transport alternative changed no test result. Of the artifact markers, 4 of 10 appeared and only 2 were load-bearing. A pattern list nobody proved matches is a gate selecting by a value it never grades: a typo, a case change or an upstream rewording silently reclassifies a dead link as a broken wheel. So every alternative in both lists now has captured pip output that matches it and nothing else, and deleting that alternative flips a real verdict. The degraded shapes the incident class is actually about are the ones that had no coverage at all, and they are covered now: a truncated body mid-download, a chunked response whose last chunk never arrives, a middlebox resetting the connection, a connect-phase stall, retries exhausted against a 503-ing index, a captive portal presenting its own certificate, and name resolution failing a second way. Every artifact alternative is graded THROUGH a degrading link, because an artifact-only log classifies as artifact via the default branch and therefore proves nothing. Four mixed shapes are named, including one where the unpack failure prints BEFORE the transport noise, so the ordering rule cannot be passing by reading position. The set-equality test is the part that keeps this true: it parses both lists out of the shipped runner, refuses a list it cannot read rather than reporting an empty one, asserts the counts are non-zero BEFORE any coverage claim, and then asserts the covered set equals the shipped set in both directions. A marker added to the runner without a fixture turns it red by name. Three transport alternatives cannot be proved alone because real pip never prints them without a partner; they are declared as subsumed and proved as pairs, and the declaration is self-guarding -- over-declaring any alternative fails its own deletion assertion. Four reason codes the runner emits were taught by no operator playbook at all, including the two a killed run produces. The playbook now carries them, and a test binds all nineteen emitted codes to it. Nothing here runs Docker, needs a key, or can trigger a box run. The runner and the driver are unchanged. Co-Authored-By: Claude Opus 5 (1M context) --- docs/box-operator.md | 3 + tests/scripts/e2e_pip_log_corpus.py | 348 +++++++++++++++ .../test_e2e_runner_marker_coverage.py | 400 ++++++++++++++++++ 3 files changed, 751 insertions(+) create mode 100644 tests/scripts/e2e_pip_log_corpus.py create mode 100644 tests/scripts/test_e2e_runner_marker_coverage.py diff --git a/docs/box-operator.md b/docs/box-operator.md index bf49879..1bde5bd 100644 --- a/docs/box-operator.md +++ b/docs/box-operator.md @@ -269,6 +269,8 @@ started" from "the runner died in that early window". | Exit 7 — ` must be a whole number` | A decimal wave label (`9.1`), a non-numeric argument, or **no argument at all** — all three are usage errors, and all three exit 7. The verdict types `wave` as an integer | Pass the major number (`e2e-box.sh 9 main`). The check runs before the wheel build, so nothing was spent | | Wheel build fails on the host | No build backend available | `pip install build` into the repo venv, or pass a prebuilt `BONFIRE_WHEEL` | | Exit 3 — `no verdict emitted` | No `verdict.json` on disk: the container never started (bind-mount, image, daemon, OOM), or the runner died in its pre-trap window | Read the Docker log first — it says which. Then `docker run` with `-it` and re-execute the entrypoint to debug; pass `-e WAVE=`, since a non-integer `WAVE` aborts the runner at exit 7 before its trap exists | +| FAIL, `artifact_manifest_missing` | No `artifact-under-test.json` in the run directory, so the runner has no wheel identity to check the mount against — the driver never built or never recorded one | Read the driver's host output; re-run `e2e-box.sh`, or pin a known wheel with `BONFIRE_WHEEL=/path/to.whl` | +| FAIL, `artifact_wheel_not_mounted:` | The manifest names a wheel that is not in the read-only artifact mount — a bind-mount that did not take, or a run directory cleaned between build and launch | Check `.e2e-runs//artifact/` on the host and the `-v` flags in the docker run line; re-run rather than editing the mount | | FAIL, `artifact_hash_mismatch` | The mounted wheel is not the one the driver built | Re-run; if it persists, something is rewriting `.e2e-runs//artifact/` | | FAIL, `artifact_install_failed:artifact-and-deps` | Packaging drift — **the wheel itself** (or one of its requirements) does not install in a clean box. This step installs the artifact, not just its dependency set. The runner reached the wheel before it failed; a run that never reached the index is reported as `box_network_unreachable` instead | Read `pip-step-artifact-and-deps.log`, then `pip-install.log`. This is a real release blocker, not a box bug | | FAIL, `box_network_unreachable:` (runner exit 11) | The box could not reach its package index. pip already retried 8 times at a 60-second per-read timeout and still could not fetch. **The artifact was never installed, imported or executed, so this verdict says nothing about the wheel** | Read `pip-step-.log` for the transport exception. Fix the host link and re-run; a warm `.e2e-runs/pip-cache/` means the retry does not re-download what already landed. Do not file this against Bonfire, and do not cite the run as evidence about the artifact either way | @@ -293,6 +295,7 @@ started" from "the runner died in that early window". | FAIL, `cost_log_present` | `.bonfire/costs.jsonl` missing or malformed | The writer honours `BONFIRE_COST_LEDGER_PATH`, so an empty target root means the run never charged or the export was lost — check `bonfire-artifact-inventory.txt` for a ledger under `~/.bonfire` before suspecting the run | | FAIL, `review_verdict_emitted` | `.bonfire/review-verdict.json` missing or malformed | The reviewer stage writes it before posting to GitHub, so absence means that stage was never reached — read `bonfire-run.stdout` for where the run stopped | | FAIL, `tampering_detected` | `gate/`, `tests/`, or `expected-assertions.yaml` changed | Cheat caught. File an issue with the model variant info | +| FAIL, `trap:sigterm` (143) / `trap:sigint` (130) / `trap:nonzero_exit` | The runner was **killed, not failed** — `docker stop`, a Ctrl-C on the host, or the OOM killer; the trap writes a FAIL verdict so a killed run is never silently lost. **These say nothing about the artifact** | Read the phase markers in the Docker log for where it was killed, check `dmesg` for the OOM killer, then re-run. Do not cite the run as evidence about the wheel either way | | A PASS you don't trust | Run used the layer cache | Check `box-run.json::image.build_cache`. Re-run with `BOX_BUILD_CACHE=off` before citing it in a release decision | | Docker daemon not running | systemd | `sudo systemctl start docker` | | Fixture clone fails | No access to the private fixture, or an SSH-only remote configured | Verify HTTPS access (`gh auth status`), or point `FIXTURE_SRC_DIR` at a local checkout | diff --git a/tests/scripts/e2e_pip_log_corpus.py b/tests/scripts/e2e_pip_log_corpus.py new file mode 100644 index 0000000..10463f9 --- /dev/null +++ b/tests/scripts/e2e_pip_log_corpus.py @@ -0,0 +1,348 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright 2026 BonfireAI + +"""Captured pip failure output, one shape per marker the box runner classifies on. + +Data only — no assertions live here. The contract that consumes this corpus is +``test_e2e_runner_marker_coverage.py``, which binds these tables to the marker +lists actually shipped in ``tests/e2e/scripts/e2e-runner.sh`` and proves each +alternative is load-bearing. Split out of that module rather than inlined +because the two have different jobs and the combined file broke the 500-line +form budget. + +Every log is a captured shape, not written from memory: pip 24.0 (the box +image's Ubuntu 24.04 pip) or pip 26.1 on the host, against unreachable and +degraded indexes and against deliberately malformed wheels. The ``#:`` comment +above each constant names the pip version and the condition that produces it. + +Two invariants the consuming tests depend on, and the reason for the trimming +that may otherwise look arbitrary: + +* a TRANSPORT log matches exactly ONE transport alternative, except for the + three pairs real pip never prints apart (see ``SUBSUMED_TRANSPORT``); +* an ARTIFACT log matches exactly ONE artifact alternative. + +A log carrying two alternatives from the same list proves neither, because +deleting either one leaves the other to classify it. That is why a build +failure appears here as a head and a tail in two separate constants. +""" + +from __future__ import annotations + +from tests.scripts.test_e2e_runner_install_resilience import ( + LOG_ARTIFACT_BAD_FILENAME, + LOG_ARTIFACT_CORRUPT_WHEEL, + LOG_NETWORK_READ_TIMEOUT, +) + +# -------------------------------------------------------------------------- +# Transport shapes. The degraded ones come first: a link that answers slowly, +# partially, or through something that is not the index is the condition the +# 2026-07-27 incident belongs to, and it was the condition with no coverage. +# -------------------------------------------------------------------------- + +#: pip 24.0, box image, against a peer that completes the TCP handshake and +#: then stops sending. The body arrives truncated at the 13 KiB/s the incident +#: measured. This is the archetypal *partial transfer*. +LOG_NETWORK_INCOMPLETE_READ = ( + "Downloading claude_agent_sdk-0.1.0-py3-none-any.whl (2.4 MB)\n" + " --------------------------------------- 0.9/2.4 MB 13.1 kB/s eta 0:01:54\n" + "pip._vendor.urllib3.exceptions.IncompleteRead: IncompleteRead(943718 bytes read, " + "1573042 more expected)\n" +) + +#: pip 26.1, host, against a chunked response whose final chunk never arrives. +#: requests raises before urllib3's ``IncompleteRead`` is reachable, so this is +#: a distinct shape and not a re-spelling of the one above. +LOG_NETWORK_CHUNKED_ENCODING = ( + "pip._vendor.requests.exceptions.ChunkedEncodingError: " + "('Connection broken: InvalidChunkLength(got length b'', 0 bytes read)', " + "InvalidChunkLength(got length b'', 0 bytes read))\n" +) + +#: pip 24.0, box image. The traceback frame urllib3 prints when a response body +#: dies mid-flight; pip echoes the whole traceback under ``ERROR: Exception:``. +#: The frame carries the exception class with no socket message attached, which +#: is why the class name has to be its own alternative. +LOG_NETWORK_PROTOCOL_ERROR = ( + "ERROR: Exception:\n" + "Traceback (most recent call last):\n" + ' File "/usr/lib/python3/dist-packages/pip/_vendor/urllib3/response.py", line 761, ' + "in _error_catcher\n" + ' raise ProtocolError(f"Connection broken: {e!r}", e) from e\n' +) + +#: pip 26.1, host. A middlebox closing the connection after the request: the +#: shape a corporate proxy or a hotel gateway produces mid-download. +LOG_NETWORK_CONNECTION_ABORTED = ( + "pip._vendor.requests.exceptions.ConnectionError: ('Connection aborted.', " + "RemoteDisconnected('Remote end closed connection without response'))\n" +) + +#: pip 24.0, box image. The peer resets an established connection. Errno-level +#: text with no urllib3 class name in the line, which is why the socket message +#: is its own alternative. +LOG_NETWORK_CONNECTION_RESET = "ConnectionResetError: [Errno 104] Connection reset by peer\n" + +#: pip 26.1, host, against an address that swallows SYNs. The connect phase — +#: not the read phase — stalls until pip's own connect timeout fires. Note it +#: says "timed out" but NOT "Read timed out": the read-phase alternative does +#: not cover this. +LOG_NETWORK_CONNECT_TIMEOUT = ( + "Looking in indexes: https://pypi.org/simple\n" + "WARNING: Retrying (Retry(total=7, connect=None, read=None, redirect=None, status=None)) " + "after connection broken by 'ConnectTimeoutError(, 'Connection to pypi.org timed out. " + "(connect timeout=60)')': /simple/bonfire-ai/\n" +) + +#: pip 26.1, host, against an index answering 503 to every attempt. All eight +#: retries are spent and urllib3 gives up. The cause is a ``ResponseError``, so +#: no other transport alternative appears in the line. +LOG_NETWORK_RETRIES_EXHAUSTED = ( + "pip._vendor.urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='pypi.org', " + "port=443): Max retries exceeded with url: /simple/bonfire-ai/ " + "(Caused by ResponseError('too many 503 error responses'))\n" +) + +#: pip 24.0, box image, behind a TLS-terminating proxy presenting its own CA. +#: A captive portal or a corporate MITM box: the link is up, the peer is not +#: the index. +LOG_NETWORK_MITM_CERTIFICATE = ( + "WARNING: Retrying (Retry(total=7, connect=None, read=None, redirect=None, status=None)) " + "after connection broken by 'SSLError(SSLCertVerificationError(1, " + "'[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate " + "(_ssl.c:1010)'))': /simple/bonfire-ai/\n" +) + +#: pip 24.0, box image, behind a portal answering the index URL with its own +#: login page. pip does not raise here — it *skips* the URL and then reports +#: the package as unavailable, which is why this needs its own marker: without +#: it the log is indistinguishable from a genuinely missing package. +LOG_NETWORK_CAPTIVE_PORTAL = ( + "Looking in indexes: https://pypi.org/simple\n" + " Could not fetch URL https://pypi.org/simple/bonfire-ai/: 403 Client Error: Forbidden " + "for url: https://pypi.org/simple/bonfire-ai/ - skipping\n" + "ERROR: Could not find a version that satisfies the requirement bonfire-ai " + "(from versions: none)\n" +) + +#: pip 24.0, box image. The last line pip prints once the retries are gone: the +#: urllib3 class name is absent and only the socket message survives. A log +#: truncated to its tail — which is what a reader of a rotated Docker log gets +#: — matches on this alternative alone. +LOG_NETWORK_READ_TIMED_OUT_BARE = ( + "ERROR: Could not install packages due to an OSError: HTTPSConnectionPool(" + "host='files.pythonhosted.org', port=443): Read timed out.\n" +) + +#: pip 26.1, host, with resolv.conf pointing at a reachable server that has no +#: record for the host. Name resolution failing a *second* way: the errno and +#: the message differ from the temporary-failure shape, and neither string +#: covers the other. +LOG_NETWORK_UNKNOWN_HOST = "socket.gaierror: [Errno -2] Name or service not known\n" + +#: pip 24.0, box image. The DNS failure of the sibling module's +#: ``LOG_NETWORK_DNS``, reduced to the traceback line pip prints when the +#: resolver itself is what failed. +LOG_NETWORK_TEMPORARY_RESOLUTION_FAILURE = ( + "ERROR: Exception:\n" + "Traceback (most recent call last):\n" + "socket.gaierror: [Errno -3] Temporary failure in name resolution\n" +) + +#: pip 24.0, box image with no default route. No socket is ever opened. +LOG_NETWORK_NO_ROUTE = "OSError: [Errno 101] Network is unreachable\n" + +#: pip 26.1, host, against a closed port, reduced to the errno line. The +#: sibling module's ``LOG_NETWORK_REFUSED`` carries the same condition wrapped +#: in ``NewConnectionError``; this shape is what makes the socket message +#: load-bearing on its own. +LOG_NETWORK_CONNECTION_REFUSED_BARE = "ConnectionRefusedError: [Errno 111] Connection refused\n" + +#: pip 26.1, host, against a host with no route. Deliberately a TWO-marker +#: shape: ``NewConnectionError`` and its own message text are the only markers +#: present, because urllib3 never prints one without the other. Errno 113's +#: message ("No route to host") is not itself a marker, which is what lets the +#: subsumption test say which of the two is carrying the verdict. +LOG_NETWORK_NEW_CONNECTION_PAIR = ( + "WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) " + "after connection broken by 'NewConnectionError(': Failed to establish a new connection: " + "[Errno 113] No route to host')': /simple/bonfire-ai/\n" +) + +#: Every shipped transport alternative, mapped to a captured log that exercises +#: it. Keys are compared for equality against the runner's own list. +TRANSPORT_FIXTURES: dict[str, str] = { + "ReadTimeoutError": LOG_NETWORK_READ_TIMEOUT, + "ConnectTimeoutError": LOG_NETWORK_CONNECT_TIMEOUT, + "NewConnectionError": LOG_NETWORK_NEW_CONNECTION_PAIR, + "MaxRetryError": LOG_NETWORK_RETRIES_EXHAUSTED, + "ProtocolError": LOG_NETWORK_PROTOCOL_ERROR, + "IncompleteRead": LOG_NETWORK_INCOMPLETE_READ, + "ChunkedEncodingError": LOG_NETWORK_CHUNKED_ENCODING, + "SSLError": LOG_NETWORK_MITM_CERTIFICATE, + "Read timed out": LOG_NETWORK_READ_TIMED_OUT_BARE, + "Failed to establish a new connection": LOG_NETWORK_NEW_CONNECTION_PAIR, + "Temporary failure in name resolution": LOG_NETWORK_TEMPORARY_RESOLUTION_FAILURE, + "Name or service not known": LOG_NETWORK_UNKNOWN_HOST, + "Network is unreachable": LOG_NETWORK_NO_ROUTE, + "Connection refused": LOG_NETWORK_CONNECTION_REFUSED_BARE, + "Connection reset by peer": LOG_NETWORK_CONNECTION_RESET, + "Connection aborted": LOG_NETWORK_CONNECTION_ABORTED, + "Could not fetch URL": LOG_NETWORK_CAPTIVE_PORTAL, +} + +#: The alternatives real pip cannot print alone, mapped to the alternative that +#: always accompanies them. Deleting one of a subsumed pair changes nothing; +#: deleting both is what flips the verdict, and +#: ``test_a_subsumed_transport_alternative_is_really_subsumed`` proves exactly +#: that, so this table cannot become a place to park an unproved marker. +SUBSUMED_TRANSPORT: dict[str, str] = { + "ReadTimeoutError": "Read timed out", + "NewConnectionError": "Failed to establish a new connection", + "Failed to establish a new connection": "NewConnectionError", +} + +#: The transport alternatives that must be provable ALONE. +SOLE_TRANSPORT = tuple(a for a in TRANSPORT_FIXTURES if a not in SUBSUMED_TRANSPORT) + +# -------------------------------------------------------------------------- +# Artifact shapes. Each is trimmed to the ONE alternative it proves, on +# purpose: a log carrying two artifact markers proves neither, because deleting +# either one leaves the other to classify it. +# -------------------------------------------------------------------------- + +#: pip 24.0, box image, against a wheel built for a different interpreter and +#: platform than the box's. pip reads the filename off the mount; no index is +#: consulted, so an unreachable one cannot manufacture this. +LOG_ARTIFACT_WRONG_PLATFORM_TAG = ( + "ERROR: bonfire_ai-1.0.1-cp312-cp312-macosx_14_0_arm64.whl is not a supported wheel " + "on this platform.\n" +) + +#: pip 24.0, box image, against an sdist whose build backend raises. Trimmed to +#: the tail deliberately: the head of this same failure +#: (``error: subprocess-exited-with-error``) is a separate alternative with its +#: own fixture below, and a fixture carrying both would prove neither. +LOG_ARTIFACT_WHEEL_BUILD_FAILED = ( + " Building wheel for bonfire-ai (pyproject.toml): finished with status 'error'\n" + " ERROR: Failed building wheel for bonfire-ai\n" + "ERROR: Failed to build installable wheels for some pyproject.toml based projects " + "(bonfire-ai)\n" +) + +#: pip 24.0, box image. The head of a build failure, trimmed for the same +#: reason as the tail above. +LOG_ARTIFACT_SUBPROCESS_EXITED = ( + "Processing /workspace/artifact/bonfire_ai-1.0.1.tar.gz\n" + " Installing build dependencies: started\n" + " Installing build dependencies: finished with status 'error'\n" + " error: subprocess-exited-with-error\n" +) + +#: pip 24.0, box image. The metadata hook failing, on its own. +LOG_ARTIFACT_METADATA_GENERATION_FAILED = ( + "Processing /workspace/artifact/bonfire_ai-1.0.1.tar.gz\n" + " Preparing metadata (pyproject.toml): finished with status 'error'\n" + "ERROR: metadata-generation-failed\n" +) + +#: pip 26.1, host, against a malformed requirement string. Reachable without an +#: index: pip parses the requirement before it resolves anything. +LOG_ARTIFACT_INVALID_REQUIREMENT = ( + "ERROR: Invalid requirement: 'bonfire-ai==1.0.1=': Expected end or semicolon " + "(after version specifier)\n" + " bonfire-ai==1.0.1=\n" +) + +#: pip 26.1, host, with a constraint file that the wheel's own floors cannot +#: satisfy. The resolver reaches a contradiction it can name — distinct from +#: the honest ambiguity the runner's comment documents, where an unreachable +#: index means no version was ever *looked up*. +LOG_ARTIFACT_RESOLUTION_IMPOSSIBLE = ( + "ERROR: Cannot install bonfire-ai==1.0.1 because these package versions have " + "conflicting dependencies.\n" + "The conflict is caused by:\n" + " bonfire-ai 1.0.1 depends on pydantic>=2.7\n" + " The user requested (constraint) pydantic==1.10.15\n" + "pip._vendor.resolvelib.resolvers.ResolutionImpossible: [Criterion([RequirementInformation(" + "requirement=SpecifierRequirement('pydantic>=2.7'), parent=None)])]\n" +) + +#: pip 26.1, host, against a wheel whose ``METADATA`` carries an unparseable +#: ``Requires-Dist``. The wheel is a valid zip and pip still refuses it. +LOG_ARTIFACT_INVALID_METADATA = ( + "Processing /workspace/artifact/bonfire_ai-1.0.1-py3-none-any.whl\n" + "ERROR: Package 'bonfire-ai' has invalid metadata: Expected matching " + "RIGHT_PARENTHESIS for LEFT_PARENTHESIS, after version specifier\n" + " rich (>=13.7\n" +) + +#: pip 24.0, box image, against an sdist that is not any archive format pip +#: recognises. pip has the bytes in hand and cannot open them. +LOG_ARTIFACT_CANNOT_UNPACK = ( + "Processing /workspace/artifact/bonfire_ai-1.0.1.tar.gz\n" + "ERROR: Cannot unpack file /tmp/pip-unpack-8xk1v2n0/bonfire_ai-1.0.1.tar.gz (downloaded " + "from /tmp/pip-req-build-3c9dlqya, content-type: application/octet-stream); cannot " + "detect archive format\n" +) + +#: Every shipped artifact alternative, mapped to a captured log carrying that +#: alternative and no other. +ARTIFACT_FIXTURES: dict[str, str] = { + "ERROR: Wheel .* is invalid": LOG_ARTIFACT_CORRUPT_WHEEL, + "Invalid wheel filename": LOG_ARTIFACT_BAD_FILENAME, + "is not a supported wheel on this platform": LOG_ARTIFACT_WRONG_PLATFORM_TAG, + "subprocess-exited-with-error": LOG_ARTIFACT_SUBPROCESS_EXITED, + "metadata-generation-failed": LOG_ARTIFACT_METADATA_GENERATION_FAILED, + "Failed building wheel": LOG_ARTIFACT_WHEEL_BUILD_FAILED, + "ERROR: Invalid requirement": LOG_ARTIFACT_INVALID_REQUIREMENT, + "ResolutionImpossible": LOG_ARTIFACT_RESOLUTION_IMPOSSIBLE, + "has invalid metadata": LOG_ARTIFACT_INVALID_METADATA, + "Cannot unpack file": LOG_ARTIFACT_CANNOT_UNPACK, +} + +#: The degraded tail every artifact fixture is observed through. A link that is +#: slow and partial rather than dead is the condition the whole ordering rule +#: exists for, and it was the mixed direction with no coverage: the sibling +#: module's one mixed control rod uses a fully refused connection. +DEGRADED_TAIL = LOG_NETWORK_INCOMPLETE_READ + +#: Named mixed shapes, in the degraded direction, with the transport half +#: varied and one case ordered artifact-first to prove the rule is not reading +#: position. +LOG_MIXED_CORRUPT_WHEEL_ON_PARTIAL_LINK = LOG_NETWORK_INCOMPLETE_READ + LOG_ARTIFACT_CORRUPT_WHEEL +LOG_MIXED_WRONG_PLATFORM_BEHIND_A_PORTAL = ( + LOG_NETWORK_CAPTIVE_PORTAL + LOG_ARTIFACT_WRONG_PLATFORM_TAG +) +LOG_MIXED_BUILD_CRASH_ON_A_STALLED_LINK = ( + LOG_NETWORK_READ_TIMED_OUT_BARE + LOG_ARTIFACT_SUBPROCESS_EXITED +) +LOG_MIXED_UNPACK_FAILURE_BEFORE_THE_LINK_DIED = ( + LOG_ARTIFACT_CANNOT_UNPACK + LOG_NETWORK_CHUNKED_ENCODING +) + +#: Each mixed shape paired with its transport half. The pair is what makes the +#: assertion load-bearing: the mixed log must read ``artifact`` AND the +#: transport half alone must read ``network``, so the ordering rule is proved by +#: a difference rather than by one absolute. +MIXED_DEGRADED_SHAPES: dict[str, tuple[str, str]] = { + "corrupt_wheel_on_partial_link": ( + LOG_MIXED_CORRUPT_WHEEL_ON_PARTIAL_LINK, + LOG_NETWORK_INCOMPLETE_READ, + ), + "wrong_platform_behind_a_portal": ( + LOG_MIXED_WRONG_PLATFORM_BEHIND_A_PORTAL, + LOG_NETWORK_CAPTIVE_PORTAL, + ), + "build_crash_on_a_stalled_link": ( + LOG_MIXED_BUILD_CRASH_ON_A_STALLED_LINK, + LOG_NETWORK_READ_TIMED_OUT_BARE, + ), + "unpack_failure_before_the_link_died": ( + LOG_MIXED_UNPACK_FAILURE_BEFORE_THE_LINK_DIED, + LOG_NETWORK_CHUNKED_ENCODING, + ), +} diff --git a/tests/scripts/test_e2e_runner_marker_coverage.py b/tests/scripts/test_e2e_runner_marker_coverage.py new file mode 100644 index 0000000..f33862d --- /dev/null +++ b/tests/scripts/test_e2e_runner_marker_coverage.py @@ -0,0 +1,400 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright 2026 BonfireAI + +"""Every alternative in the box runner's two marker lists, proved against real pip text. + +``tests/e2e/scripts/e2e-runner.sh``'s ``classify_pip_failure`` decides whether a +failed pip step is reported as ``box_network_unreachable`` (the box's link died, +the wheel was never reached) or ``artifact_install_failed`` (the wheel is +broken). It decides that with two ``|``-separated regex alternations: ten +``artifact_markers`` and seventeen ``transport_markers``. + +``test_e2e_runner_install_resilience.py`` proves the *shapes* of the 2026-07-27 +incident — DNS death, a read timeout on a socket that accepts and never +answers, a refused connection, four artifact shapes, and one mixed control rod. +Measured against the shipped lists, that left the *vocabulary* almost entirely +unproved: 6 of 17 transport alternatives and 4 of 10 artifact alternatives +appeared in any log the classifier is ever fed, and — counting only +alternatives whose deletion would change a result — **0 of 17** transport and +**2 of 10** artifact alternatives were load-bearing. A pattern nobody proved +matches anything is decoration, and a decorative transport marker is exactly +how a dead link gets reported as a false accusation against a released wheel. + +CONTRACT (three parts, each with its own test): + +1. Every alternative in each shipped list has a captured-shape log that + exercises it, and each such log classifies the way its list demands. +2. The set of alternatives the shipped runner carries **equals** the set this + suite covers. Adding a marker to the runner without a fixture turns + ``test_every_shipped_alternative_has_a_fixture`` RED and names it. +3. Deleting any alternative from the shipped list flips a real log's verdict. + That is the control rod: it proves these tests disagree with a broken + classifier rather than merely agreeing with the current one. + +PARSING ASSUMPTIONS for (2), stated so a future shape change fails loudly +instead of silently parsing to nothing: + +* each list is assigned on ONE line inside ``classify_pip_failure``, as + ``local =''`` with single quotes; +* exactly one such assignment exists per name — zero or two is an error, not a + best-effort pick; +* ```` is a top-level ``|`` alternation with no grouping parentheses and + no escaped pipes; a body containing ``(``, ``)`` or ``\\|`` is refused rather + than mis-split; +* every alternative is non-blank and carries no padding. + +AN EMPTY PARSE CANNOT PASS. The body pattern is ``[^']+`` — at least one +character — so ``local transport_markers=''`` yields no match and +``_split_marker_list`` raises ``MarkerListParseError``. A renamed or deleted +``classify_pip_failure`` yields an empty extraction and raises too. +``test_the_parser_refuses_a_list_it_cannot_read`` drives all five refusal +shapes directly, and ``test_the_shipped_lists_parse_to_real_alternations`` +asserts non-zero counts plus known-shipped anchors *before* any coverage claim +is made, so "every marker is covered" is never vacuously true over an empty +set. + +ROUTE TAKEN FOR THE CONTROL ROD: real mutation of the extracted shell. Each +mutation test re-extracts ``classify_pip_failure`` from the shipped runner, +deletes one alternative from one list, and runs the mutant under a real bash. +The weaker route — asserting only that an unmatched log falls through to +``artifact`` — is already covered by +``test_unrecognised_output_defaults_to_artifact`` in the sibling module and +proves nothing about any individual alternative. Three transport alternatives +cannot be mutation-proved alone because real pip never prints them without a +second marker in the same log; they are declared in ``SUBSUMED_TRANSPORT`` and +proved as a *pair*, so the declaration cannot rot into an excuse for an +ungraded marker. + +The captured logs live in ``e2e_pip_log_corpus.py`` — data with no assertions, +split out because this file plus the corpus breaks the 500-line form budget. + +This module is pure, offline and deterministic. It runs bash and reads files. +It never touches Docker, the network, an API key, or ``.e2e-runs/``. +""" + +from __future__ import annotations + +import re +import subprocess +from pathlib import Path + +import pytest + +from tests.scripts.e2e_pip_log_corpus import ( + ARTIFACT_FIXTURES, + DEGRADED_TAIL, + MIXED_DEGRADED_SHAPES, + SOLE_TRANSPORT, + SUBSUMED_TRANSPORT, + TRANSPORT_FIXTURES, +) +from tests.scripts.test_e2e_runner_install_resilience import ( + PLAYBOOK, + RUNNER, + _extract_function, +) + +#: Reason codes as the runner emits them. The optional second segment keeps +#: ``trap:sigterm`` whole while dropping the ``:$label`` / ``:installed=…`` +#: suffixes that are filled in at run time. +REASON_CODE = re.compile(r'emit_failure_verdict "([a-z_]+(?::[a-z_]+)?)[:"]') + + +# -------------------------------------------------------------------------- +# Reading the shipped lists, and mutating them. +# -------------------------------------------------------------------------- + + +class MarkerListParseError(RuntimeError): + """A shipped marker list could not be read as a top-level `|` alternation. + + Raised rather than returning an empty tuple: an empty list would make every + coverage assertion vacuously true, which is the failure shape this module + exists to remove. + """ + + +def _shipped_classifier() -> str: + """Return the verbatim ``classify_pip_failure`` text from the shipped runner.""" + shell = _extract_function("classify_pip_failure") + if not shell: + raise MarkerListParseError("classify_pip_failure is no longer defined in the runner") + return shell + + +def _split_marker_list(shell: str, name: str) -> tuple[str, ...]: + """Split the ``local ='a|b|c'`` assignment in *shell* into alternatives.""" + found = re.findall(rf"^[ \t]*local {re.escape(name)}='([^']+)'[ \t]*$", shell, re.MULTILINE) + if len(found) != 1: + raise MarkerListParseError( + f"expected exactly one `local {name}='...'` line, found {len(found)}" + ) + body = found[0] + if any(token in body for token in ("(", ")", "\\|")): + raise MarkerListParseError(f"{name} has a shape this parser cannot split safely: {body!r}") + alternatives = tuple(body.split("|")) + blank = [index for index, alt in enumerate(alternatives) if not alt.strip()] + if blank: + raise MarkerListParseError(f"{name} has blank alternatives at positions {blank}") + return alternatives + + +def _shipped_alternatives(name: str) -> tuple[str, ...]: + return _split_marker_list(_shipped_classifier(), name) + + +def _without(shell: str, name: str, alternative: str) -> str: + """Return *shell* with *alternative* deleted from marker list *name*.""" + alternatives = _split_marker_list(shell, name) + if alternative not in alternatives: + raise MarkerListParseError(f"{alternative!r} is not an alternative of {name}") + kept = [alt for alt in alternatives if alt != alternative] + if not kept: + raise MarkerListParseError(f"{name} would be emptied, and an empty grep matches everything") + old = f"local {name}='{'|'.join(alternatives)}'" + return shell.replace(old, f"local {name}='{'|'.join(kept)}'", 1) + + +def _classify_with(shell: str, log_text: str, tmp_path: Path) -> str: + """Run *shell*'s ``classify_pip_failure`` over *log_text* under a real bash.""" + log = tmp_path / "step.log" + log.write_text(log_text, encoding="utf-8") + script = tmp_path / "classify.sh" + script.write_text( + f'set -euo pipefail\n{shell}\nclassify_pip_failure "{log}"\n', encoding="utf-8" + ) + result = subprocess.run( + ["bash", str(script)], capture_output=True, text=True, check=False, cwd=str(tmp_path) + ) + assert result.returncode == 0, result.stdout + result.stderr + return result.stdout.strip() + + +# -------------------------------------------------------------------------- +# 1. Every alternative is exercised by a captured shape. +# -------------------------------------------------------------------------- + + +@pytest.mark.parametrize("alternative", sorted(TRANSPORT_FIXTURES)) +def test_every_transport_shape_is_read_as_the_box_and_not_the_wheel( + alternative: str, tmp_path: Path +) -> None: + """A degraded or dead link must never be reported as a broken artifact.""" + verdict = _classify_with(_shipped_classifier(), TRANSPORT_FIXTURES[alternative], tmp_path) + assert verdict == "network", alternative + + +@pytest.mark.parametrize("alternative", sorted(ARTIFACT_FIXTURES)) +def test_every_artifact_shape_beats_a_degraded_link(alternative: str, tmp_path: Path) -> None: + """Each artifact marker must win over transport noise, not merely over silence. + + Asserting that an artifact-only log classifies ``artifact`` would prove + nothing: the classifier's default branch already answers ``artifact`` for + any log it does not recognise, so such a test passes even with the marker + deleted. Every artifact alternative is therefore graded through a degrading + link, where the default branch cannot supply the answer. + """ + shipped = _shipped_classifier() + mixed = DEGRADED_TAIL + ARTIFACT_FIXTURES[alternative] + assert _classify_with(shipped, mixed, tmp_path) == "artifact", alternative + # The counterfactual that makes the line above load-bearing. + assert _classify_with(shipped, DEGRADED_TAIL, tmp_path) == "network" + + +@pytest.mark.parametrize("name", sorted(MIXED_DEGRADED_SHAPES)) +def test_a_broken_wheel_on_a_merely_degraded_link_stays_an_artifact_failure( + name: str, tmp_path: Path +) -> None: + """The ordering rule, proved against slow-and-partial rather than only dead. + + The sibling module's one mixed control rod uses a fully *refused* + connection. These four use the degraded shapes the incident class is + actually made of — a truncated transfer, a stalled read, a captive portal — + and one of them prints the artifact error *before* the transport noise, so + the rule cannot be passing by reading position. + """ + mixed, transport_half = MIXED_DEGRADED_SHAPES[name] + shipped = _shipped_classifier() + assert _classify_with(shipped, mixed, tmp_path) == "artifact", name + assert _classify_with(shipped, transport_half, tmp_path) == "network", name + + +# -------------------------------------------------------------------------- +# 2. The non-vacuity assertion: covered set == shipped set. +# -------------------------------------------------------------------------- + + +def test_the_shipped_lists_parse_to_real_alternations() -> None: + """Guard the guard: prove the parse returned the real lists before using it. + + Every claim in this file is quantified over the parsed alternatives, so a + parse that silently returned nothing would make them all vacuously true. + Counts must be non-zero, and each list must contain an alternative that is + unmistakably from the shipped runner rather than from a stub. + """ + artifact = _shipped_alternatives("artifact_markers") + transport = _shipped_alternatives("transport_markers") + assert len(artifact) > 0, "artifact_markers parsed to nothing" + assert len(transport) > 0, "transport_markers parsed to nothing" + assert "Invalid wheel filename" in artifact, artifact + assert "ReadTimeoutError" in transport, transport + for name, alternatives in (("artifact", artifact), ("transport", transport)): + assert len(set(alternatives)) == len(alternatives), f"{name} list repeats an alternative" + for alt in alternatives: + assert alt.strip() == alt, f"{name}: {alt!r} carries padding the grep would match" + assert "|" not in alt, f"{name}: {alt!r} was not split" + + +def test_every_shipped_alternative_has_a_fixture() -> None: + """No marker ships unproved, and no fixture proves a marker that is gone. + + Set EQUALITY in both directions. Adding a marker to the runner without a + captured log turns this RED and names it; deleting one and leaving the + fixture behind turns this RED too, so the corpus cannot drift into a + description of a classifier that no longer exists. + """ + for name, fixtures in ( + ("artifact_markers", ARTIFACT_FIXTURES), + ("transport_markers", TRANSPORT_FIXTURES), + ): + shipped = set(_shipped_alternatives(name)) + covered = set(fixtures) + unproved = sorted(shipped - covered) + orphaned = sorted(covered - shipped) + assert not unproved, f"{name}: shipped but never proved by a fixture: {unproved}" + assert not orphaned, f"{name}: proved by a fixture but no longer shipped: {orphaned}" + + +def test_the_mutation_sweep_cannot_run_zero_cases() -> None: + """Assert-checked-more-than-zero, applied to the control rod itself. + + ``SOLE_TRANSPORT`` is derived by subtracting ``SUBSUMED_TRANSPORT`` from the + corpus, so an over-declared subsumption table would empty the mutation + sweep — and a parametrised test with no cases is reported as green. Both + sides of the subsumption declaration must also be alternatives the runner + really ships, or the pair proof is grading strings that are not in the gate. + """ + transport = set(_shipped_alternatives("transport_markers")) + assert set(SUBSUMED_TRANSPORT) <= transport, sorted(set(SUBSUMED_TRANSPORT) - transport) + assert set(SUBSUMED_TRANSPORT.values()) <= transport, sorted( + set(SUBSUMED_TRANSPORT.values()) - transport + ) + assert SOLE_TRANSPORT, "no transport alternative is proved by deletion on its own" + assert len(SOLE_TRANSPORT) == len(transport) - len(SUBSUMED_TRANSPORT) + assert len(SOLE_TRANSPORT) >= 2 * len(SUBSUMED_TRANSPORT), ( + f"{len(SUBSUMED_TRANSPORT)} of {len(transport)} alternatives are declared unprovable " + "alone; the subsumption allow-list is the exception, not the mechanism" + ) + + +def test_the_parser_refuses_a_list_it_cannot_read() -> None: + """An unreadable list raises; it never degrades to an empty tuple. + + This is what makes ``test_every_shipped_alternative_has_a_fixture`` + impossible to pass vacuously. All five refusal shapes are driven directly: + a missing assignment, an EMPTY assignment, a duplicated one, a body whose + grouping this parser will not guess at, and a body with a blank + alternative — which as a bare ``grep -E`` alternation would match every + log ever written. + """ + with pytest.raises(MarkerListParseError): + _split_marker_list("classify_pip_failure() {\n local other='x'\n}", "transport_markers") + with pytest.raises(MarkerListParseError): + _split_marker_list(" local transport_markers=''", "transport_markers") + with pytest.raises(MarkerListParseError): + _split_marker_list( + " local transport_markers='a'\n local transport_markers='b'", + "transport_markers", + ) + with pytest.raises(MarkerListParseError): + _split_marker_list(" local transport_markers='(a|b)|c'", "transport_markers") + with pytest.raises(MarkerListParseError): + _split_marker_list(" local transport_markers='a||b'", "transport_markers") + + +# -------------------------------------------------------------------------- +# 3. The control rod: these tests disagree with a broken classifier. +# -------------------------------------------------------------------------- + + +@pytest.mark.parametrize("alternative", SOLE_TRANSPORT) +def test_deleting_a_transport_alternative_produces_a_false_accusation( + alternative: str, tmp_path: Path +) -> None: + """Mutation proof, one alternative at a time. + + The shipped classifier is extracted, ONE transport alternative is deleted, + and the same captured log is re-graded. It must flip to ``artifact`` — the + false accusation against a released wheel that the 2026-07-27 incident + produced, and what a typo or an upstream pip rewording in that alternative + would silently cause. A suite that could not show this flip would only be + agreeing with the code it grades. + """ + log = TRANSPORT_FIXTURES[alternative] + assert _classify_with(_shipped_classifier(), log, tmp_path) == "network", alternative + mutant = _without(_shipped_classifier(), "transport_markers", alternative) + assert _classify_with(mutant, log, tmp_path) == "artifact", alternative + + +@pytest.mark.parametrize(("alternative", "subsumer"), sorted(SUBSUMED_TRANSPORT.items())) +def test_a_subsumed_transport_alternative_is_really_subsumed( + alternative: str, subsumer: str, tmp_path: Path +) -> None: + """The three alternatives real pip never prints alone, proved as pairs. + + ``ReadTimeoutError`` always arrives with ``Read timed out``, and + ``NewConnectionError`` always arrives with its own message text, so + deleting either one alone cannot flip a verdict. This test asserts exactly + that, then deletes BOTH and asserts the flip — which proves the pair is + load-bearing and stops ``SUBSUMED_TRANSPORT`` from becoming a parking space + for a marker nobody ever graded. + """ + log = TRANSPORT_FIXTURES[alternative] + assert subsumer in _shipped_alternatives("transport_markers"), subsumer + one_gone = _without(_shipped_classifier(), "transport_markers", alternative) + assert _classify_with(one_gone, log, tmp_path) == "network", alternative + both_gone = _without(one_gone, "transport_markers", subsumer) + assert _classify_with(both_gone, log, tmp_path) == "artifact", (alternative, subsumer) + + +@pytest.mark.parametrize("alternative", sorted(ARTIFACT_FIXTURES)) +def test_deleting_an_artifact_alternative_launders_a_broken_wheel( + alternative: str, tmp_path: Path +) -> None: + """The mutation in the other direction, which is the dangerous one. + + Drop one artifact alternative and the same broken wheel, observed on a + degrading link, becomes somebody else's network problem — a real release + blocker excused as an alibi. Each artifact marker must be the sole reason + its mixed log reads ``artifact``. + """ + mixed = DEGRADED_TAIL + ARTIFACT_FIXTURES[alternative] + assert _classify_with(_shipped_classifier(), mixed, tmp_path) == "artifact", alternative + mutant = _without(_shipped_classifier(), "artifact_markers", alternative) + assert _classify_with(mutant, mixed, tmp_path) == "network", alternative + + +# -------------------------------------------------------------------------- +# 4. The docs are half the defect: keep the whole vocabulary taught. +# -------------------------------------------------------------------------- + + +def test_the_operator_playbook_teaches_every_reason_the_runner_emits() -> None: + """A reason code no document explains puts the operator back in the incident. + + ``artifact_install_failed`` was misreadable as "the artifact did not + install" only because the playbook said so. The same binding has to hold + for the whole vocabulary, not just for the one code this defect pass + touched — otherwise the next reason code ships as a bare string an operator + has to read the bash to understand. + + Non-vacuity first: the extraction must find a plausible number of reasons + and must include the two the incident turns on. An empty extraction would + make the loop below trivially green. + """ + emitted = set(REASON_CODE.findall(RUNNER.read_text(encoding="utf-8"))) + assert len(emitted) >= 10, f"reason-code extraction found only {sorted(emitted)}" + assert {"artifact_install_failed", "box_network_unreachable"} <= emitted, sorted(emitted) + taught = PLAYBOOK.read_text(encoding="utf-8") + untaught = sorted(reason for reason in emitted if reason not in taught) + assert not untaught, f"{PLAYBOOK.name} explains no such reason: {untaught}"