Conversation
…ionWorkflow, budgets, dedup/correlation, audit)
…id-session signing-key rotation
…cket), mid-session signing-key rotation, real-cluster Swarm walkthrough
…fix racy approval-card assertion in App.test.tsx
…python from either venv the CI job actually creates
…ainer resource contention between registry and m2_probe_link, with the matching manifest-guard pin update
… same Postgres-testcontainer contention the -race step already fixed
ingest-gateway's own Docker image (design.md §11 one-service/one-image) never installs the worker package, so TemporalWorkflowStarter's direct import of worker.workflows.investigation.InvestigationWorkflow raised ModuleNotFoundError on every real alert in any real deployment -- masked in tests only by a fake worker module the old unit test injected into sys.modules. Switched to Temporal's untyped (string) workflow-start form, which needs no import of worker's code at all. Found by actually running the e2e suite against real built images for the first time.
…nst real images
- E0: accept the real prestodb/presto:0.298 image's actual /v1/info
version string ("0.298-e121953", a build-hash suffix) without
reopening the loose-prefix hole a prior review closed (still rejects
0.297/0.299/0.2980 etc.)
- E1/E3: fix wrong tpch column names (l_orderkey/o_orderkey, the raw
TPC-H spec's prefixed names) -- Presto's built-in tpch connector uses
unprefixed names (orderkey), confirmed via DESCRIBE against the real
image
- E2: connector.name=hive matches no factory this image ships (real
name is hive-hadoop2) and crashes the whole coordinator rather than
just that catalog, defeating the scenario; hive-hadoop2 itself doesn't
take connection-url either. Switched to postgresql, a real JDBC
connector whose connection-url legitimately carries the embedded
password this scenario needs to prove gets redacted
- E3: resource-groups.configuration-manager/config-file are not
config.properties keys -- Presto's core Bootstrap validates
config.properties against only its own modules and fails startup
("Configuration property ... was not used") if they're appended
there instead of their own etc/resource-groups.properties file,
confirmed against the real image and Presto's own docs. Added that
file (empty by default) to the coordinator manifests and pointed E3's
fault injection at it instead
All four confirmed empirically against the real prestodb/presto:0.298
image outside Kubernetes before being applied here.
…t the top-level name The previous version's precondition (assert worker not in sys.modules) was order-dependent and false whenever services/worker/tests ran first in the same pytest process -- exactly what happened in the real functional CI job, which runs worker and gateway tests together in one shared venv. Switched to poisoning sys.modules with the None sentinel, which forces ImportError regardless of prior state -- but poisoning only the top-level 'worker' entry was still insufficient: CPython can resolve 'from worker.workflows.investigation import X' through an already-cached submodule entry without re-touching the poisoned parent, confirmed by reverting the fix locally and finding the old bug slipped through. Now poisons every existing worker/worker.* entry plus the top-level name pre-emptively. Verified red (old code, ModuleNotFoundError) before green (fixed code) in the exact combined worker+gateway session, and green in gateway-only isolation.
The e2e job's first-ever real-CI run (GitHub Actions, 4-vCPU) failed
B1/E1-E4: tests/e2e/presto/{coordinator,worker}.yaml had no resources:
block, so those pods were BestEffort QoS -- the first thing starved
when the node comes under pressure, unlike every product pod (which
gets resources: from the dbagent Helm chart). B1's own burst load
against ingest-gateway put real pressure on the single shared kind
node; Presto, unprotected, went unresponsive, and E1-E4 cascaded from
that (E1 fails 3s later on a direct Presto query; E2/E3 burn their
180s coordinator-rollout waits; E4 stalls on a platform that hasn't
recovered).
B1 itself is out of scope here: its request path (client -> ingest-
gateway -> Postgres) never touches Presto, so this fix cannot and does
not claim to explain or fix B1 -- that's a separate, still-open defect.
Fix: resources: added to Presto's two manifests, mock-llm and
webhook-capture (the other BestEffort pods in the same namespace;
mock-llm backs every LLM call, on E4's path); JVM MaxDirectMemorySize
bounded so a real OOM exits cleanly instead of SIGKILL; kubectl
diagnostics (events/describe/logs, timeout-bounded) wired into
run.sh's failure path, which previously uploaded only a phase-timing
log despite the CI step's name.
The regression test (tests/delivery/test_delivery_e2e_fixtures.py)
went through several rounds: an initial static text-parsing approach
proved repeatedly forgeable (a comment or string could satisfy it
without any real kubectl call ever running), so it was replaced with
an executable test -- a stub kubectl on PATH, run.sh sourced for real
under bash, a forced phase failure, assertions on real observed
output. A process-group cleanup fix (start_new_session + killpg) was
also needed so the test's own timeout can't orphan a real docker
build/kind cluster if the sourcing guard it depends on ever regresses.
…packaging defects design.md Section 11.3 (FP-IG-1..19) plus fix.md D1-D4 and the B1 oracle. - ingest transaction moved off the event loop (concurrency was exactly one) - advisory lock restores correlation atomicity that serialization had provided - bounded correlation lookup; probe params from one helper; derived CPU sizing - worker image was missing agents/prompts/*.txt since M3 - Presto resource-groups.properties no longer shipped empty (crashlooped from first deploy); deploy_presto gate no longer passes a crashlooping coordinator - _submit_query follows nextUri; B1 oracle no longer counts rejects as success - new reference-environment B1 benchmark; e2e link narrowed to what it can attest Sizing basis is recorded at cpus=16 and must be re-derived by the CI benchmark job on the four-vCPU reference runner.
design.md Section 11.3 errata passes 8-13, FP-IG-20..25. Answers the red B1 reference benchmark: one Python process peaks near 546 req/s against B1's 1,000 req/s offer. - gateway serves under uvicorn's worker manager (workers=4) via create_worker_app; Section 11's one-process rule amended to a per-image runtime topology table with two documented deviations (gateway 1+1+W, dashboard-web nginx master+workers) - B11 writer model 4 -> 7 instances; derivation rule unchanged, input moved - five-run sizing ledger with run-id/provenance validation; the recorded cpus=16 basis is void on provenance, so FP-IG-4/18/23 stay red until CI produces five valid four-vCPU runs - e2e moves to needs: functional so a red benchmark no longer hides it - temporal.task_queue now honoured by both gateway and worker (a non-default queue previously stranded investigations silently) Process topology observed, not predicted: step 0 records the five unchanged images, step 2 the rebuilt gateway. Known: four workers did not move the measured rate at the benchmark's 1,000-concurrency operating point (187.9/s vs 183.4/s single-process, off-reference). CI's reference run decides whether Section 11.3.3's closing branch applies.
FP-IG-23's gate ran in `functional` and is red until five ledger observations exist, but the only producer of an observation is B1 inside `benchmark`, which is `needs: functional`. CI could never bootstrap the ledger: run 31712981261 had `functional` red on that one test and both `benchmark` and `e2e` skipped. The gate moves, not the graph. The ledger surface splits into tests/delivery/test_delivery_sizing_ledger.py, collected only by a new `benchmark` step at index 20 (after B1 at 17), with `functional` gaining a matching --ignore. No needs: edge, job or other step moves, so EXPECTED_NEEDS_GRAPH and CONDITIONAL_JOBS are untouched. FP-IG-26/UT-IG-10 pin the property: the consumer must not be in the computed needs: closure of the producer. The closure is computed, not read from the pinned literal, so a bare EXPECTED_NEEDS_GRAPH update cannot satisfy it. Roots and --ignore/--ignore-glob operands resolve against each step's working-directory, and run: bodies are tokenized before splitting on the separators test_manifests already models (; && | newline) -- a wd-blind resolver finds five collectors and an &&-joined edit would otherwise re-create the deadlock green. CI arithmetic re-derived across all eight pin carriers: 92 steps, benchmark 21, 52 run: (47+5), 113 commands, 21 guarded, 14 pytest, COMMAND_OPERANDS row 3 at 23; UNPARSED_RUN_STEPS and the local unparsed_expected stay 5. Step 2b only. The ledger stays empty and its gate stays red until five valid benchmark runs at the collection head are recorded -- terminal state 1 of design.md 11.3.3 AC. The red now skips nothing. No product code changed.
…ad with append-only history
…ay-split, EPYC 9V74 stays unhostable
…d isolate the basis oracle
… and size the gateway chart from it
…essure and the postgres volume identity
…s; dedupe the launcher region splitter
…lify two stall comments
…ad of a spoofed address
…e the runner-pool machinery
…ench runbook wording
…, index the threshold checker
…n a false threshold verdict
…fest wording from job names and README
…ME functional example to CI
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.