Repository navigation
Investigate unexpectedly low Flask JSON throughput without changing official results #70
Description
Priority and purpose
P1 — recommended next task. Type: investigation, not a confirmed implementation bug. Explain the unexpectedly low published Flask JSON throughput before optimizing the implementation or adding more official comparisons.
Verified baseline
Audit baseline, rechecked on 2026-09-18: main at 9b92c74cba34bd5a4bfc389075b9bef135131034. Re-read current main and related issues before starting work; this SHA is evidence, not an instruction to ignore later changes.
The stored official report, measured on 2026-09-15 from source c8e033ea176366644f5bed5ded12cab79e6fcbeb in run 34925168324, contains:
| Flask endpoint | Selected requests/s | Mean response time |
|---|---|---|
/json |
158.42619180247627 | 315.4516753732944 ms |
/db/42 |
1207.2212656929482 | 41.387314663200996 ms |
The three /json throughput samples are approximately 159.101, 158.426 and 152.652 requests/s. The DB endpoint is about 7.62 times faster in this report. This is an observation requiring explanation, not proof of an invalid report, a Flask defect, or a universal framework ranking.
The entry point uses Waitress with threads=1 and asyncore_use_poll=True; the JSON handler constructs a small native Python dictionary. Server/runtime behavior is part of the measured stack.
Scope
- Inspect existing retained raw evidence first, if available. Record unavailable or expired artifacts honestly; do not regenerate and label them as original evidence.
- Reproduce the current stack using a bounded, explicitly non-publishing diagnostic. Record source/tree, dependency and image identities, host/runner, exact profile, endpoint order, readiness behavior, all raw samples, errors and cleanup results.
- Check relevant evidence across the application, Waitress, worker/IO scheduling, queue/log behavior, HTTP connections and the load generator. These are hypotheses to test, not established causes.
- Use same-host paired comparisons and one changed factor at a time. Control or document ordering/carry-over effects. Predefine the diagnostic matrix and report every attempted sample; do not retry until a preferred result appears.
- Explain whether the observation is reproducible, a configuration/runtime characteristic, a harness artifact, an implementation defect, or unresolved. Propose any production fix or baseline change separately after the cause is supported.
Acceptance criteria
- The baseline behavior and all diagnostic conditions are recorded with traceable evidence.
- The investigation distinguishes observed facts from hypotheses and states limitations when reproduction or attribution is unsuccessful.
- Diagnostic tooling has tests for non-publication, invalid/partial output rejection and scoped cleanup where tooling is added.
- No diagnostic value is presented as an official result or as a measured improvement in an existing publication.
- A supported explanation and recommended next action are recorded in the issue/PR; an unresolved outcome includes concrete remaining questions rather than a claimed fix.
- Existing APIs, dependency pins, cohort/profile definitions,
results/**and generated official README values remain unchanged by the investigation.
Dependencies and boundaries
No hard dependency on #52 or #59. #52 remains deferred. Any later justified and explicitly authorized official run may also satisfy #59, but this investigation must not trigger one solely to close an issue.
No official workflow dispatch, result publication, Pages rollout, cohort activation, recurring measurement, or production tuning is authorized by this issue. Keep investigation notes under ignored docs/investigations/ and Superpowers plans under ignored docs/superpowers/; do not commit or force-add either directory. Durable findings belong in the issue/PR and, when appropriate, a reviewed public guide.
Activity
tappe9 commentedon Sep 18, 2026
Investigation started — 2026-09-18
Fresh main:
9b92c74cba34bd5a4bfc389075b9bef135131034; no open PRs or existing #70 comments were found. Latest main CI and Pages runs remain successful.Original evidence inspected
Original artifact
official-benchmark-34925168324-1(ID10380696186) is available, not expired. Its downloaded ZIP SHA-256 matches GitHub's recorded digest:cc376a99e573d65c5cedc165ee00fda294795f45ed491a5586ef419f96b9989b.The original Flask JSON warm-up and all three measured samples are slow. All have HTTP 200-only status distributions and empty transport-error distributions. In JSON run 2, mean latency is 315.451675 ms and mean time to first byte is 315.450804 ms: the observation is not explained by downloading the 48-byte response body. This does not yet identify the server-side cause. Original artifacts retain memory samples but not per-thread CPU/GIL traces or server queue logs; missing traces must not be invented.
Bounded diagnostic plan, defined before new measurement
Use only the unchanged pinned Flask container with external readiness, 1 CPU / 512 MiB / one Waitress worker / pool limit 10. Reuse the existing verified oha installer, strict raw parser, contract checks, state checks and owned-project cleanup.
Compare three arms on one GitHub-hosted job: (A) unchanged baseline, (B) only the Python thread switch interval changed to 1 ms through a diagnostic-only startup hook, (C) only
waitress.queuewarning output disabled through the same kind of hook. No production source, image/dependency pin or official profile will be edited. B/C are hypothesis probes, not proposed production tuning.Use two blocks with reversed arm order: A/B/C then C/B/A. Recreate the owned Compose environment for every arm; reverse JSON/DB endpoint order in the second block. Each endpoint receives a 5-second warm-up and three 10-second measured windows at 50 connections. Retain every attempt, raw sample, source/image identity, hook/override, bounded server-log sample and available CPU/throttling counters. A failed cell prevents any complete-success diagnostic. No selective retries. A repeatable arm effect supports that mechanism; non-reproduction or mixed effects remain inconclusive rather than proving a fix.
The local container has neither Docker nor GitHub DNS access. Use a temporary read-only, branch-scoped GitHub Actions workflow for the real-container diagnostic; remove it before any merge-ready PR. This is not the official workflow and cannot publish results or deploy Pages. Start with local tests for the diagnostic's output and cleanup boundaries, then verify with the repository and real-container gates. Keep planning/investigation scratch files untracked per
AGENTS.md.No official measurement/publication/cohort activation or #52/#59 change is included.
test: add bounded non-publishing Flask hypothesis diagnostic
tappe9 commentedon Sep 18, 2026
First diagnostic result and next bounded probe — 2026-09-18
Verified evidence
benchmark.report.audit_raw; its selected report matches the current published JSON.cbebd385b0f78503d797fcc73167f1ecce1b8611, tree8ed825dbce85333e3af39e46f07120a44287170b.10529950752,flask-diagnostic-35299848616-1, ZIP SHA-2564df46b3f0843e1486d9fe3382c2962efe952dd414283c2f46ba731f6de0e9bf8matches the downloaded archive. All 36 measured windows were independently checked against raw oha JSON and memory samples. Six distinct containers used the same built imagesha256:b774000f4bc2ed9d13850ec4ac1e14af336f17cd5e15d798662b3c9a74473bd0.Results (selected whole-run requests/s; diagnostic only)
The JSON/DB inversion reproduced. Neither the shorter switch interval nor disabling queue warnings improved JSON in this experiment. Queue logging affects DB throughput, but its removal did not explain/fix the JSON anomaly. This contradicts a simple switch-interval or logging-cost explanation; do not describe either intervention as a production fix.
The cgroup observations provide a more specific lead: baseline JSON spans recorded 350 throttled periods out of 361/362, with about 13.3/13.5 seconds of accumulated throttle time. Baseline DB spans recorded only 16/18 throttled periods out of 358. These spans include warm-up, measurement and observation overhead; throttle time is not a direct per-request latency attribution. Both endpoints consumed roughly one CPU. The current snapshots do not attribute CPU time to individual server threads.
Second diagnostic plan (defined before execution)
Investigate scheduling/CPU-quota interaction, without changing the official profile or production files. On one new job, run baseline / single-CPU-affinity / baseline, recreating containers and PostgreSQL each time. Keep 1 CPU quota, 512 MiB, one Waitress worker, pool10, 50 connections, JSON then DB, 5-second warm-up and 3 x 10-second samples per endpoint: 18 measured windows total. Apply affinity only through a diagnostic startup hook using the first already-allowed CPU; never change host settings or relax the CPU quota. Bracketing baselines and all failed/successful attempts must be retained.
Add outside-load per-thread
/proc/1/task/*CPU counters/names and runtime configuration evidence. Label their collection overhead and distinguish the helper interpreter's metadata from PID1's observed startup hook. The goal is to localize wasted CPU and test the affinity mechanism, not to optimize until a favorable number appears. Results from this second job form their own comparison; do not pool them with the first job as paired measurements.The first 36-window diagnostic remains immutable evidence. This additional bounded probe is not an official benchmark, does not publish or activate a cohort, and does not change #52/#59. Local review also found and fixed a diagnostic CLI safety issue:
--help/unknown arguments must exit before installing tools or starting load (red/green verified). This does not alter the already measured no-argument matrix.test: probe Flask scheduling with bracketing affinity controls
tappe9 commentedon Sep 18, 2026
Scheduling probe verified — 2026-09-18
Resumed from freshly read main
9b92c74cba34bd5a4bfc389075b9bef135131034and investigation brancha153a103f445af8cd571687fb095cc0d2f4cb9a3. The second predefined diagnostic had already completed successfully; no additional measurement or retry was needed in this continuation.Independent evidence replay
a153a103f445af8cd571687fb095cc0d2f4cb9a3, tree677680d7f014b6f4e0d4f0435c8deac83e21f998.10529473210(flask-diagnostic-35300974388-1). ZIP SHA-256 matches GitHub:ae16af92916f62f198ed509d57273831b5b118c8de4c2f054d57037e42123bec.sha256:bfec60549c3ed744e2c6be1065ef7da80d06aa055769656e16f48c2d467ebe60.benchmark.report.audit_raw; the original selected JSON is byte-identical to the published JSON at the audit main.Same-job bracketing comparison — diagnostic only
Affinity JSON throughput is 18.54x / 18.33x the two bracketing baselines. The quota stayed at one CPU, memory at 512 MiB, one Waitress worker, pool10, 50 connections, five-second warm-up and three ten-second windows. PID1's startup marker and per-thread
/procsnapshots confirm CPU0 affinity; baselines allow CPUs0-3. This is CPU placement, not an increased CPU quota or a reserved physical core.JSON observation-span deltas:
Counters span warm-up, measurement and outside-load observation overhead. Scheduler slices are not GIL handoffs, and accumulated throttling is not a per-request latency decomposition. Pool helper threads did not show material CPU growth in these snapshots.
Supported conclusion and limitations
The JSON/DB inversion is reproducible in the pinned stack, and the predefined affinity intervention strongly changes JSON behavior while bracketing baselines remain slow and DB remains similar. This supports a CPU-placement-sensitive scheduling/CPU-quota interaction, not a JSON aggregation error or proof that Flask is intrinsically slow.
Waitress separates main-thread client I/O and worker application execution (upstream design); one CPU of aggregate quota is not equivalent to affinity to one logical CPU (kernel bandwidth documentation). GIL, I/O wakeups, lock contention and kernel scheduling remain plausible lower-level mechanisms. No stack sampling or quota-only controlled experiment isolates a specific function or proves GIL/throttling alone as the cause. There is only one affinity cell on one host; do not generalize the 18x effect or reinterpret the original run as having these later traces.
Recommended next decision: document common CPU-allocation semantics before any production/methodology change. Keeping the existing quota-only baseline with an explanation is valid. A common affinity policy would need separate review across the cohort; do not selectively boost Flask or overwrite history.
Preparing one focused PR with the reusable diagnostic, regression tests and a durable public interpretation guide. The temporary branch workflow will be removed. A small additional fail-closed image-drift guard is being tested; the retained experiments already used matching images. No main change, official run/publication, cohort activation, or #52/#59 change has occurred.
test: finalize Flask scheduling investigation and preserve evidence
tappe9 commentedon Sep 18, 2026
Investigation PR ready for review — 2026-09-18
The implementation and evidence review are now in PR #76, with draft status removed after successful final verification. The PR is open, not merged; this issue intentionally remains open for acceptance.
1ee3a87cda97a280e54745e425579ca5ee3222b8.e0298668618fa31124a823502cc1886d8c8264be, identical to the local reviewed tree and GitHub-tested merge commite846e6b97447dce75abb6abc8dbc7049d134dad9.required.-O, including all 15 final Flask diagnostic tests; 38 contract tests, 21 workflow tests, registry/README/Ruff, real DB acceptance and 40 Node/browser tests also passed.The final four-file change retains the reusable diagnostic, its regression tests, a public evidence/interpretation guide and a link from the Flask guide. The temporary investigation workflow is removed. No application, dependency pin, Compose resource setting, official cohort/profile, publication workflow, README result or
results/**changed. Ignored investigation and Superpowers notes were not committed.The prior verified outcome remains the conclusion: the JSON/DB inversion reproduced, and a same-job baseline / one-CPU-affinity / baseline experiment showed roughly 102 / 1,883 / 103 JSON requests/s while DB stayed near 1,200. This supports CPU-placement-sensitive scheduling/CPU-quota behavior, not proof that the GIL alone, a particular Waitress defect or throttling alone is the root cause. The original 72 measured windows and both diagnostic archives (54 measured windows plus 18 warm-ups) were audited without replacing their original provenance.
A later common CPU-allocation/methodology decision is separate from accepting this investigation. Keeping the quota-only baseline with an explanation is valid; do not apply a Flask-only boost or reinterpret the diagnostic contrast as an official speedup. No new official measurement or publication is authorized by this result. #52 and #59 remain unchanged.
Review is self-review, not independent approval. Local complete-suite timeouts and unavailable local Docker remain disclosed in the PR; the complete executed verification above is GitHub CI. The retained performance experiments used their original measured commits, not the final formatting/documentation/image-guard commit. No additional Flask experiment, selective retry, merge, official publication or settings change was performed in this continuation.
tappe9 commentedon Sep 26, 2026
Simpler alternatives screening — predeclared plan (2026-09-26)
The maintainer now prioritizes a simple, maintainable solution and explicitly wants several alternatives tested; retaining Flask is not a requirement. Hold PR #76 unmerged while evaluating alternatives. Its existing diagnostic evidence remains valid, but retaining its entire diagnostic implementation is not a prerequisite for a future fix.
Fresh main:
9b92c74cba34bd5a4bfc389075b9bef135131034; PR #76 head:1ee3a87cda97a280e54745e425579ca5ee3222b8. This is a disposable, non-publishing screening experiment, not a new official comparison or authorization to replace an implementation.Fixed matrix before collection
On one GitHub-hosted Linux job, compare (A) unchanged Flask/Waitress, (B) the same application with only
asyncore_use_poll=False, (C) Flask/Gunicorn 26.0.0, explicitlygthread, one request worker and one request thread, (D) Flask/Gunicorn 26.0.0sync, one request worker, (E) a minimal Starlette/Uvicorn alternative using the existing pinned FastAPI image's Starlette/Uvicorn/asyncpg dependencies, and (F) unchanged FastAPI/Uvicorn as a same-host control.Run A/B/C/D/E/F, then F/E/D/C/B/A. Recreate the owned PostgreSQL/API environment for each cell. Block 1 measures JSON then DB; block 2 reverses endpoint order. Each endpoint gets five seconds of warm-up and three five-second measured windows, 50 connections, HTTP/1.1, 15-second request timeout, one CPU quota and 512 MiB. This is 72 measured windows and 24 warm-ups; shorter windows are screening evidence only. The CPU endpoint is checked by the shared correctness contract but not performance-measured in this first screen.
Do not change CPU affinity, enlarge quotas or add request workers/threads. Gunicorn has an arbiter plus one request worker (two OS processes); record that explicitly rather than weaken the production one-process check. Verify connection behavior: sync intentionally closes connections, so its numbers belong to a changed transport model, not a like-for-like keep-alive claim. Framework replacement changes the stack and is not an isolated attribution experiment.
Reuse existing strict oha parsing, complete shared contract checks, container-state validation and project-scoped cleanup. Pin the added Gunicorn/packaging wheels and verify their published SHA-256 hashes. Retain all attempts, failures, raw windows, image identities, actual installed versions, startup/process evidence, exact source/tree and workflow run/attempt. No selective retries. A failed cell must not produce a successful candidate result; continue the remaining predeclared cells only when scoped cleanup succeeds.
Screen for a reproduced baseline anomaly, consistent behavior in both order blocks, no HTTP/transport errors, and no hidden concurrency/transport change. Do not choose by the single fastest sample. Also compare permanent code/dependency/test maintenance requirements. A production choice still needs focused implementation and full acceptance/performance confirmation.
Boundaries
Use an isolated experiment branch and temporary read-only workflow. Remove its executable workflow after evidence collection; keep findings in this discussion. No main or PR #76 source changes, merge, official workflow invocation, results/history/README publication, cohort/profile activation, release/tag, settings or credential changes. No committed scratch plans or investigation notes.
tappe9 commentedon Sep 26, 2026
Simpler alternatives: completed screening and evidence audit — 2026-09-26
The predeclared six-arm screen completed successfully in run 36219805501, attempt 1, job
108342785839. A simpler server replacement avoided the extreme JSON slowdown without changing Flask handlers or CPU affinity. This is screening evidence, not a production fix, an official performance update, or a universal framework ranking.Original measured source and artifact
9b92c74cba34bd5a4bfc389075b9bef135131034.717011ac41ed7b390f675fe8061689fd9c91e862; tree:2c19b80accc0987463ca9d5238e98d504d2a41b9.flask-alternatives-36219805501-1, 253,998 bytes. Downloaded ZIP SHA-256 matches GitHub metadata:198ef17285fdb0b89b150c662e178a1767757159639ad7221f025f68054c66c0.probe.pybytes match the locally reviewed source (SHA-256551491cf1bc1545fbba27a67f0d0ddca3ffd55c6a6f147eabfbaab756adbfc0b).20260920.314.1, AMD EPYC 9V74, four visible logical CPUs. It is not the original publication's host/profile, so do not compare these absolute numbers to September 15 as a measured speedup.Results — selected whole-run requests/s, diagnostic only
Every cell used a fresh owned PostgreSQL/API environment, one CPU quota, 512 MiB, pool maximum 10, HTTP/1.1, 50 connections, five-second warm-up and three five-second measured windows per endpoint. Block 2 reversed both candidate and endpoint order. Each table entry is the middle-throughput whole run of its three samples, not a best sample or a pooled estimate.
asyncore_use_poll=Falsegthread, one request worker / one request threadsync, one request worker, no keep-aliveValidation actually executed
All 12 cells, 72 measured windows and 24 warm-ups completed on the one predeclared attempt. All 12 fresh container IDs are distinct. Each cell passed all 14 shared HTTP checks (168 checks total, including CPU correctness), the actual connection-reuse probe, container state/identity checks, stable process checks and scoped cleanup. There were no failed cells or selectively retried measurements.
The existing strict
parse_oha/validate_run/ whole-run selection path was used during execution. A separate local archive audit reconciled every raw window's successful-request counts, elapsed time, throughput, mean latency, response bytes, histogram totals, memory samples, selected whole run, image/process identities and declared limits. All 96 raw outputs contain only HTTP 200 responses and no transport-error entries. No output is accepted as an official report.Six process-isolation safety tests passed locally and on the runner; the helper had a recorded local red/green cycle. Workflow source and the embedded probe were checked for byte equality and branch/read-only/non-publication boundaries. These are not claims that a new production implementation passed the full repository CI, shutdown, database-failure or long-duration acceptance gates.
Interpretation and maintenance trade-offs
gthreadpreserved actual keep-alive, the Flask handlers/DB adapter, one request-processing worker/thread, and the aggregate resource limits. JSON was about 10.26x / 10.59x its same-block baseline, while DB was about 1,714–1,715 requests/s. This supports Flask + Gunicorn gthread as the smallest application-change candidate to validate next, not the need for a CPU-affinity policy change. It does not isolate a particular Waitress/GIL/kernel defect.syncavoided the JSON drop but closed connections, as both socket probes confirmed. It is not a like-for-like keep-alive replacement; DB throughput was below its same-block baseline. Prefergthreadoversyncfor a focused Flask-preserving follow-up.Both Gunicorn-gthread and Starlette merit consideration. For minimum disruption and retaining a distinct synchronous Flask representative, recommend validating Gunicorn-gthread first. For prioritizing a uniform one-process ASGI setup over framework diversity, Starlette is a supported replacement candidate. Do not select merely the fastest sample. Keeping quota-only measurements with an explanation remains possible; no CPU-affinity change is required to explore these alternatives.
Cleanup and scope
The temporary workflow was removed in commit
7adc1aff545fdf02ae00e89b31d677a2c5706e8a. A fresh compare shows zero file differences between the experiment branch and baseline main; the original experiment commit/run/artifact remain traceable. No permanent diagnostic subsystem was added.Fresh main is still
9b92c74cba34bd5a4bfc389075b9bef135131034. PR #76 is still open/unmerged at1ee3a87cda97a280e54745e425579ca5ee3222b8; it was not edited or merged. Its prior diagnostic findings remain historical evidence, but retaining all its tooling is not a prerequisite for the smaller solution. Consider preserving only useful findings rather than merging it solely to proceed.No official measurement/publication, cohort/profile change, production app/pin change, result/history/README rewrite, release/tag or settings/credential mutation was performed. A future framework replacement must use an appropriate new implementation/cohort identity and preserve old Flask history. A selected production change still needs TDD, full acceptance/CI and a bounded longer non-publishing confirmation across JSON/DB/CPU before any separately authorized official run.
tappe9 commentedon Sep 27, 2026
Implementation status reconciliation — 2026-09-28
This issue remains closed/completed. The investigation and bounded alternatives evidence are retained in the earlier comments. The adopted minimal production follow-up is merged PR #77 (
fix: run Flask with Gunicorn gthread), merge commiteba127e65761ced4d0fef06a6c52cc2dc1fdb6ab, which is the parent of current mainc1d8e81b959e2c0ee0e5c4a1d8a7491790479f93.The runtime now uses Gunicorn 26.0.0 with one gthread request worker, one request thread and keep-alive; its arbiter is a separate OS process. The retained non-publishing evidence and normal acceptance/CI must not be presented as a new official performance measurement.
The maintainer has now separately authorized an official run and audited publication after this runtime change. That operation and its exact producer-to-Pages verification are tracked in #59. No new official dispatch or publication was performed by the current session because its tools do not expose new workflow dispatch. The existing September 15 Waitress report remains historical measurement evidence, not a Gunicorn result.
There is no reason to reopen this completed implementation issue merely because the separately authorized official-publication operation is still pending.