Skip to content

backlog: amend #1112 -- the concurrent-uploads half is answered, the cross-shard half is what remains - #353

Merged
wshallwshall merged 5 commits into
mainfrom
claude/amend-1112-shard-quota
Aug 12, 2026
Merged

backlog: amend #1112 -- the concurrent-uploads half is answered, the cross-shard half is what remains#353
wshallwshall merged 5 commits into
mainfrom
claude/amend-1112-shard-quota

Conversation

@wshallwshall

Copy link
Copy Markdown
Collaborator

Docs-only. #1112 stays OPEN -- this is an amendment, not a closure. No new number was allocated, deliberately.

Why no new number

The owner asked for the store-side ASVS 2.3.4 fix to be filed. The ASVS lane ran the duplicate check before drafting and it fired: #1112's own title already reads "a quota that holds across concurrent uploads and across engine shards at once." The store-side fix is its second half.

Filing it fresh would have burned a number on a duplicate -- the failure section 7 records twice, and which I made myself today on the glyph item. Running the check first is the whole reason it did not happen again.

Verified against origin/main, not taken on report

uploads.py:309   self._quota_lock = asyncio.Lock()                      (PR #325)
uploads.py:488   async with self._quota_lock:   ... :496 _scan_metas_sync()
"the check-then-write is not atomic"   0 occurrences  -- and there is no line 394
settings.py:439  "engine shards sharing one dir see each other's files and the budget
                  does NOT multiply (measured 2026-08-10 ... live positive control)"

Two strikes, both false in the ALARMING direction

  1. The banner's "no lock constructed in the module" and its :394 citation. The lock exists and is held across both the scan and the write -- releasing between them was the race.
  2. The severity paragraph's "N engine shards would each apply the budget separately, multiplying it." The real remainder is an at-most N-1 file overshoot, one per competing shard caught between its scan and its file landing, each bounded by max_upload_bytes.

Both struck rather than deleted. An overstated severity is not the conservative error -- it misdirects whoever costs the fix, and this one overstated by a multiplier.

What actually keeps it open -- one sentence

asyncio.Lock is per-event-loop, therefore per-process, and the module says so itself at :487: "One critical section per process." Engine sharding is the built, shipped, default scaling axis, and nothing partitions uploads_dir per shard -- so N shards over one directory hold N independent locks.

The fix has a known home, so "cannot honestly reach pass" is not the finding here: ADR 0063 already puts every shard on one unified store, so the cross-shard coordination point exists. Move the quota accounting into the store, or take an advisory lock on the directory.

Acceptance is stated as EXECUTION, never as reading: the quota holds across N shards sharing one uploads_dir, with concurrent writers on at least 2 shards, and the single-process path keeps its current behaviour.

Owner ruling, recorded in the item: ASVS 2.3.4 holds at partial -- V2.3.4 names business-logic locking as the means, and a per-process lock does not span a resource shared across processes.

One blind check I caught in myself

My first grep for the struck string ran against messagefoundry/api/uploads.py -- which does not exist; the file is messagefoundry/uploads.py. It printed 0 and I labelled it "the string is gone." Right answer, no evidence. Re-ran against the real path with a negative control (49 def/class lines) before believing it.

…cross-shard half is what remains

NO NEW NUMBER. The ASVS lane ran the duplicate check BEFORE drafting and it fired:
#1112's own title already names both halves ("concurrent uploads AND across engine
shards"), so the store-side fix is its second half, not a new item. Filing it fresh
would have burned a number on a duplicate -- the failure section 7 records twice, and
which I made myself today on the glyph item.

VERIFIED AGAINST origin/main, not taken on report:
  uploads.py:309  _quota_lock = asyncio.Lock()          (PR #325)
  uploads.py:488  async with self._quota_lock:  ... :496 _scan_metas_sync()
  "the check-then-write is not atomic"  0 occurrences -- and there is no line 394
  settings.py:439 states shards over one dir enforce ONE budget, measured with a
                  live positive control

TWO STRIKES, both false in the ALARMING direction, kept rather than deleted:
- the banner's "no lock constructed in the module" and its :394 citation;
- the severity paragraph's "N shards would each apply the budget, multiplying it".
  The real remainder is an at-most N-1 file overshoot, each bounded by
  max_upload_bytes. An overstated severity is not conservative -- it misdirects
  whoever costs the fix.

WHAT KEEPS IT OPEN is one sentence: asyncio.Lock is per-process, and the module says
so itself at :487. Engine sharding is the shipped default scaling axis, nothing
partitions uploads_dir per shard, so N shards hold N independent locks.

The fix has a known home -- ADR 0063 already puts every shard on one unified store --
so "cannot honestly reach pass" is NOT the finding. Acceptance is stated as EXECUTION
with concurrent writers on at least 2 shards, never by reading.

OWNER RULING recorded in the item: ASVS 2.3.4 holds at partial.

CAUGHT MY OWN BLIND CHECK doing this: my first grep for the struck string ran against
messagefoundry/api/uploads.py, which does not exist -- the file is messagefoundry/uploads.py.
It printed 0 and I labelled it "the string is gone". Re-ran against the real path with a
negative control (49 def/class lines) before believing it.
@wshallwshall
wshallwshall enabled auto-merge (squash) August 12, 2026 18:32
test_link_resolution caught it: docs/BACKLOG.md:6000 wrote
[ADR 0063](docs/adr/...) which resolves RELATIVE TO docs/, so it became
docs/docs/adr/... and did not exist.

The convention is unambiguous and I was the outlier: 74 links in this file use the
adr/ prefix, exactly one used docs/adr/, and it was mine.

Verified after fixing: zero docs/adr/ links remain, and the target file exists.

I could NOT run the guard locally -- pytest dies on ModuleNotFoundError: pydantic,
because a worktree venv installs fewer extras than CI. That is BACKLOG #1230
demonstrating itself on the PR that amends the ledger, an hour after its scope was
ruled. CI is the authority here, which is the whole point of #1230's loud-omission
half: a local run that cannot collect must not read as a clean one.
@wshallwshall
wshallwshall merged commit 47cf57e into main Aug 12, 2026
33 of 34 checks passed
@wshallwshall
wshallwshall deleted the claude/amend-1112-shard-quota branch August 12, 2026 18:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant