[roychying]

Why?

Operators need queue depth: how many requests are in each state, and how long the oldest one has been there. Event metrics cannot provide this. It is a property of stored state, and it matters most when a queue is stuck and no events are flowing.

The request table is keyed by (queue, id), so "requests of queue Q in state S" is not a key-based read. Adding a (queue, state) secondary index would make RequestStore depend on a query-by-attribute, which the storage contract rules out. The sanctioned shape is a first-class mapping store keyed by the lookup attribute, the same pattern as QueueBatchState and ChangeRecord.

This PR adds that mapping for requests. A periodic job can then report per-state request counts and the oldest time-in-state with key-prefix scans only.

What?

• Entity
  • entity.QueueRequestState{Queue, State, RequestID, EnteredAtMs}: one record per in-queue request, under its current state.
  • entity.InQueueRequestStates: every non-terminal request state. A test enforces that each valid state is either in this list or terminal.
• Contract (read-only)
  • storage.QueueRequestStateStore.List(ctx, state), with its mock.
  • GetQueueRequestStateStore() on the orchestrator storage aggregate.
• MySQL
  • New table queue_request_state, PK (queue, state, request_id). It leads with queue, as the queue-shard linter requires, and has no secondary index.
  • List is a primary-key-prefix scan.
  • The request store writes the record in the same local transaction as the request row. On Create and Update it deletes the request's record from every other in-queue state and INSERT IGNOREs it under the new one, so a version-only update keeps entered_at_ms. A terminal state leaves no record, so the table only ever holds in-queue requests. A failed CAS or duplicate create rolls back with no record change.
• Rollout gate
  • mysql.NewStorage(db, scope, WithQueueRequestStateRecords()) enables the records. It is off by default, so a deployment that upgrades before creating the table keeps working.
  • With the records off, List returns an error instead of reporting every queue as idle.
  • The reference orchestrator server enables the records.
• Docs
  • Storage README (key-value contract section), schema README, the store count in AGENTS.md, and the store/table map in doc/rfc/service-scoped-extensions.md.

Exact records maintained by the store, rather than advisory records maintained in core.** QueueBatchState records are moved by submitqueue/orchestrator/core/batch around the batch CAS, are advisory, and need repair on redelivery. Here the backend keeps the record in the same transaction as the request write. That way:

• The records are exact, so readers can count them without hydrating each request.
• The contract is read-only, with no transition protocol or repair path.
• Every current and future request write path is covered without touching controllers.

The cost is that a backend must be able to update the request and its record atomically. A key-value backend without local transactions would need its own write ordering.

Test Plan

• Unit tests:
  • TestRequestStore_WritesQueueRequestStateRecords covers:
    • create files started in the same transaction
    • a duplicate create rolls back
    • a state change moves the record
    • a terminal update removes the record and files nothing
    • a version mismatch rolls back
    • a failed record write rolls back the request write
  • TestQueueRequestStateStore_List: rows, empty, query error, row error.
  • TestDisabledQueueRequestStateStore_ListFails.
  • TestNewStorage_QueueRequestStateRecords: off by default, on with the option.
  • TestInQueueRequestStates: every valid state is either in-queue or terminal.
• Contract test (integration, real MySQL): TestStorage_QueueRequestStateRecordsFollowRequestWrites covers:
  • records follow create and state changes
  • a version-only update keeps EnteredAtMs
  • a failed CAS changes nothing
  • a terminal request has no record
  • queues stay isolated

Run locally:

go test ./submitqueue/entity/ ./submitqueue/orchestrator/extension/storage/...
make integration-test    # requires Docker; includes the contract suite
make lint check-gazelle check-tidy check-mocks

Issue

[behinddwalls]

why is this in orchestrator? why do we need it? shouldn't this be in gateway later?

[roychying]

we need it for the queue-depth panels on dashboard, like how many requests are in each state per queue right now, and how long the oldest has been waiting. Adding a new index will break the key-value storage contract so I prefer to add a new table.

gw is also fine, we can leverage the RequestLog to maintain the table, will do it in another PR