Optimize usage rollups with secondary indexes to remove full-store scans

[mikewheeleer]

Refactor usage rollups to use secondary indexes instead of O(n) scans

Description

Many read endpoints in src/index.ts iterate the entire usageStore on every request: GET /api/v1/agents, /agents/:agent/total, /agents/:agent/usage, /services/:serviceId/usage, /services/:serviceId/agents, /services/:serviceId/agents/top, /billing/total, /stats, and /metrics all do for (const ... of usageStore.entries()) with startsWith/endsWith/split per key. As the store grows these become O(n) per call. This issue adds maintained indexes so common rollups are near-constant time.

Requirements and context

• Repository scope: Agentpay-Org/Agentpay-backend only.
• Maintain a byAgent and byService index (e.g. Map<string, Set<string>>) updated whenever usageStore is written in POST /usage, POST /usage/bulk, and reset in POST /settle.
• Rewrite the listed rollup endpoints to consult the indexes instead of scanning the whole map; keep response shapes identical.
• Keep a running protocol-wide total to back /billing/total, /stats, and /metrics without a full scan.
• Add a consistency check (test-only) asserting indexes match a brute-force scan.

Suggested execution

• Fork the repo and create a branch
• git checkout -b refactor/performance-29-usage-indexes
• Implement changes
  • Write code in: the store-write paths and rollup handlers in src/index.ts (or src/store/ if modularized).
  • Write comprehensive tests in: new src/usage-index.test.ts — index/scan parity after add, bulk, settle, and overwrite.
  • Add documentation: note the indexing strategy in docs/architecture.md if present, else in code comments.
  • Add TSDoc on the index-maintenance helpers.
  • Validate security assumptions: indexes cannot drift to expose stale cross-agent data.
• Test and commit

Test and commit

• Run npm run build, npm test, and npm run lint.
• Cover edge cases: settle resets to 0 but keeps the key, agent across many services, top-N tie-breaking.
• Include the full npm test output and a brief before/after complexity note in the PR description.

Example commit message

perf: back usage rollups with secondary indexes to avoid full scans

Guidelines

• Minimum 95 percent test coverage for impacted modules.
• Clear, reviewer-focused documentation.
• Timeframe: 96 hours.

Community & contribution rewards

• 💬 Join the AgentPay community on Discord for questions, reviews, and faster merges: https://discord.gg/eXvRKkgcv
• ⭐ This is a GrantFox OSS / Official Campaign task and may be rewarded. When your PR is merged you'll be prompted to rate the project — if this issue and the maintainers helped you ship, we'd be grateful for a 5-star rating. Clear questions in Discord and tidy, well-tested PRs are the fastest path to a merge and a reward.