Skip to content

fix(portal): enforce team scheduling limits on retry and answer - #6659

Merged
Lightheartdevs merged 1 commit into
Osmantic:mainfrom
tang-vu:fix/portal-team-resume-admission
Sep 28, 2026
Merged

Lightheartdevs merged 1 commit into
Osmantic:mainfrom
tang-vu:fix/portal-team-resume-admission

Conversation

@tang-vu

@tang-vu tang-vu commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Apply the existing team admission rules to every entry point that schedules work: start, retry, and answering a pending question. Check before changing persisted state.

Why this matters

Portal exposes Retry on historical failed workers and Continue on pending questions. Those controls call /api/pixel/agents/retry and /api/pixel/agents/answer respectively.

Two reachable paths bypass the constraints enforced by Start:

  1. An explorer fails and its builder/reviewer are skipped. Start a newer team in the same conversation, then retry the old explorer. The old code returns HTTP 200 and queues the old builder/reviewer too. The global execution lane serializes individual workers, but both teams can now take turns working in the shared workspace despite the one-active-team-per-conversation rule.
  2. A team waits for an answer, releasing its scheduler task. Four other conversations fill the queue. Answer the waiting team: the old code accepts a fifth scheduled team, clears its pending questions, and increments the saved turn.

The shared admission guard rejects either action with HTTP 409 before consuming a retry, appending an answer, clearing questions, or requeuing workers. It also rejects a duplicate scheduled controller. Waiting/interrupted teams continue to reserve their conversation, as they already do for Start.

Recovery remains an explicit user action: finish or stop the conflicting team, or wait for queue capacity, then retry/submit the preserved answer again. Start request-id replay remains idempotent even when the queue is full. No storage migration or automatic replay is introduced.

Overlap check

Searched open and closed upstream PRs for pixel_agent_teams.py, pixel_teams, team retry, team queue, and team admission. Inspected the relevant existing changes: #5689 handles corrupted team-store rows; #5945 handles missing conversation history; #6656 concerns tool-limit completion/owner replies. Other results cover protocol field validation and history recovery. None applies scheduling admission to Retry and Answer. This PR changes only the team manager and its existing regression test file.

Regression evidence and validation

  • Base: upstream main at 250f35450a7c66f1af9cd1933e1387242bb81035.
  • Authenticated ASGI requests use the real FastAPI router, TeamManager, on-disk TeamStore, and asynchronous scheduler, with controlled model event streams.
  • Before the fix, both original regression scenarios failed: expected HTTP 409, received HTTP 200.
  • After the fix, five cases pass: retry conflicts with running/waiting/interrupted teams, and full-queue answers in both team and goal modes. Each case verifies unchanged persisted state on rejection and successful explicit recovery afterward.
  • Focused backend suite: python -m pytest tests/test_pixel_agent_teams.py tests/test_portal_goal.py tests/test_pixel_chat_context.py -q --tb=short --disable-warnings — 73 passed.
  • Candidate commit: c02ad81ce5c1cb781159048e4fd18d15091e5edd.
  • Dashboard API CI: 4,481 passed, 1 skipped on Python 3.11.
  • Dashboard lint, tests, and build passed on Ubuntu, macOS, and Windows. Windows frontend CI: 1,677 passed across 201 files, including all eight PortalSubagents cases.
  • Overall CI: 42 successful checks, four skipped, no failed or pending checks, including integration-smoke.
  • Ruff on both changed files and git diff --check: pass.

Validation limits and review

Local backend environment: Windows host, WSL Ubuntu x86_64, Python 3.14. Model output is controlled in the regression harness; this is not a claim of live model/provider validation. No fresh installation or full browser/model journey was completed on this candidate. An earlier isolated Debian 13 installation attempt on main ee738a9a07454b589555de00bb3e6bb91f215843 stopped at preflight because jq was unavailable and the test user lacked package-install privileges; that is not installation evidence for this newer base.

The local Windows frontend test attempt failed to start Vitest forks workers (timeout before any tests ran). Retrying with one threads worker passed the three project-helper tests, but the PortalSubagents worker also timed out before running its tests. All frontend CI platforms subsequently passed, including PortalSubagents. No frontend code changes are included. Existing UI code retains answer drafts and displays controller errors.

The guard is process-local, matching the existing scheduler and Start semantics; it does not add cross-process coordination. Reverting restores the old admission behavior without a data migration. Maintainer review should check the admission/recovery policy before merge.

@Lightheartdevs
Lightheartdevs merged commit 76f41e9 into Osmantic:main Sep 28, 2026
46 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants