diff --git a/adr/4173-issue-tracker-seam.md b/adr/4173-issue-tracker-seam.md index cd3424ff98..e345ea5a27 100644 --- a/adr/4173-issue-tracker-seam.md +++ b/adr/4173-issue-tracker-seam.md @@ -1,15 +1,15 @@ # ADR-4173: The issue tracker seam -**Status**: Accepted (PRD #4173, M3 landed; M4–M6 pending) -**Date**: 2026-11-11 +**Status**: Accepted (PRD #4173, M4 landed; M5–M6 pending) +**Date**: 2026-11-11 (updated 2026-11-13) **Deciders**: the maintainer + agent team -**PRD**: [prds/4173-external-issue-trackers.md](../prds/4173-external-issue-trackers.md) — the PRD carries the milestones and the Decision Log; this ADR carries the durable shape of what M3 built and the invariants a later change must keep. +**PRD**: [prds/4173-external-issue-trackers.md](../prds/4173-external-issue-trackers.md) — the PRD carries the milestones and the Decision Log; this ADR carries the durable shape of what M3 and M4 built and the invariants a later change must keep. ## Decision (summary) -Issue trackers other than the repo's forge sit behind a `Driver` in their own package, `api/internal/issuetracker`. A driver is `forge.IssueTracker` plus four connection operations. Connections live apart from forge connections (`issuetrackersvc`, table `issue_tracker_connections`) with their own base URL allowlist. The YouTrack driver adapts YouTrack's string ids inside itself, is built for exactly one project, and treats tags as user-owned objects chosen by id. Through M3 no repo can link to an issue tracker, so no behaviour changes for forges. +Issue trackers other than the repo's forge sit behind a `Driver` in their own package, `api/internal/issuetracker`. A driver is `forge.IssueTracker` plus four connection operations. Connections live apart from forge connections (`issuetrackersvc`, table `issue_tracker_connections`) with their own base URL allowlist. The YouTrack driver adapts YouTrack's string ids inside itself, is built for exactly one project, and treats tags as user-owned objects chosen by id. From M4 a repo created after the M4 migration can link to one before its first enable (D13); a repo with no link behaves as before. -Not shipped yet: the repo link, `IssueTrackerForRepo` returning an issue tracker (M4), the board and run behaviour on YouTrack (M5), and the live pass against a real YouTrack with docs (M6). +Shipped in M4: the repo link and `IssueTrackerForRepo` returning the linked issue tracker, with intake and runs from YouTrack issues. Not shipped yet: issue writes and board moves on YouTrack (M5), the web and CLI settings surfaces, docs and the live pass against a real YouTrack (M6). On a linked repo every issue write is refused with `ErrIssueWriteNotOffered`. ## Context @@ -40,31 +40,32 @@ Not shipped yet: the repo link, `IssueTrackerForRepo` returning an issue tracker - `issueIID` is YouTrack's `numberInProject`. Each per-issue call checks the issue reached is in the bound project and carries the number, so a moved issue reads as not found. YouTrack has no conditional write, so a move between that check and the write is not caught. - `IssueNote.ID` is 0, not derived from YouTrack's string comment id. Account ids are 0, with the login kept as text. - `ListStates` takes the project it reads, because states belong to a project. -- The nullable `issue_key` column is M4's. +- The readable id is `issues.issue_key` (`-`), nullable with a CHECK, written only by the linked sync and NULL for forge issues. ### D14. The tag rules - The eligibility tag is chosen by id. Its name resolves to the stored id whoever owns the tag, for reads and for writes. - Any other name resolves only to a tag owned by the connection's account, which the driver learns from `users/me`. A missing one is created, owned by that account, with the eligibility tag's sharing settings. The eligibility tag is never created, and a drifted name never resolves to the eligibility id. - Reads drop a tag that only shares the eligibility name. -- `ListTags` returns the tags the token can see, with owner and read/use sharing settings. `AccountCanApply` and `OthersCanApply` judge the use settings, and the owner counts as covered. A named group counts for others only with `UsersCount >= 2`; the anonymous response recorded from a real YouTrack (`testdata/youtrack_tags_recorded.json`) omits `usersCount`, so on such a response only the All Users group counts. This fails closed. Using these to refuse a link is M4. +- `ListTags` returns the tags the token can see, with owner and read/use sharing settings. `AccountCanApply` and `OthersCanApply` judge the use settings, and the owner counts as covered. A named group counts for others only with `UsersCount >= 2`; the anonymous response recorded from a real YouTrack (`testdata/youtrack_tags_recorded.json`) omits `usersCount`, so on such a response only the All Users group counts. This fails closed. The link step uses these to refuse a link (M4). +- Intake lists the eligibility tag's issues by id (`GET /api/tags/{id}/issues`), filtered on the client by the stored tag id, and `ListLabels` returns only the tags the driver can use: the stored eligibility name plus tags the connection's account owns. ## Invariants a future change must keep - **S1. An issue tracker never gains code access.** `Driver` embeds `forge.IssueTracker`, not `forge.CodeHost`; a driver passed as a code host does not compile. -- **S2. Code-host checks stay on the forge.** The privilege check, the protected-branch check and the worker-held forge credential act on the repo's forge connection and fail closed. Nothing in M3 touches them, and no repo is linked yet to test the claim further. +- **S2. Code-host checks stay on the forge.** The privilege check, the protected-branch check and the worker-held forge credential act on the repo's forge connection and fail closed. M4 tests that a linked repo keeps them on the forge and that they fail closed with the issue tracker unreachable. - **S3. Credentials follow the forge rules.** Sealed token with AAD and no reveal path, https-only base URL on the separate empty-by-default allowlist re-checked in `DriverFor`, redirects refused, errors redacted. The redirect, oversized-body and token-in-error cases each have a test in `api/internal/issuetracker`. -- **S4. Issue tracker text is untrusted input.** The driver bounds what it reads (response bytes, listing pages and items, nested items, label name length). Fencing of titles, descriptions and comments is done by the existing callers once a repo can link in M4; it is not exercised on YouTrack text yet. -- **S5. The worker never holds an issue tracker token.** The allowlist does not feed worker egress, and the token is opened only inside `DriverFor` in the API. The worker read routes are M4. +- **S4. Issue tracker text is untrusted input.** The driver bounds what it reads (response bytes, listing pages and items, nested items, label name length). YouTrack titles, descriptions and comments go through the same `issueinput` fencing as forge text; an oversized description is refused and an oversized comment by the connection's own account is cut to the same bound as on a forge, while an unmapped author's comment is withheld; M4 tests each. +- **S5. The worker never holds an issue tracker token.** The allowlist does not feed worker egress, and the token is opened only inside `DriverFor` in the API. The worker read routes go through the API's resolver, so the worker never holds the token. ## Open questions for M6's live pass - Whether YouTrack accepts `readSharingSettings` and `tagSharingSettings` in the tag-create body. The fake accepts them; no real server has been asked. - Whether an authenticated token receives `usersCount` on a tag's sharing groups (the anonymous recording has none). -- Whether the server query `tag: {name}` resolves every tag of that name, including ones owned by others. The client side filters by id, so a wrong answer here would drop issues, not mis-tag them. +- Whether the id-scoped listing `GET /api/tags/{id}/issues` behaves on a real instance as the fake and YouTrack's documented shape say. The earlier question, whether `tag: {name}` resolves every tag of that name, is settled: intake sends no tag name in a search query (PRD Decision Log 2026-11-12). If the listing proves unusable, the documented fallback walks the whole project and filters by stored id, bounded by the same listing ceilings. ## Consequences -- Forge drivers, the neutral types and every caller are unchanged. A second tracker (Asana) is the real test of the seam: YouTrack specifics (the id adapter, the tag rules) must stay inside its driver. +- Forge drivers and the neutral types are unchanged; callers resolve their issue side through the owner-scoped resolvers, and on a repo with no link they behave as before. A second issue tracker (Asana) is the real test of the seam: YouTrack specifics (the id adapter, the tag rules) must stay inside its driver. - Operators get one new empty setting. Until it is set no issue tracker connection can be created, and uzi boots as before. -- The `projectID` guard is one half of a binding. M4 owns the other half, the resolver. +- The `projectID` guard is one half of a binding. M4 added the other half, the resolver (`IssueTrackerForRepo`, owner-scoped). diff --git a/agent/src/linked-issue.ts b/agent/src/linked-issue.ts index b44ef7b842..bc7ceafcfd 100644 --- a/agent/src/linked-issue.ts +++ b/agent/src/linked-issue.ts @@ -16,7 +16,7 @@ const ISSUE_KEY_RE = /^[A-Za-z][A-Za-z0-9_]*-[1-9][0-9]*$/u; const ISSUE_KEY_MAX = 64; /** What a linked run's text says when its key is absent or malformed. */ -export const LINKED_ISSUE_FALLBACK_LABEL = "the linked issue"; +const LINKED_ISSUE_FALLBACK_LABEL = "the linked issue"; export function linkedIssueKey(claim: { issue_key?: unknown }): LinkedIssue { const raw = claim.issue_key; diff --git a/api/cmd/server/main.go b/api/cmd/server/main.go index 76d89c8ee4..ffa382e169 100644 --- a/api/cmd/server/main.go +++ b/api/cmd/server/main.go @@ -1007,7 +1007,7 @@ func run() error { h := handler.New(pool, q, cfg, box, svc, wsvc, pcheck, liveHub, settingsCache) // External issue tracker connections and the repo link (PRD #4173 M4). The link and // the reconnect rule run in transactions on the pool. An empty allowlist - // (UZI_ISSUE_TRACKER_ALLOWED_BASE_URLS) still boots: Connect then refuses every URL. + // (ISSUE_TRACKER_ALLOWED_BASE_URLS) still boots: Connect then refuses every URL. itSvc := issuetrackersvc.New(q, box, cfg.IssueTrackerBaseURLAllowed, issuetracker.New). UseTransactions(pool, func(tx pgx.Tx) issuetrackersvc.Querier { return q.WithTx(tx) }) h.SetIssueTrackerService(itSvc) diff --git a/api/internal/handler/linked_repo_d11_livedb_test.go b/api/internal/handler/linked_repo_d11_livedb_test.go new file mode 100644 index 0000000000..7c31232256 --- /dev/null +++ b/api/internal/handler/linked_repo_d11_livedb_test.go @@ -0,0 +1,112 @@ +package handler + +import ( + "context" + "encoding/json" + "net/http" + "strings" + "testing" + + "github.com/vtmocanu/uzi/api/internal/issueinput" + "github.com/vtmocanu/uzi/api/internal/issuetracker/youtracktest" +) + +// PRD #4173 M4 D11 through the real run capture: comment and author mapping on a linked repo. + +func (f *linkedHandlerFixture) wireRuns(t *testing.T) { + t.Helper() + f.h.wsvc.SetForges(f.h.svc) + f.h.wsvc.SetIssueSources(f.h.svc) + f.seedRunCredentials(t) +} + +// Acceptance 2: a comment by an unmapped author is withheld from the stored run snapshot; +// the connection account's own comment is kept. +func TestLinkedRunSnapshotWithholdsAnUnmappedAuthorsCommentLiveDB(t *testing.T) { + f := newLinkedHandlerFixture(t, true) + f.srv.SeedComment(f.issueN, youtracktest.OtherLogin, "UNMAPPED_RAW_TEXT") + f.srv.SeedComment(f.issueN, youtracktest.BotLogin, "the account's own comment") + f.syncOK(t) + f.wireRuns(t) + + rr := f.call(f.h.CreateRun, http.MethodPost, map[string]any{"issue_iid": f.issueN}) + if rr.Code != http.StatusCreated { + t.Fatalf("manual run: status = %d body = %s", rr.Code, rr.Body.String()) + } + var raw []byte + if err := f.pool.QueryRow(context.Background(), `SELECT issue_comments FROM runs WHERE repo_id = $1 AND issue_iid = $2`, f.repo, f.issueN).Scan(&raw); err != nil { + t.Fatal(err) + } + if strings.Contains(string(raw), "UNMAPPED_RAW_TEXT") { + t.Fatalf("an unmapped author's comment reached the stored snapshot: %s", raw) + } + var th issueinput.Thread + if err := json.Unmarshal(raw, &th); err != nil { + t.Fatal(err) + } + if len(th.Comments) != 2 { + t.Fatalf("comments = %+v, want 2", th.Comments) + } + if c := th.Comments[0]; c.AuthorUsername != youtracktest.OtherLogin || c.Body != issueinput.Placeholder || c.Reason != issueinput.Unknown { + t.Fatalf("unmapped comment = %+v", c) + } + if c := th.Comments[1]; c.AuthorUsername != youtracktest.BotLogin || c.Body != "the account's own comment" || c.Reason != "" { + t.Fatalf("the account's own comment = %+v", c) + } +} + +// Acceptance 3 (manual half): an unmapped issue author never stops a manual start. +func TestLinkedUnmappedIssueAuthorManualStartIsCreatedLiveDB(t *testing.T) { + f := newLinkedHandlerFixture(t, true) + f.srv.SetReporter(f.issueN, youtracktest.OtherLogin) + f.syncOK(t) + f.wireRuns(t) + + rr := f.call(f.h.CreateRun, http.MethodPost, map[string]any{"issue_iid": f.issueN}) + if rr.Code != http.StatusCreated { + t.Fatalf("manual run on an unmapped author's issue: status = %d body = %s", rr.Code, rr.Body.String()) + } + var n int + if err := f.pool.QueryRow(context.Background(), `SELECT count(*) FROM runs WHERE repo_id = $1 AND issue_iid = $2`, f.repo, f.issueN).Scan(&n); err != nil || n != 1 { + t.Fatalf("runs = %d (%v), want 1", n, err) + } + // The unmapped path ran: the run records the author as permission_unknown. + var reason *string + if err := f.pool.QueryRow(context.Background(), `SELECT issue_input_reason FROM runs WHERE repo_id = $1 AND issue_iid = $2`, f.repo, f.issueN).Scan(&reason); err != nil || reason == nil || *reason != "permission_unknown" { + t.Fatalf("issue_input_reason = %v (%v), want permission_unknown", reason, err) + } +} + +// Acceptance 3 (scheduled half): an unmapped author's scheduled run is created but not +// auto-approved and records permission_unknown; a mapped author's is auto-approved. +func TestLinkedUnmappedIssueAuthorScheduledRunIsNotAutoApprovedLiveDB(t *testing.T) { + for _, tc := range []struct { + name string + reporter string + wantApprove bool + wantReasons string + }{ + {"unmapped author", youtracktest.OtherLogin, false, issueinput.Unknown}, + {"mapped author", youtracktest.BotLogin, true, ""}, + } { + t.Run(tc.name, func(t *testing.T) { + f := newLinkedHandlerFixture(t, true) + f.srv.SetReporter(f.issueN, tc.reporter) + f.syncOK(t) + f.wireRuns(t) + + run, err := f.h.wsvc.CreateScheduledAutopilotRun(context.Background(), f.user.ID, f.repo, f.issueN, "", nil, nil, nil, false, nil, nil) + if err != nil { + t.Fatalf("scheduled run: %v", err) + } + var approve bool + var reasons []string + if err := f.pool.QueryRow(context.Background(), `SELECT auto_approve, auto_approve_blocked_reasons FROM runs WHERE id = $1`, run.ID).Scan(&approve, &reasons); err != nil { + t.Fatal(err) + } + if approve != tc.wantApprove || strings.Join(reasons, ",") != tc.wantReasons { + t.Fatalf("auto_approve = %v reasons = %v, want %v / %q", approve, reasons, tc.wantApprove, tc.wantReasons) + } + }) + } +} diff --git a/api/internal/handler/linked_repo_intake_livedb_test.go b/api/internal/handler/linked_repo_intake_livedb_test.go index 49fe7d3af6..25f71b5ef3 100644 --- a/api/internal/handler/linked_repo_intake_livedb_test.go +++ b/api/internal/handler/linked_repo_intake_livedb_test.go @@ -5,6 +5,7 @@ import ( "encoding/json" "errors" "net/http" + "net/http/httptest" "strconv" "strings" "testing" @@ -159,6 +160,66 @@ func TestLinkedRepoIncrementalSyncCachesBelowTheMarksAndNeverEvictsLiveDB(t *tes } } +// Acceptance 9: an incremental sync whose marks are earlier than the issue's updated time +// advances them to the newest updated time, and a later incremental sync after the issue +// moved to another project evicts nothing. +func TestLinkedRepoIncrementalSyncAdvancesEarlierMarksLiveDB(t *testing.T) { + f := newLinkedHandlerFixture(t, true) + ctx := context.Background() + early := forgesvc.Marks{PRD: time.Unix(1000, 0), Open: time.Unix(1000, 0)} + got, err := f.h.svc.SyncIssuesIncremental(ctx, f.user.ID, f.repo, 77, f.tracker(t), early) + if err != nil { + t.Fatal(err) + } + if !got.PRD.After(early.PRD) { + t.Fatalf("marks = %+v, want PRD advanced past %v", got, early.PRD) + } + // The PRD mark advances to exactly the newest updated time the walk returned. + row, ok := f.issueRow(t, f.issueN) + if !ok { + t.Fatal("the incremental sync did not cache the issue") + } + if !got.PRD.Equal(row.ForgeUpdatedAt.Time) { + t.Fatalf("PRD mark = %v, want the issue's updated time %v", got.PRD, row.ForgeUpdatedAt.Time) + } + f.srv.MoveIssue(f.issueN, youtracktest.OtherProjectID) + if _, err := f.h.svc.SyncIssuesIncremental(ctx, f.user.ID, f.repo, 77, f.tracker(t), got); err != nil { + t.Fatal(err) + } + if _, ok := f.issueRow(t, f.issueN); !ok { + t.Fatal("an incremental sync evicted the moved issue") + } +} + +// Acceptance (board contract): an unlinked repo's board cards carry no issue_key member. +func TestUnlinkedBoardCardsCarryNoIssueKeyLiveDB(t *testing.T) { + ctx := context.Background() + f := newBoardWriterFixture(ctx, t, &boardWriterStub{}) + mustExecT(ctx, t, f.pool, `INSERT INTO issues (repo_id, forge_issue_iid, title, state, labels, web_url, forge_updated_at, synced_at) + VALUES ($1, 5, 'plain card', 'opened', $2, 'https://forge.example/i/5', now(), now())`, f.repoID, `["`+wantUziLabel+`"]`) + rr := httptest.NewRecorder() + f.h.GetBoard(rr, boardWriterReq(f.user, f.repoID, "", "")) + if rr.Code != http.StatusOK { + t.Fatalf("board status = %d body = %s", rr.Code, rr.Body.String()) + } + var body struct { + Board struct { + Cards []map[string]any `json:"cards"` + } `json:"board"` + } + if err := json.Unmarshal(rr.Body.Bytes(), &body); err != nil { + t.Fatal(err) + } + if len(body.Board.Cards) == 0 { + t.Fatalf("the board has no cards, the check would be vacuous: %s", rr.Body.String()) + } + for _, c := range body.Board.Cards { + if _, has := c["issue_key"]; has { + t.Fatalf("an unlinked repo's card carries issue_key: %v", c) + } + } +} + // A failed listing evicts nothing: the cached row survives a full sync whose listing // fails (the issue tracker is gone), and the endpoint reports the failure. func TestLinkedRepoFailedListingEvictsNothingLiveDB(t *testing.T) { diff --git a/api/internal/workersvc/issue_key_claim_deleted_row_livedb_test.go b/api/internal/workersvc/issue_key_claim_deleted_row_livedb_test.go new file mode 100644 index 0000000000..82735f573d --- /dev/null +++ b/api/internal/workersvc/issue_key_claim_deleted_row_livedb_test.go @@ -0,0 +1,24 @@ +package workersvc + +import ( + "testing" + + "github.com/vtmocanu/uzi/api/internal/capability" +) + +// PRD #4173 M4 acceptance 12: the claim's issue key comes from the repo's project short name +// and the run's iid, so it survives the cached issues row being evicted (deleted). +func TestIssueKeyClaimSurvivesADeletedIssueRowLiveDB(t *testing.T) { + f := newCRFix(t) + f.linkFixtureRepo(t) + f.env.exec(`INSERT INTO issues (repo_id, forge_issue_iid, title, state, labels, web_url, forge_updated_at, synced_at, issue_key) + VALUES ($1, 12, 't', 'opened', '[]', 'https://yt.test/i', now(), now(), 'ABC-12')`, f.repo) + run := f.startQueuedIssueRun(t, 12) + f.env.exec(`DELETE FROM issues WHERE repo_id = $1 AND forge_issue_iid = 12`, f.repo) + + capable := f.protocolWorker(t, 0, capability.IssueKeyV1) + p := f.acceptClaim(t, capable, run, "linked run whose cached issue row is gone") + if p.IssueKey != "ABC-12" { + t.Fatalf("issue_key = %q, want ABC-12", p.IssueKey) + } +} diff --git a/api/internal/workersvc/issue_tracker_bounds_test.go b/api/internal/workersvc/issue_tracker_bounds_test.go index 17f07ef9fd..67a99e8cef 100644 --- a/api/internal/workersvc/issue_tracker_bounds_test.go +++ b/api/internal/workersvc/issue_tracker_bounds_test.go @@ -2,6 +2,7 @@ package workersvc import ( "context" + "encoding/json" "net/http" "strings" "testing" @@ -124,3 +125,39 @@ func TestOversizedIssueCommentIsBoundedLikeTheForgeEquivalent(t *testing.T) { t.Fatalf("forge thread bytes=%d truncated=%v, want exactly %d and truncated", ftotal, want.Thread.Truncated, bound) } } + +// An oversized YouTrack comment by the connection's own (mapped) account reaches the run +// through the linked capture and is cut to exactly the forge bound, marked truncated, like +// the forge equivalent above. +func TestOversizedMappedIssueCommentIsTruncatedLikeTheForgeEquivalent(t *testing.T) { + const bound = 32768 + srv := youtracktest.NewServer(t) + n := srv.SeedIssue("card", "body", youtracktest.EligibilityTagID) + srv.SetReporter(n, youtracktest.BotLogin) + srv.SeedComment(n, youtracktest.BotLogin, strings.Repeat("c", 3<<20)) + + row := linkedRepoRow() + row.ForgeProjectID = 77 // the driver refuses any other project id + row.EligibilityTagName.String = youtracktest.EligibilityTagName + fs := &fakeStore{repoRow: row, issueByID: store.Issue{Labels: []byte(`["` + youtracktest.EligibilityTagName + `"]`)}, createRunResult: store.Run{ID: uuid.New()}} + svc := New(fs, newBox(t), testParams()) + svc.SetForges(m2Builder{&m2Forge{disposition: forge.AuthorEligible}}) + svc.SetIssueSources(&fakeSources{tr: linkedYouTrack{Driver: youTrackDriver(t, srv)}}) + if _, err := svc.CreateRun(context.Background(), uuid.New(), uuid.New(), n, "", nil, nil, false, nil, nil, nil, nil); err != nil { + t.Fatal(err) + } + var th issueinput.Thread + if err := json.Unmarshal(fs.createRunParams.IssueComments, &th); err != nil { + t.Fatal(err) + } + total := 0 + for _, c := range th.Comments { + total += len(c.Body) + } + if total != bound || !th.Truncated { + t.Fatalf("YouTrack thread bytes=%d truncated=%v, want exactly %d and truncated", total, th.Truncated, bound) + } + if len(th.Comments) == 0 || th.Comments[0].Reason != "" || !strings.Contains(th.Comments[0].Body, "cccc") { + t.Fatalf("the mapped account's comment did not reach the run: %+v", th.Comments) + } +} diff --git a/changelog.d/3596cf4e-b899-430d-ab89-ff2f43ae3085.md b/changelog.d/3596cf4e-b899-430d-ab89-ff2f43ae3085.md new file mode 100644 index 0000000000..0ea98d09f9 --- /dev/null +++ b/changelog.d/3596cf4e-b899-430d-ab89-ff2f43ae3085.md @@ -0,0 +1,4 @@ +### Added + +- **A repo created after this release can be linked to a YouTrack project and eligibility tag, and tagged issues start runs (#4173).** + The link is made through the API before the repo's first enable, once the operator allows the YouTrack base URL (web and CLI screens come later); tagged issues sync onto the board with their YouTrack key (`ABC-12`), and comments and issue text by accounts other than the connection's own are withheld or need approval; issue writes on a linked repo come in a later release. Roll out the API before the workers: a worker needs `issue_key_v1` to claim a linked run. diff --git a/deploy/compose/compose.yaml b/deploy/compose/compose.yaml index d54f3778f2..46e6bc2c67 100644 --- a/deploy/compose/compose.yaml +++ b/deploy/compose/compose.yaml @@ -49,6 +49,9 @@ services: UZI_OIDC_ADMIN_GROUPS: ${UZI_OIDC_ADMIN_GROUPS:-} UZI_OIDC_ALLOWED_GROUPS: ${UZI_OIDC_ALLOWED_GROUPS:-} FORGE_ALLOWED_BASE_URLS: ${FORGE_ALLOWED_BASE_URLS:-https://gitlab.example.com} + # Issue tracker SSRF allowlist (PRD #4173 S3): separate from the forge one, https + # only, empty by default (no issue tracker connection can be created). Not worker egress. + ISSUE_TRACKER_ALLOWED_BASE_URLS: ${ISSUE_TRACKER_ALLOWED_BASE_URLS:-} FORGE_POLL_INTERVAL: ${FORGE_POLL_INTERVAL:-60s} FORGE_RECONCILE_EVERY: ${FORGE_RECONCILE_EVERY:-10} FORGE_HTTP_TIMEOUT: ${FORGE_HTTP_TIMEOUT:-15s} diff --git a/e2e/README.md b/e2e/README.md index 48d2a07c8b..beb8b90466 100644 --- a/e2e/README.md +++ b/e2e/README.md @@ -154,11 +154,11 @@ a plain `go test ./...` when `UZI_TEST_DATABASE_URL` is unset. The full PRD #24 is `./e2e/run-e2e.sh` **and** `./e2e/run-store-it.sh`. The script has since grown past that one query: it sweeps **every `*LiveDB` test in -the seven packages that carry one**, `store`, `handler`, `forgesvc`, `schedsvc`, -`workersvc`, `recovery` and `poller`, serially (`-p 1`, one shared database). CI runs this script once, in +the eight packages that carry one**, `store`, `handler`, `forgesvc`, `schedsvc`, +`workersvc`, `recovery`, `poller` and `issuetrackersvc`, serially (`-p 1`, one shared database). CI runs this script once, in `test-codex-refresh-lostreply-e2e`, and `test-api-store-it` is a thin context that requires that run to have succeeded. A `*LiveDB` test in a package the script's list -does not name runs nowhere and still prints `ok`, so a new live-DB test in an eighth +does not name runs nowhere and still prints `ok`, so a new live-DB test in a ninth package must add that package to `PKGS` in the script. The script runs each package in a separate Go invocation against the same DSN, diff --git a/e2e/reset-recovery-tables.sh b/e2e/reset-recovery-tables.sh index 8e5c6e9a5f..a81a3bcbe2 100755 --- a/e2e/reset-recovery-tables.sh +++ b/e2e/reset-recovery-tables.sh @@ -3,7 +3,7 @@ # THROWAWAY test database. HARNESS ISOLATION ONLY. # # WHY THIS EXISTS. The store-it suite (e2e/run-store-it.sh) runs every *LiveDB test in -# seven packages against ONE shared throwaway Postgres (`-p 1`), and — as run-store-it.sh +# eight packages against ONE shared throwaway Postgres (`-p 1`), and — as run-store-it.sh # itself documents for `workers` — nothing truncates a table between package binaries, so # rows one package's test seeds outlive it. The recovery tables (recovery_custody_holds, # recovery_captures, recovery_capture_chunks, custody_episode_notices) are seeded by the diff --git a/e2e/run-store-it.sh b/e2e/run-store-it.sh index 5b06a8df26..37d8197f7f 100755 --- a/e2e/run-store-it.sh +++ b/e2e/run-store-it.sh @@ -12,8 +12,8 @@ # quota that lets four workers through here. # # Spins up a THROWAWAY Postgres, points UZI_TEST_DATABASE_URL at it, applies the -# real goose migrations, seeds fixtures, and runs every *LiveDB test in the seven -# packages that carry one (store, handler, forgesvc, schedsvc, workersvc, recovery, poller -- the +# real goose migrations, seeds fixtures, and runs every *LiveDB test in the eight +# packages that carry one (store, handler, forgesvc, schedsvc, workersvc, recovery, poller, issuetrackersvc -- the # list on the `go test` line below; ci.yml runs this script once, via # `task test:codex-refresh-lostreply-e2e`, for both CI contexts). # Isolated: unique container + published loopback port, torn down on exit; never diff --git a/e2e/run-store-it.test.sh b/e2e/run-store-it.test.sh index b94edf4c26..073e23c8b7 100755 --- a/e2e/run-store-it.test.sh +++ b/e2e/run-store-it.test.sh @@ -147,7 +147,7 @@ else fi # Issue #2665: a ready server exposes the public Docker/Go seams without waiting -# for real LiveDB tests. Each scenario is bounded to seven package calls plus the +# for real LiveDB tests. Each scenario is bounded to eight package calls plus the # optional handler call; a failed scenario is recorded without blocking siblings. READYBIN="$TMP/ready-bin" mkdir -p "$READYBIN" @@ -262,7 +262,7 @@ assert_case() { } # Explicit expected order, independent of the runner's PKGS array. -expected_pkgs=(store handler forgesvc schedsvc workersvc recovery poller) +expected_pkgs=(store handler forgesvc schedsvc workersvc recovery poller issuetrackersvc) for scenario in success optional retained-missing-run retained-missing-closure-run retained-missing-pass retained-top-skip retained-sub-skip retained-go-fail go-fail no-tests missing-ok top-skip sub-skip; do case_dir="$TMP/$scenario" mkdir -p "$case_dir" @@ -306,8 +306,8 @@ for scenario in success optional retained-missing-run retained-missing-closure-r case "$scenario" in success|optional) assert_case "successful sweep exits zero" test "$rc" -eq 0 - assert_case "aggregate log contains all seven passes" \ - grep -qF 'LiveDB: 7 passed, 0 skipped' "$case_dir/output" + assert_case "aggregate log contains all eight passes" \ + grep -qF 'LiveDB: 8 passed, 0 skipped' "$case_dir/output" assert_case "success banner appears" grep -qF 'Store integration tests passed.' "$case_dir/output" assert_case "successful log files removed" \ test -z "$(find "$case_dir" -name '*-log.*' -print)" @@ -338,7 +338,7 @@ for scenario in success optional retained-missing-run retained-missing-closure-r ;; top-skip|sub-skip) assert_case "aggregate guard retains and rejects an early skip at any depth" \ - grep -qF 'LiveDB: 7 passed, 1 skipped' "$case_dir/output" + grep -qF 'LiveDB: 8 passed, 1 skipped' "$case_dir/output" ;; esac ;; @@ -428,14 +428,14 @@ for scenario in checkpoint-success checkpoint-disabled checkpoint-go-fail checkp fi calls="$(wc -l < "$case_dir/go.log")" if [ "$enabled" = 1 ]; then - assert_case "normal seven packages plus checkpoint leg" test "$calls" -eq 8 + assert_case "normal eight packages plus checkpoint leg" test "$calls" -eq 9 assert_case "checkpoint command bounded, uncached, serialized, race-enabled" \ grep -Fq "$(printf '\ttest\t-buildvcs=false\t-count=1\t-v\t-race\t-p\t1\t-timeout\t15m\t-run\t^TestTaskCheckpointCrossWorkerRecoveryE2E$\t./internal/workersvc/...')" "$case_dir/go.log" proof_log="$(awk '/^Task checkpoint recovery proof log: / {print $NF}' "$case_dir/output")" assert_case "checkpoint log retained" test -f "$proof_log" case "$proof_log" in "$ROOT/.uzi/scratch/"*) rm -f "$proof_log" ;; esac else - assert_case "non-1 value leaves seven-package sweep unchanged" test "$calls" -eq 7 + assert_case "non-1 value leaves eight-package sweep unchanged" test "$calls" -eq 8 fi done done diff --git a/prds/4173-external-issue-trackers.md b/prds/4173-external-issue-trackers.md index 25ccaedce1..747d4e77d0 100644 --- a/prds/4173-external-issue-trackers.md +++ b/prds/4173-external-issue-trackers.md @@ -4,7 +4,7 @@ **Priority:** Medium -**Status:** In progress, r7 2026-11-12. M1, M2 and M3 delivered; M4 in progress. r3 narrowed scope to one real issue tracker; r4 fixed a repo's issue source before first enable (D13); r4 and r5 fixed the YouTrack id, tag and author decisions before the driver; r6 records the M3 rework that follows them; r7 records the M4 preconditions from the maintainer's M3 approval (id-scoped intake, `ListLabels` limited to usable tags) and the M4 design decisions. +**Status:** In progress, r8 2026-11-13. M1–M4 delivered; M5 and M6 pending. r3 narrowed scope to one real issue tracker; r4 fixed a repo's issue source before first enable (D13); r4 and r5 fixed the YouTrack id, tag and author decisions before the driver; r6 records the M3 rework that follows them; r7 records the M4 preconditions from the maintainer's M3 approval (id-scoped intake, `ListLabels` limited to usable tags) and the M4 design decisions; r8 records M4's completion. **Delivery:** one PR per milestone, each merged before the next starts. No implementation or validation step touches `.github/workflows/**`. @@ -137,7 +137,7 @@ Progress (one PR per milestone, each merged before the next starts): - [x] **M1:** the `IssueTracker` and `CodeHost` interfaces - [x] **M2:** every caller on its half. The type-checked inventory shows 45 issue-side calls through `IssueTracker`, 66 code-host calls through `CodeHost` and none through `forge.Forge`, and handing `privcheck` an `IssueTracker`-only value fails to compile. - [x] **M3:** issue tracker connections and the YouTrack driver. The contract suite, with the `ProjectGuard` and `SameNameTagUsesEligibilityTag` cases, runs on the real YouTrack driver and the three real forge drivers; the tag sharing settings parse is tested against a response recorded from a real YouTrack; the chart renders `issueTracker.allowedBaseURLs` and refuses a non-https entry. -- [ ] **M4:** issue identity, the repo link and issue intake from YouTrack +- [x] **M4:** issue identity, the repo link and issue intake from YouTrack. Live-DB acceptance tests with the YouTrack fake cover linked intake, D11 author mapping and eviction; the linked claim golden is read by the agent contract test; the recorded-call golden is unchanged. - [ ] **M5:** the board and issue writes on YouTrack - [ ] **M6:** web and CLI surfaces, docs and live acceptance @@ -274,3 +274,4 @@ Settings → Issue trackers, the CLI commands, a setup page under `docs/`. - **2026-11-12: a linked run's board-free MR watch keeps the reopen edge (M4 review).** The board lane excludes a linked repo, so its board-free lane also watches a linked issue run's closed MR while that run is the issue's latest and the cached issue is open (the board lane's Lane A bound), and records a reopen as `opened`; live rows sort ahead of closed ones under the lane's `LIMIT 100`, so a closed watch never starves a live MR. An issue evicted from the cache (moved, untagged) drops the watch. Left for M5, where issue writes become real: a linked run whose issue is not cached yet has its pending card move cleared as moved (harmless while writes are refused), and the PRD link patch reads the issue before its refused write settles the marker. - **2026-11-12: the id-scoped walk fails closed at the listing ceilings, which other projects' issues count toward (D14, M4 security review).** The tag listing returns the tag's issues from every project, resolved ones included, and the item, page and byte ceilings count them before the client keeps only the bound project. So whoever can apply the eligibility tag, or its slow accumulation on resolved issues, can push the walk past a ceiling (about 50,000 tagged issues by items, fewer by bytes because descriptions are read); then every listing fails and intake for the bound project stops, logged, evicting nothing, until the tag is taken off issues elsewhere. Memory stays bounded. The people who can apply the tag are the ones the admin confirmed at link time should start work (D14), so this is a fail-closed availability limit, not an escalation. Operator remedy: remove the eligibility tag from issues outside the linked project, or from resolved ones; the documented project-walk fallback would avoid the cross-project part. Tested in the driver: one eligible issue plus more than a lowered item ceiling in another project fails the listing rather than returning part of it. - **2026-11-12: a worker must advertise `issue_key_v1` to claim a run on a linked repo (M4 review).** A worker image from before M4 ignores `issue_key` and would write `Closes #` into its first merge request body, closing an unrelated code-host issue; the server stamping the publication non-closing does not stop it. uzi already allows the API and the workers to run different versions (each protocol change has its own claim capability), so "ship together" is not a guarantee. `ClaimRun` gets a standalone clause, outside `fn_worker_can_claim`, `required_capabilities` and the kill-switch: a run whose repo has an issue tracker link is claimable only by a worker whose stored protocol capabilities contain `issue_key_v1`. The clause is mirrored at every site of the `ci_config_guard_v1` clause and pinned by a test, and the ephemeral placement listing counts such a run as demand. Without such a worker the run waits, with a health reason naming the capability. Roll the API out before the workers; a newer worker against an older API is unaffected. Rejected: a documented ship-together boundary (nothing enforces it), and refusing at claim time (the run would have to be released and requeued). +- **2026-11-13: M4 is complete; three gaps go to M6.** The final M4 run added live-DB acceptance tests for linked intake and D11 (comments and unmapped author, manual and scheduled start), incremental marks, a deleted-row claim keeping its `issue_key`, and a board with no `issue_key` for an unlinked repo; and a web board card with an `issue_key` counts as uzi's and shows its key. Deferred to M6: the issue detail page and its API payload do not carry `issue_key` yet, so on a linked repo the detail page decides eligibility by labels and shows `#iid`, and can offer Promote, whose write is refused until M5; board search does not match the key; screen-reader labels still name `#iid`. Correction to the 2026-11-11 entry "M3 layout": M4 wired the connection and repo link routes into `cmd/server` (`PUT /api/repos/{id}/issue-tracker-link`, session-only); the web and CLI screens remain M6, and a connection still needs the operator's `ISSUE_TRACKER_ALLOWED_BASE_URLS`. S4's comment bound is tested for the connection's own account (truncated to the forge bound) and for an unmapped author (withheld). diff --git a/web/src/lib/apiTypes.ts b/web/src/lib/apiTypes.ts index 6106806bd8..edbfa1760a 100644 --- a/web/src/lib/apiTypes.ts +++ b/web/src/lib/apiTypes.ts @@ -1196,6 +1196,10 @@ export interface Card { title: string; state: string; labels: string[]; + // The issue tracker's readable id (e.g. "ABC-12", PRD 4173 M4), set only for a repo linked + // to an issue tracker. Shown in place of `#iid`, and a keyed card is eligible by its + // issue tracker tag, so the board treats it as uzi's without the `uzi` label. + issue_key?: string; // Forge user ids assigned to the issue (PRD #767 M5). The board widens "is this card // uzi's to run" to "carries the `uzi` label OR the board's bot is one of these ids", // so this rides the card alongside labels. A current server always sends [] not null, diff --git a/web/src/lib/boardCards.test.ts b/web/src/lib/boardCards.test.ts index 1f97bd31a0..cfa7ac589b 100644 --- a/web/src/lib/boardCards.test.ts +++ b/web/src/lib/boardCards.test.ts @@ -82,6 +82,18 @@ describe("isUziCard", () => { expect(isUziCard(card([], false, [BOT]), "uzi", undefined)).toBe(false); expect(isUziCard(card(["uzi"], false, []), "uzi", undefined)).toBe(true); }); + + it("counts a card carrying an issue_key as uzi's even without the uzi label", () => { + const keyed = { ...card(["Bug"]), issue_key: "ABC-12" }; + expect(isUziCard(keyed, "uzi", NO_BOT)).toBe(true); + expect(canPromote(keyed, "uzi", NO_BOT)).toBe(false); + }); + + it("keeps the label/assignee rule when issue_key is absent or empty", () => { + expect(isUziCard({ ...card(["Bug"]) }, "uzi", NO_BOT)).toBe(false); + expect(isUziCard({ ...card(["Bug"]), issue_key: "" }, "uzi", NO_BOT)).toBe(false); + expect(isUziCard({ ...card(["uzi"]), issue_key: "" }, "uzi", NO_BOT)).toBe(true); + }); }); describe("visibleCards", () => { diff --git a/web/src/lib/boardCards.ts b/web/src/lib/boardCards.ts index 94444841db..22d33baecf 100644 --- a/web/src/lib/boardCards.ts +++ b/web/src/lib/boardCards.ts @@ -42,6 +42,9 @@ export const SELF_IMPROVE_LABEL = "uzi-self-improve"; * render filter, the runnable marker, the solid-vs-quiet treatment and the * Start-vs-Promote affordance, so the button state matches what a Start would do. * + * A card with a non-empty `issue_key` is also uzi's: the linked sync caches only issues + * that carry the link's eligibility tag, so its presence is the eligibility signal. + * * `botForgeUserID > 0` guards an unresolved bot id (mirrors the server's `@bot_id>0` / * M2's `botID<=0` guard): a 0/absent id must never mark every assigned card runnable. * @@ -52,11 +55,14 @@ export const SELF_IMPROVE_LABEL = "uzi-self-improve"; * predicate falls back to label-only rather than throwing on `undefined.includes`. */ export function isUziCard( - card: Pick, + card: Pick, uziLabel: string, botForgeUserID: number | undefined, ): boolean { const botId = botForgeUserID ?? 0; + // A card the linked sync cached carries an issue_key and is eligible by its issue tracker + // tag (a stored name that may differ from the configured uzi label), so it needs no label. + if (card.issue_key) return true; return card.labels.includes(uziLabel) || (botId > 0 && (card.assignee_ids ?? []).includes(botId)); } diff --git a/web/src/pages/board/IssueCard.test.tsx b/web/src/pages/board/IssueCard.test.tsx index 3eb7ac417a..17a78af26e 100644 --- a/web/src/pages/board/IssueCard.test.tsx +++ b/web/src/pages/board/IssueCard.test.tsx @@ -163,3 +163,14 @@ it.each([true, false])("active account wait card respects owner metadata (mine=% if (!is_mine) expect(container.textContent).not.toContain("waiting for account cooldown"); expect(r.id).toBe("same-run"); }); + +it("shows the issue key instead of #iid when the card carries one", () => { + const { container } = renderCard({ issue_key: "ABC-12" }); + expect(container.textContent).toContain("ABC-12"); + expect(container.textContent).not.toContain("#7"); +}); + +it("shows #iid when the card has no issue key", () => { + const { container } = renderCard(); + expect(container.textContent).toContain("#7"); +}); diff --git a/web/src/pages/board/IssueCard.tsx b/web/src/pages/board/IssueCard.tsx index 972c03e94f..dec4407a15 100644 --- a/web/src/pages/board/IssueCard.tsx +++ b/web/src/pages/board/IssueCard.tsx @@ -426,7 +426,7 @@ export function IssueCard({ )} - #{card.iid} + {card.issue_key || `#${card.iid}`}