Skip to content

Feat/el status - #289

Open
AlexAxthelm wants to merge 9 commits into
mainfrom
feat/el-status
Open

Feat/el status#289
AlexAxthelm wants to merge 9 commits into
mainfrom
feat/el-status

Conversation

@AlexAxthelm

@AlexAxthelm AlexAxthelm commented Sep 24, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Add progress indicators to the Entity-linkage status endpoint (rather than a None while running), and surface in frontend, so that very long (multi-day) EL runs don't look like they've stalled. Updates progress every 100 resources processed.

Also includes behavior change of setting "initate merges" to be default behavior

resolves STIT-740

image

Claude's description:

feat(entity-linkage): surface linkage run progress (STIT-740)

Why

A full entity-linkage pass streams the whole resource set and can run for over an
hour (often longer) at production scale. Until now the status endpoint only
reported a coarse running state with no numerator, denominator, or timestamp,
so the Entity Linkage page sat on "running" indefinitely with no sense of
progress or whether anything was happening. This addresses the STIT-740 AC of
surfacing a progress signal (count processed / total + last-updated).

What changed

Backend (deployments/entity-linkage)

  • New LinkProgress model (scanned / total / created / skipped / failed +
    updated_at), carried on JobRecord as an optional progress field.
  • JobManager run thunk now receives the live record, so the running pass can
    write progress onto it; matching.link_all emits a LinkProgress snapshot
    every 100 resources plus a final one.
  • The resource total (the "of N" denominator) is fetched once at run start, and
    only when a progress consumer is listening.
  • The existing GET /oil-gas-fields/link/status returns progress as-is; no new
    endpoint. The frontend's existing 2s poll surfaces it automatically.

Frontend (deployments/stitch-frontend)

  • The Entity Linkage page renders a progress bar with "Processing X of Y", a
    floored percent (never shows 100% before completion), per-outcome counts, and a
    date+time "Last updated".
  • Unknown or stale totals fall back to an indeterminate bar instead of a stuck
    100%.
  • Run controls reordered (buttons above the "Initiate merges" checkbox).

Behavior change

The "Initiate merges" checkbox now defaults to on, so a default "Start run"
creates merge candidates rather than doing a dry run. Untick it for a dry run.

Testing

  • Automated — added/updated unit tests:
    • backend: progress written onto the live record; link_all emits progress via
      callback; throttled mid-run emission fires over many resources; total-fetch is
      skipped when no progress consumer; run-thunk signature change.
    • frontend: progress bar + counts render; near-complete shows 99% not 100%;
      indeterminate bar when total is unknown; indeterminate when scanned passes a
      stale total; checkbox defaults on and posts apply_merges: true.
  • Manual — ran the full local stack, seeded to ~3,300 resources (incl. ~200
    duplicate groups), started a run, and confirmed the bar advances live via the
    2s poll and lands on the final result view.
  • make check before opening.

AI assistance

Implemented with Claude Code (Claude Opus). AI was used to draft the progress
plumbing, the frontend progress view, and the tests; I reviewed the diff, ran a
code review over it, and verified behavior manually against a seeded local stack.
The tests listed above validate the change.

Known follow-ups (not blocking)

  • Progress state is in-memory per worker (pre-existing jobs.py limitation);
    must move to shared storage before scaling past one uvicorn worker.
  • On a dry run the progress line shows "0 candidates created · 0 skipped" (always
    zero without apply-merges) — cosmetic.
  • STIT-740 AC on infra sizing / demo-safe pre-run guidance is tracked separately.

🤖 Generated with Claude Code

AlexAxthelm and others added 8 commits September 24, 2026 17:43
@AlexAxthelm @claude
feat(entity-linkage): surface linkage run progress (STIT-740)
5369f1f 
The bulk linkage pass runs for over an hour at production scale, but the
status endpoint only reported a coarse "running" state — progress counters
lived as locals inside link_all and never reached the polled JobRecord.

Surface in-flight progress so the Entity Linkage page can show how far a run
has gotten:

- add a LinkProgress model (scanned/total/created/skipped/failed + updated_at)
- carry progress on JobRecord; RunThunk now receives the record so the run
  body can update it live
- fetch the resource total once per run for the "X of Y" denominator
- emit progress from link_all every 100 resources plus a final snapshot
- render a progress bar, counts, last-updated, and expected-duration copy on
  the Entity Linkage page (surfaced automatically by the existing 2s poll)

Progress state remains in-memory per worker (pre-existing limitation).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@AlexAxthelm
remove RUN_DURATION_HINT
714af76 
it was way understating
@AlexAxthelm
Select "Initiate Merges" by default, move checkbox
1c353ac
@AlexAxthelm @claude
test(entity-linkage): cover linkage-progress UI edge cases
3b088f2 
Guard two branches of the Entity Linkage page that were previously untested:

- the "Initiate merges" checkbox defaults to checked and Start run posts
  apply_merges:true, so a regression flipping the default back fails CI
- LinkProgressView renders an indeterminate bar (no percent, no aria-valuenow)
  when the run reports an unknown total_resources

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@AlexAxthelm
include date in updated string
215bb90
@AlexAxthelm @claude
correct progress percent and trim linkage overhead
15b2be3 
Address review findings on the linkage progress feature:

- floor the progress percent instead of rounding, so the bar no longer shows
  100% while the run is still on its final ~0.5% of resources
- fetch the resource total only when a progress consumer is listening, avoiding
  an extra API round trip for link_all callers that ignore progress

Add tests covering the near-complete (99% not 100%) case and that the total
fetch is skipped when no on_progress callback is passed.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@AlexAxthelm @claude
test(entity-linkage): cover throttled mid-run progress emission
ce65fa1 
link_all previously only had progress tests with <= 3 resources, exercising the
final snapshot but never the periodic PROGRESS_UPDATE_EVERY heartbeat that
delivers live progress during a long run.

Add a test scanning 2*PROGRESS_UPDATE_EVERY+5 resources that asserts on_progress
fires multiple times with strictly increasing resources_scanned, a single
denominator fetch, and total_resources on every snapshot.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@AlexAxthelm @claude
fix(entity-linkage): keep progress indeterminate past a stale total
72e521b 
The progress denominator is a start-of-run snapshot, so on a long run a growing
dataset can push resources_scanned to or past it. Rather than showing a stuck
"100%" (or "3,050 of 3,000"), treat total-known-but-scanned>=total as
indeterminate — same rendering as an unknown total.

Add a test covering scanned > total while running.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@AlexAxthelm
AlexAxthelm requested a balanced review from Copilot September 24, 2026 16:12
Copilot AI reviewed Sep 24, 2026
View reviewed changes

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Initial progress can remain unavailable, and exact-total runs incorrectly become indeterminate.

Get a fresh assessment by requesting another Copilot review.

Review effort: Balanced
Findings: 2 Medium severity · 1 Low severity

Open (3)
What changed in this PR

Adds live entity-linkage progress reporting across the backend status API and frontend.

Changes:

  • Tracks scanned resources, totals, outcomes, and timestamps.
  • Displays determinate or indeterminate frontend progress.
  • Enables “Initiate merges” by default.
File Description
deployments/​stitch-frontend/​src/​pages/​EntityLinkagePage.jsx Displays progress and changes merge defaults.
deployments/​stitch-frontend/​src/​pages/​EntityLinkagePage.test.jsx Tests progress UI and default merges.
deployments/​entity-linkage/​src/​stitch/​entity_linkage/​entities.py Defines the progress model.
deployments/​entity-linkage/​src/​stitch/​entity_linkage/​client.py Fetches the resource total.
deployments/​entity-linkage/​src/​stitch/​entity_linkage/​matching.py Emits progress snapshots during linkage.
deployments/​entity-linkage/​src/​stitch/​entity_linkage/​jobs.py Allows jobs to update live records.
deployments/​entity-linkage/​src/​stitch/​entity_linkage/​routers/​link.py Connects progress updates to job status.
deployments/​entity-linkage/​tests/​test_matching.py Tests progress emission and totals.
deployments/​entity-linkage/​tests/​test_jobs.py Tests live progress storage.
deployments/​entity-linkage/​tests/​test_link_api.py Tests serialized status progress.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread deployments/entity-linkage/src/stitch/entity_linkage/matching.py
Comment thread deployments/stitch-frontend/src/pages/EntityLinkagePage.jsx Outdated
Comment thread deployments/entity-linkage/src/stitch/entity_linkage/jobs.py
@github-actions

github-actions Bot commented Sep 24, 2026

Copy link
Copy Markdown

CD summary 72e521b

Frontend: https://witty-mushroom-017a3dc1e-289.westus2.1.azurestaticapps.net

Deployments (4)
service url fqdn
api open pr-0289-api.purplegrass-c07d0a94.westus2.azurecontainerapps.io
entity-linkage open pr-0289-el.purplegrass-c07d0a94.westus2.azurecontainerapps.io
frontend https://witty-mushroom-017a3dc1e-289.westus2.1.azurestaticapps.net
stitch-llm open pr-0289-llm.purplegrass-c07d0a94.westus2.azurecontainerapps.io
Database (1)
db_name postgres_host postgres_port postgres_db
pr_0289 stitch-dev.postgres.database.azure.com 5432 pr_0289
Jobs (2)
job image postgres_db api_url auth_mode
db-migrations ghcr.io/rmi/stitch-api:pr-0289@sha256:ded5ff29af2b17572128e64b74d1165f4030c12cc16fef8762be1d26e41ef7f3 pr_0289
seed ghcr.io/rmi/stitch-seed:pr-0289@sha256:bc9584f84ad2a038038150b20f5c00c02f8ad98bfb14e0e1817fb06da717507f https://pr-0289-api.purplegrass-c07d0a94.westus2.azurecontainerapps.io/api/v1 stitch-client-bearer-token
Images (4)
build_time commit_time git_sha image image_digest
2026-09-24T16:13:42Z 2026-09-24T16:12:51Z ef328c7 ghcr.io/rmi/stitch-api:pr-0289 ghcr.io/rmi/stitch-api:pr-0289@sha256:ded5ff29af2b17572128e64b74d1165f4030c12cc16fef8762be1d26e41ef7f3
2026-09-24T16:13:44Z 2026-09-24T16:12:51Z ef328c7 ghcr.io/rmi/stitch-entity-linkage:pr-0289 ghcr.io/rmi/stitch-entity-linkage:pr-0289@sha256:d822f4fb0e7c7f0bc49bce68aa8ce001a4286129093f57202f10e8f8f5429472
2026-09-24T16:13:46Z 2026-09-24T16:12:51Z ef328c7 ghcr.io/rmi/stitch-seed:pr-0289 ghcr.io/rmi/stitch-seed:pr-0289@sha256:bc9584f84ad2a038038150b20f5c00c02f8ad98bfb14e0e1817fb06da717507f
2026-09-24T16:13:46Z 2026-09-24T16:12:51Z ef328c7 ghcr.io/rmi/stitch-stitch-llm:pr-0289 ghcr.io/rmi/stitch-stitch-llm:pr-0289@sha256:12d78d01b1cc9acfa11e9ae4e87463fc22b249d93e97fb3de8b846c610b2e62a

@AlexAxthelm @claude
address progress review feedback
a637e05 
- emit an initial 0/total progress snapshot before scanning, so pollers see a
  denominator immediately and progress is observable on sub-100-resource runs
- keep the progress bar determinate when scanned == total (clamped to 99%);
  only scanned > total (a stale, grown-past total) falls back to indeterminate
- correct the JobManager docstring: the run body now receives the live JobRecord

Add tests for the initial snapshot and the scanned == total boundary.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@AlexAxthelm
AlexAxthelm marked this pull request as ready for review September 24, 2026 16:26
@AlexAxthelm AlexAxthelm self-assigned this Sep 24, 2026
@github-actions

github-actions Bot commented Sep 24, 2026

Copy link
Copy Markdown

CD summary a637e05

Frontend: https://witty-mushroom-017a3dc1e-289.westus2.1.azurestaticapps.net

Deployments (4)
service url fqdn
api open pr-0289-api.purplegrass-c07d0a94.westus2.azurecontainerapps.io
entity-linkage open pr-0289-el.purplegrass-c07d0a94.westus2.azurecontainerapps.io
frontend https://witty-mushroom-017a3dc1e-289.westus2.1.azurestaticapps.net
stitch-llm open pr-0289-llm.purplegrass-c07d0a94.westus2.azurecontainerapps.io
Database (1)
db_name postgres_host postgres_port postgres_db
pr_0289 stitch-dev.postgres.database.azure.com 5432 pr_0289
Jobs (1)
job image postgres_db
db-migrations ghcr.io/rmi/stitch-api:pr-0289@sha256:b00f4b5156451d816f74786c6d7886af77c304fa27a79cd1846bf56dbd91705e pr_0289
Images (4)
build_time commit_time git_sha image image_digest
2026-09-24T16:24:03Z 2026-09-24T16:23:44Z 16e9626 ghcr.io/rmi/stitch-api:pr-0289 ghcr.io/rmi/stitch-api:pr-0289@sha256:b00f4b5156451d816f74786c6d7886af77c304fa27a79cd1846bf56dbd91705e
2026-09-24T16:24:03Z 2026-09-24T16:23:44Z 16e9626 ghcr.io/rmi/stitch-entity-linkage:pr-0289 ghcr.io/rmi/stitch-entity-linkage:pr-0289@sha256:5c39c3a5d611cc003b1fadf211d191898d771add8590820a366aeed6e3140f8a
2026-09-24T16:24:01Z 2026-09-24T16:23:44Z 16e9626 ghcr.io/rmi/stitch-seed:pr-0289 ghcr.io/rmi/stitch-seed:pr-0289@sha256:5c551acfa165e09e801df656b72c0064e707dd553ba8e9cbc11cc26edd5130cf
2026-09-24T16:24:01Z 2026-09-24T16:23:44Z 16e9626 ghcr.io/rmi/stitch-stitch-llm:pr-0289 ghcr.io/rmi/stitch-stitch-llm:pr-0289@sha256:50c63844cda4d58971e8fab211a0c3f7db67d446c099f3f63bf27aa4f9e68802

This branch was successfully deployed

1 active deployment
development — a637e057 Deployed Sep 24, 2026 by AlexAxthelm via Deploy frontend / Build and Deploy Job #1040
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot
Copilot review effort, defaults to Balanced
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments
@mbarlow12 mbarlow12 Awaiting requested review from mbarlow12
@jdhoffa jdhoffa Awaiting requested review from jdhoffa

At least 1 approving review is required to merge this pull request.

Assignees

@AlexAxthelm AlexAxthelm

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@AlexAxthelm