Skip to content

feat(runner): publish the project catalog before Git enrichment - #1415

Merged
swjnxyf merged 1 commit into
yyjeqhc:mainfrom
HarpeLm:feat/runner-metadata-first-inventory
Nov 10, 2026
Merged

swjnxyf merged 1 commit into
yyjeqhc:mainfrom
HarpeLm:feat/runner-metadata-first-inventory

Conversation

@HarpeLm

@HarpeLm HarpeLm commented Nov 10, 2026

Copy link
Copy Markdown
Contributor

Fixes #1271

What changed

The first project inventory waited for serial Git probes of every saved project, so one slow repository held back the whole list after each Runner start.

The one owned scan worker now publishes in two stages through the existing acknowledged inventory pages:

  1. The validated registration catalog, with git_branch, git_head and git_dirty absent.
  2. The same catalog with Git details, as a new inventory generation that replaces the first.

How it meets the acceptance points

Point from the issue Implementation
Current, validated facts first The metadata stage uses the same summary code as the full scan with Git probes off, so path resolution, root_fingerprint, revision and disabled are unchanged. An unresolved root still has no fingerprint.
Missing Git fields mean unknown The fields are absent, not defaulted. Tests assert None, including git_dirty.
Reuse generation/ACK and scan ownership Both stages use ProjectInventorySync. InitialProjectScanState.invalidated becomes a generation counter.
Delayed enrichment can't overwrite newer state Each publication records the generation it was read under. The worker discards a stale result under the same lock as invalidation, and streams skip a buffered one.
Add/remove during a scan A mutation during enrichment republishes the catalog immediately, with no Git probe, and the worker restarts.
One bounded, cancellable flow Still one worker per process, with the existing shutdown flag and owned Git trees.

If enrichment ends without a final result after the catalog was published, the session keeps the catalog, with Git details left unknown, instead of failing.

Tests

New in transport_tests/initial_project_scan.rs, using a held enrichment as the slow repository:

  • initial_inventory_publishes_catalog_before_git_enrichment_then_replaces_it
  • mutation_during_git_enrichment_republishes_catalog_and_discards_stale_result
  • reconnect_during_git_enrichment_gets_catalog_and_shares_the_one_worker
  • shutdown_during_git_enrichment_ends_the_session_without_a_scan_error
  • project_metadata_keeps_authority_facts_and_leaves_git_unknown, against a real Git repository and an unresolved root

The existing initial-scan tests are adapted to the two-stage seam. Their hold now sits in the metadata stage, so they still cover a mutation before first publication.

Removing or breaking each of these made at least one test fail: the metadata publication, the worker's stale check, the stream's generation filter, the immediate correction, worker reuse during enrichment, Git-free metadata, and completion handling.

Run on macOS arm64:

  • cargo test -p webcodex-runner --bin webcodex-runner: 1230 passed, 1 failed. The failure is coding_agent::tests::diagnostics::explicit_path_mapping_restores_an_interpreter_outside_default_search_paths, which fails the same way on this machine without this change.
  • The 12 initial-scan tests repeated 25 times with no failure.
  • cargo clippy --locked -p webcodex-runner --bins --tests -- -D clippy::correctness, cargo fmt --check, cargo check --locked -p webcodex-runner --bins --profile dogfood.
  • python3 scripts/check_markdown_links.py, git diff --check.

Limits

  • Enrichment stays serial and publishes once at the end: a slow repository no longer delays the catalog, but it can still delay other projects' Git details.
  • The metadata stage still resolves each project path, so a path that blocks the filesystem call itself can still hold the first publication.
  • Polling transports are unchanged.
  • Tested over loopback WebSocket fixtures, not against a real Server with a large catalog.
  • docs/RUNNER.md is updated. docs/RUNNER.zh-CN.md has no peer paragraph for this behavior, so I left it alone.

The first inventory waited for serial Git probes of every saved project,
so one slow repository delayed the whole list after each Runner start.

The one owned scan worker now publishes the validated catalog first,
with Git fields absent to mean unknown, then replaces it once enrichment
completes. Each publication records the inventory generation it was read
under, so a result from before a Project mutation is never published,
and a mutation during enrichment republishes the catalog immediately.

Fixes yyjeqhc#1271

@swjnxyf swjnxyf left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed PR #1415 on the special main repository at 3ec817986b1b419d4d448b7f80a0ea8b80633e95 without creating a worktree. No blocking issues found (LGTM).

I checked the metadata-first and Git-enriched publications, the generation fencing of stale worker results, mutation-triggered metadata resnapshot, reconnect/reuse of the owned scan worker, shutdown handling, and the acknowledged paged inventory sync. The initial catalog preserves registration authority fields while leaving git_branch, git_head and git_dirty genuinely unknown until enrichment. The fallback for an unavailable enrichment stage keeps the already-published catalog rather than failing the stream.

Validation: GitHub's full CI matrix has passed (Linux, Windows, macOS and native aggregate gates). On special, cargo test --locked -p webcodex-runner --profile dogfood initial_project_scan -- --test-threads=4 passed (7 tests), the initial_inventory test passed, the Transport suite passed (112 tests), the four Git-enrichment tests passed, and the metadata authority test passed. cargo fmt --all -- --check and git diff --check also passed.

Non-blocking limitation: enrichment remains a serial final pass, so a slow repository can still delay the Git details of other projects, as documented in the PR. No follow-up code changes needed.

@swjnxyf
swjnxyf merged commit 24148a5 into yyjeqhc:main Nov 10, 2026
18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Runner start waits for Git enrichment of every project before publishing the inventory

2 participants