composite sqlite-elasticsearch: CompartmentDefinitions never reach the ES index — /CompartmentDefinition serves empty and the UI caches the emptiness

[angela-helios]

Surfaced by the ui-tests-matrix validation run for #449 (30582018605) once the Docker-host quota fixes (#452/#454) stopped masking everything else: the sqlite-elasticsearch leg fails 18 browser tests in one cluster — the compartments page, the resources type rail, the builder's parameter datalist, and their a11y/nojs variants.

Evidence (from the workflow's conformance probe, 15s after readiness)

GET /CompartmentDefinition?_count=10000 → {"resourceType":"Bundle","type":"searchset","total":null,"entry":[]}
ES _cat/indices → hfs_default_searchparameter  7404 docs      ← SPs fully synced
ES hfs_default_compartmentdefinition/_count → index_not_found_exception

The suite then ran for 21 more minutes and the compartments tests still failed at the end — this is not sync lag; the CompartmentDefinition documents never reach ES while SearchParameters do. On the composite, search is served by ES, so GET /CompartmentDefinition is empty for the life of the process.

Curiously the s3-elasticsearch leg passed the same tests in the same run (94/1, the 1 being #442) — whatever orders seeding vs. the sync worker differs between the two composites, which smells like a startup-ordering race that sqlite (fast seeding) loses and S3 (slow seeding) wins.

Second facet: the UI caches the emptiness

CompartmentCatalog/SpCatalog only refuse to cache failed fetches; an empty 200 is cached for the process lifetime, so even after ES catches up the pages stay broken until restart. The cache should treat empty-when-the-primary-is-known-to-seed as provisional (or simply not cache empty sets), independent of the server-side fix.

Found while validating #449; the nightly reds attributed to the Docker host were hiding this.

[angela-helios]

Investigation status

Ruled out:

• The generic composite/sync plumbing: a new differential test (composite with a second SQLite as the search secondary, async workers on, real seed_tenant_conformance) drains both SPs and CDs to the secondary. Green.
• Seeding itself: the failing run's hfs.log shows Seeded spec CompartmentDefinitions into storage created=5 failed=0 for the default tenant and every e2e-provisioned tenant.
• Sync errors: zero Async sync failed and zero Failed to sync create to secondaries lines in the whole log.
• ES sync being generally broken: hfs_default_patient exists in ES later in the run (the suite's created patients got there through the same worker).

New leads from the same log:

• ES itself was struggling: no_shard_available_action_exception / all shards failed 503s on hfs_default_patient mid-run — a 512MB single-node under real load. A CD index whose single shard failed to allocate could present as missing while nothing on the HFS side logs an error.
• The SP index held 7,404 docs 17s after startup, when only 1,377 SPs had been seeded — something besides the seeding writes SP docs (multiple tenants had not been provisioned yet at that point). Worth identifying; it may also explain why SPs 'survive' whatever eats the CDs.

Suggested next steps (need a live-ES session or one instrumented matrix run):

1. Bump the matrix ES heap (512m → 1g) and re-run — if the CD index appears, this was ES shard pressure, not HFS.
2. Log per-type sync outcomes in the async worker (type + index + status) at info level for one run.
3. _cat/indices?v&health in the workflow probe (already added in ci(ui-matrix): capture evidence when HFS cannot reach Elasticsearch #451's diagnostics on failure) to catch red/unallocated indices, not just missing ones.