feat: require a warehouse for every execution while the feature is enabled - #7752
feat: require a warehouse for every execution while the feature is enabled#7752mengw15 wants to merge 7 commits into
Conversation
Per-user warehouses enabled now means every execution writes to one: the Run button leads to the create-warehouse modal when none is selected, mirroring the computing-unit Connect flow, and the picker stays visible with zero warehouses so the create path is reachable. The picker carries the caller's avatar like the CU picker, warehouse names open a read-only details modal, and the warehouse icon is hdd everywhere (database was already taken by Datasets).
…abled
With per-user warehouses enabled, an execution carrying no warehouseId
silently wrote into the shared default warehouse: resolveWarehouseName
mapped None to None whenever the flag was on. So "a run writes into the
user's own warehouse" was a UI convention rather than a system property,
and one path already violated it -- SyncExecutionResource hardcoded
warehouseId = None, so on a deployment with the feature on every
agent-driven execution landed in the shared warehouse.
Make the pick required the same way a computing unit is, and carry it
along the same path the computing unit already travels:
- WorkflowService.resolveWarehouseName fails instead of falling back
when the flag is on and nothing was picked. The websocket path needs
nothing more -- the UI always sends its pick.
- SyncExecutionRequest gains warehouseId and the endpoint forwards it
instead of hardcoding None.
- agent-service threads warehouseId through ExecutionConfig and the
delegate config into that request body, mirroring computingUnitId.
- The frontend sends the warehouse the user picked when creating an
agent, right where it already sends the selected computing unit.
Flag-off behaviour is unchanged: an explicit pick is still refused loudly
(apache#6930) and no pick still means the shared warehouse.
Part of apache#6870. Closes apache#7751.
Automated Reviewer SuggestionsBased on the
|
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #7752 +/- ##
============================================
+ Coverage 89.65% 90.94% +1.28%
- Complexity 4397 4448 +51
============================================
Files 1177 1183 +6
Lines 46996 47428 +432
Branches 5268 5306 +38
============================================
+ Hits 42136 43133 +997
+ Misses 3094 2616 -478
+ Partials 1766 1679 -87
*This pull request uses carry forward flags. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
| config | throughput | MB/s | latency | max Δ latest / 7d | |
|---|---|---|---|---|---|
| 🔴 | bs=10 sw=10 sl=64 | 383 | 0.234 | 25,016/36,384/36,384 us | 🔴 +20.1% / 🔴 +127.6% |
| 🔴 | bs=100 sw=10 sl=64 | 870 | 0.531 | 108,428/176,474/176,474 us | 🔴 +47.0% / 🔴 +64.5% |
| ⚪ | bs=1000 sw=10 sl=64 | 1,068 | 0.652 | 933,296/1,024,407/1,024,407 us | ⚪ within ±5% / ⚪ within ±5% |
Baseline details
Latest main 60300e3 from same runner
| config | metric | PR | latest main | 7d avg | Δ latest | Δ 7d |
|---|---|---|---|---|---|---|
| bs=10 sw=10 sl=64 | throughput | 383 tuples/sec | 431 tuples/sec | 779.07 tuples/sec | -11.1% | -50.8% |
| bs=10 sw=10 sl=64 | MB/s | 0.234 MB/s | 0.263 MB/s | 0.476 MB/s | -11.0% | -50.8% |
| bs=10 sw=10 sl=64 | p50 | 25,016 us | 20,833 us | 12,818 us | +20.1% | +95.2% |
| bs=10 sw=10 sl=64 | p95 | 36,384 us | 35,917 us | 15,986 us | +1.3% | +127.6% |
| bs=10 sw=10 sl=64 | p99 | 36,384 us | 35,917 us | 19,339 us | +1.3% | +88.1% |
| bs=100 sw=10 sl=64 | throughput | 870 tuples/sec | 966 tuples/sec | 1,011 tuples/sec | -9.9% | -14.0% |
| bs=100 sw=10 sl=64 | MB/s | 0.531 MB/s | 0.59 MB/s | 0.617 MB/s | -10.0% | -14.0% |
| bs=100 sw=10 sl=64 | p50 | 108,428 us | 104,259 us | 100,965 us | +4.0% | +7.4% |
| bs=100 sw=10 sl=64 | p95 | 176,474 us | 120,021 us | 107,295 us | +47.0% | +64.5% |
| bs=100 sw=10 sl=64 | p99 | 176,474 us | 120,021 us | 115,531 us | +47.0% | +52.8% |
| bs=1000 sw=10 sl=64 | throughput | 1,068 tuples/sec | 1,082 tuples/sec | 1,049 tuples/sec | -1.3% | +1.8% |
| bs=1000 sw=10 sl=64 | MB/s | 0.652 MB/s | 0.66 MB/s | 0.64 MB/s | -1.2% | +1.9% |
| bs=1000 sw=10 sl=64 | p50 | 933,296 us | 912,902 us | 978,248 us | +2.2% | -4.6% |
| bs=1000 sw=10 sl=64 | p95 | 1,024,407 us | 1,045,784 us | 1,021,881 us | -2.0% | +0.2% |
| bs=1000 sw=10 sl=64 | p99 | 1,024,407 us | 1,045,784 us | 1,050,075 us | -2.0% | -2.4% |
Raw CSV
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,521.78,200,128000,383,0.234,25016.47,36383.94,36383.94
1,100,10,64,20,2299.01,2000,1280000,870,0.531,108428.17,176473.83,176473.83
2,1000,10,64,20,18728.94,20000,12800000,1068,0.652,933295.95,1024407.38,1024407.38
What changes were proposed in this PR?
With per-user warehouses enabled, an execution carrying no
warehouseIdsilently wrote into the shared default warehouse:WorkflowService.resolveWarehouseNamemappedNonetoNonewhenever the flag was on. So "a run writes into the user's own warehouse" was a UI convention rather than a system property — and one path already violated it:SyncExecutionResourcehardcodedwarehouseId = None, so on a deployment with the feature on, every agent-driven execution landed in the shared warehouse.The fix makes the pick required the same way a computing unit already is, and carries it along the same path the computing unit already travels:
WorkflowService.resolveWarehouseNamerejects instead of falling back: with the flag on and nothing picked, the execution fails rather than being routed into shared storage. The websocket path needs nothing more — the UI always sends its pick.SyncExecutionRequestgainswarehouseIdand the endpoint forwards it instead of hardcodingNone.agent-servicethreadswarehouseIdthroughExecutionConfigand the delegate config into that request body, mirroringcomputingUnitIdline for line.agent.service.ts).This is not a privilege change: an explicit
whidwas, and still is, checked against the caller'suid. Flag-off behaviour is unchanged — an explicit pick is still refused loudly (#6930), and no pick still means the shared warehouse.Worth settling while the flag is off everywhere: tightening this later, once a deployment runs with the lenient fallback, would break callers that work today.
Any related issues, documentation, discussions?
Closes #7751. Part of #6870, follow-up to #6932. Stacked on #7536 (warehouse picker).
How was this PR tested?
WorkflowServiceWarehouseSpec's contract case flips from "no pick keeps the shared warehouse" to "no pick is refused while enabled", with the flag-off fallback kept as its own case.WorkflowServiceWarehouseSpec+SyncExecutionResourceSpecrun locally: 38/38 passed;WorkflowExecutionService/scalafmtCheck(main + Test) passes.tsc --noEmitpasses andbun test workflow-execution-tools.spec.tsis 23 pass / 0 fail.agent.service.spec.tsgains aWarehouseServicestub mirroring the existing computing-unit stub, asserting the payload carries the selectedwarehouseIdand omits it when nothing is selected;ng test --include=...passes.Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Code (claude-opus-4-8)