Skip to content

feat: require a warehouse for every execution while the feature is enabled - #7752

Draft
mengw15 wants to merge 7 commits into
apache:mainfrom
mengw15:feat/7751-require-warehouse
Draft

feat: require a warehouse for every execution while the feature is enabled#7752
mengw15 wants to merge 7 commits into
apache:mainfrom
mengw15:feat/7751-require-warehouse

Conversation

@mengw15

@mengw15 mengw15 commented Aug 18, 2026

Copy link
Copy Markdown
Contributor

Stacked PR — based on #7536 (feat/6933-warehouse-frontend), not on main. That branch adds the warehouse picker this change reads from, so the diff and file list here include its commits; only the top commit belongs to this PR. Please review/merge #7536 first — this is a draft until it lands, after which the diff collapses to just the change described below.

What changes were proposed in this PR?

With per-user warehouses enabled, an execution carrying no warehouseId silently wrote into the shared default warehouse: WorkflowService.resolveWarehouseName mapped None to None whenever the flag was on. So "a run writes into the user's own warehouse" was a UI convention rather than a system property — and one path already violated it: SyncExecutionResource hardcoded warehouseId = None, so on a deployment with the feature on, every agent-driven execution landed in the shared warehouse.

The fix makes the pick required the same way a computing unit already is, and carries it along the same path the computing unit already travels:

  • WorkflowService.resolveWarehouseName rejects instead of falling back: with the flag on and nothing picked, the execution fails rather than being routed into shared storage. The websocket path needs nothing more — the UI always sends its pick.
  • SyncExecutionRequest gains warehouseId and the endpoint forwards it instead of hardcoding None.
  • agent-service threads warehouseId through ExecutionConfig and the delegate config into that request body, mirroring computingUnitId line for line.
  • The frontend sends the warehouse the user picked when creating an agent, right where it already sends the selected computing unit (agent.service.ts).

This is not a privilege change: an explicit whid was, and still is, checked against the caller's uid. Flag-off behaviour is unchanged — an explicit pick is still refused loudly (#6930), and no pick still means the shared warehouse.

Worth settling while the flag is off everywhere: tightening this later, once a deployment runs with the lenient fallback, would break callers that work today.

Any related issues, documentation, discussions?

Closes #7751. Part of #6870, follow-up to #6932. Stacked on #7536 (warehouse picker).

How was this PR tested?

  • Backend: WorkflowServiceWarehouseSpec's contract case flips from "no pick keeps the shared warehouse" to "no pick is refused while enabled", with the flag-off fallback kept as its own case. WorkflowServiceWarehouseSpec + SyncExecutionResourceSpec run locally: 38/38 passed; WorkflowExecutionService/scalafmtCheck (main + Test) passes.
  • agent-service: tsc --noEmit passes and bun test workflow-execution-tools.spec.ts is 23 pass / 0 fail.
  • Frontend: agent.service.spec.ts gains a WarehouseService stub mirroring the existing computing-unit stub, asserting the payload carries the selected warehouseId and omits it when nothing is selected; ng test --include=... passes.

Was this PR authored or co-authored using generative AI tooling?

Generated-by: Claude Code (claude-opus-4-8)

mengw15 and others added 7 commits August 10, 2026 22:14
Per-user warehouses enabled now means every execution writes to one:
the Run button leads to the create-warehouse modal when none is selected,
mirroring the computing-unit Connect flow, and the picker stays visible
with zero warehouses so the create path is reachable. The picker carries
the caller's avatar like the CU picker, warehouse names open a read-only
details modal, and the warehouse icon is hdd everywhere (database was
already taken by Datasets).
…abled

With per-user warehouses enabled, an execution carrying no warehouseId
silently wrote into the shared default warehouse: resolveWarehouseName
mapped None to None whenever the flag was on. So "a run writes into the
user's own warehouse" was a UI convention rather than a system property,
and one path already violated it -- SyncExecutionResource hardcoded
warehouseId = None, so on a deployment with the feature on every
agent-driven execution landed in the shared warehouse.

Make the pick required the same way a computing unit is, and carry it
along the same path the computing unit already travels:

  - WorkflowService.resolveWarehouseName fails instead of falling back
    when the flag is on and nothing was picked. The websocket path needs
    nothing more -- the UI always sends its pick.
  - SyncExecutionRequest gains warehouseId and the endpoint forwards it
    instead of hardcoding None.
  - agent-service threads warehouseId through ExecutionConfig and the
    delegate config into that request body, mirroring computingUnitId.
  - The frontend sends the warehouse the user picked when creating an
    agent, right where it already sends the selected computing unit.

Flag-off behaviour is unchanged: an explicit pick is still refused loudly
(apache#6930) and no pick still means the shared warehouse.

Part of apache#6870. Closes apache#7751.
@github-actions

Copy link
Copy Markdown
Contributor

Automated Reviewer Suggestions

Based on the git blame history of the changed files, we recommend the following reviewers:

  • Contributors with relevant context: @Neilk1021, @aglinxinyuan, @bobbai00
    You can notify them by mentioning @Neilk1021, @aglinxinyuan, @bobbai00 in a comment.

@github-actions github-actions Bot added feature engine frontend Changes related to the frontend GUI agent-service labels Aug 18, 2026
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 96.33333% with 11 lines in your changes missing coverage. Please review.
✅ Project coverage is 90.94%. Comparing base (12169c2) to head (2727aa5).
⚠️ Report is 43 commits behind head on main.
✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
...create-modal/warehouse-create-modal.component.html 77.77% 4 Missing ⚠️
.../user/user-warehouse/user-warehouse.component.html 89.28% 3 Missing ⚠️
...wer-button/computing-unit-selection.component.html 92.68% 3 Missing ⚠️
...power-button/computing-unit-selection.component.ts 97.77% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##               main    #7752      +/-   ##
============================================
+ Coverage     89.65%   90.94%   +1.28%     
- Complexity     4397     4448      +51     
============================================
  Files          1177     1183       +6     
  Lines         46996    47428     +432     
  Branches       5268     5306      +38     
============================================
+ Hits          42136    43133     +997     
+ Misses         3094     2616     -478     
+ Partials       1766     1679      -87     
Flag Coverage Δ *Carryforward flag
access-control-service 70.00% <ø> (ø) Carriedforward from 94c35f4
agent-service 98.62% <ø> (-0.01%) ⬇️ Carriedforward from 94c35f4
amber 87.46% <100.00%> (+1.10%) ⬆️
computing-unit-managing-service 72.46% <ø> (ø) Carriedforward from 94c35f4
config-service 77.31% <ø> (ø) Carriedforward from 94c35f4
file-service 68.90% <ø> (ø) Carriedforward from 94c35f4
frontend 92.64% <96.25%> (+1.97%) ⬆️
notebook-migration-service 78.89% <ø> (ø) Carriedforward from 94c35f4
pyamber 97.57% <ø> (ø) Carriedforward from 94c35f4
workflow-compiling-service 57.89% <ø> (ø) Carriedforward from 94c35f4

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions

Copy link
Copy Markdown
Contributor

⚠️ Benchmark changes need a look

🟢 0 better · 🔴 7 worse · ⚪ 8 noise (<±5%) · 0 without baseline

Compared against main 60300e3 benchmarked on this same runner, so the delta is largely free of cross-runner hardware noise. The "7d avg" column still reflects the gh-pages dashboard. Treat <±5% as noise unless repeated.

Dashboard · Run

config throughput MB/s latency max Δ latest / 7d
🔴 bs=10 sw=10 sl=64 383 0.234 25,016/36,384/36,384 us 🔴 +20.1% / 🔴 +127.6%
🔴 bs=100 sw=10 sl=64 870 0.531 108,428/176,474/176,474 us 🔴 +47.0% / 🔴 +64.5%
bs=1000 sw=10 sl=64 1,068 0.652 933,296/1,024,407/1,024,407 us ⚪ within ±5% / ⚪ within ±5%
Baseline details

Latest main 60300e3 from same runner

config metric PR latest main 7d avg Δ latest Δ 7d
bs=10 sw=10 sl=64 throughput 383 tuples/sec 431 tuples/sec 779.07 tuples/sec -11.1% -50.8%
bs=10 sw=10 sl=64 MB/s 0.234 MB/s 0.263 MB/s 0.476 MB/s -11.0% -50.8%
bs=10 sw=10 sl=64 p50 25,016 us 20,833 us 12,818 us +20.1% +95.2%
bs=10 sw=10 sl=64 p95 36,384 us 35,917 us 15,986 us +1.3% +127.6%
bs=10 sw=10 sl=64 p99 36,384 us 35,917 us 19,339 us +1.3% +88.1%
bs=100 sw=10 sl=64 throughput 870 tuples/sec 966 tuples/sec 1,011 tuples/sec -9.9% -14.0%
bs=100 sw=10 sl=64 MB/s 0.531 MB/s 0.59 MB/s 0.617 MB/s -10.0% -14.0%
bs=100 sw=10 sl=64 p50 108,428 us 104,259 us 100,965 us +4.0% +7.4%
bs=100 sw=10 sl=64 p95 176,474 us 120,021 us 107,295 us +47.0% +64.5%
bs=100 sw=10 sl=64 p99 176,474 us 120,021 us 115,531 us +47.0% +52.8%
bs=1000 sw=10 sl=64 throughput 1,068 tuples/sec 1,082 tuples/sec 1,049 tuples/sec -1.3% +1.8%
bs=1000 sw=10 sl=64 MB/s 0.652 MB/s 0.66 MB/s 0.64 MB/s -1.2% +1.9%
bs=1000 sw=10 sl=64 p50 933,296 us 912,902 us 978,248 us +2.2% -4.6%
bs=1000 sw=10 sl=64 p95 1,024,407 us 1,045,784 us 1,021,881 us -2.0% +0.2%
bs=1000 sw=10 sl=64 p99 1,024,407 us 1,045,784 us 1,050,075 us -2.0% -2.4%
Raw CSV
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,521.78,200,128000,383,0.234,25016.47,36383.94,36383.94
1,100,10,64,20,2299.01,2000,1280000,870,0.531,108428.17,176473.83,176473.83
2,1000,10,64,20,18728.94,20000,12800000,1068,0.652,933295.95,1024407.38,1024407.38

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agent-service engine feature frontend Changes related to the frontend GUI

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BYO-S3] Require a warehouse for every execution while the feature is enabled

2 participants