Skip to content

test(pagebroker): add k3s smoke and Qwen restore validation - #564

Closed
tzachi-runai wants to merge 9 commits into
dyn-4108/pagebroker-integration-testsfrom
dyn-4108/pagebroker-k3s-smoke
Closed

tzachi-runai wants to merge 9 commits into
dyn-4108/pagebroker-integration-testsfrom
dyn-4108/pagebroker-k3s-smoke

Conversation

@tzachi-runai

@tzachi-runai tzachi-runai commented Oct 13, 2026 •

Copy link
Copy Markdown
Collaborator

Adds a k3s smoke harness for direct PageBroker filesystem/S3 client operations and opt-in real Qwen GPU checkpoint/restore. The orchestrator deploys isolated synthetic Jobs, runs Qwen validation against an installed Snapshot release, and collects independently validated evidence.

Filesystem synthetic transfers use POSIX: three publications and two restores per publication, with 11 reported steps. S3 uses Model Streamer: three publications and 15 steps covering publication, metadata, repeated restore, prepared abort, and wrong-store rejection. Payload contents, sizes, permissions, bounded cleanup, and staging removal are checked.

Each synthetic Job runs a dedicated broker and client with a shared Unix socket, PID namespace and tmpfs staging. A pidfd identifies the exact daemon. Reports and resource samples survive process termination and are retrieved through a bounded read-only exporter. Credentials are restricted to the S3 components and an owned object prefix.

The opt-in Qwen cases use staged POSIX filesystem restores; the removed filesystem Model Streamer selector is rejected. Rendered native tuning contains shared and S3 settings only. Resource reports retain memory, descriptor, thread, and staging measurements; they do not infer a leak verdict.

Based on #715 (integration coverage after #469); #622 adds repeated endurance validation. Refs #267.

Validation: make check, make test, and make build pass at this PR head. The propagated smoke/endurance helper suite covers all 190 tests (the Kubernetes-model test was also run separately with its dependencies); 28 Qwen helper tests pass. Current changes were not executed on a physical GPU or k3s cluster.

@copy-pr-bot

copy-pr-bot Bot commented Oct 13, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@coderabbitai

coderabbitai Bot commented Oct 13, 2026

Copy link
Copy Markdown
Contributor

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

@tzachi-runai
tzachi-runai force-pushed the dyn-4108/pagebroker-k3s-smoke branch from 556a524 to d66884f Compare October 13, 2026 17:55
@tzachi-runai tzachi-runai changed the title test(pagebroker): add direct client for k3s smoke validation test(pagebroker): add client and jobs for k3s smoke validation Oct 13, 2026
@tzachi-runai
tzachi-runai force-pushed the dyn-4108/pagebroker-k3s-smoke branch 2 times, most recently from d8ca5fd to 3c8bd31 Compare October 13, 2026 19:49
@tzachi-runai tzachi-runai changed the title test(pagebroker): add client and jobs for k3s smoke validation test(pagebroker): add k3s smoke and Qwen restore validation Oct 13, 2026
@tzachi-runai
tzachi-runai force-pushed the dyn-4108/pagebroker-streamer-config branch from 942c41f to e73583f Compare October 13, 2026 20:02
@tzachi-runai
tzachi-runai force-pushed the dyn-4108/pagebroker-k3s-smoke branch from 3c8bd31 to 1e1e6d6 Compare October 14, 2026 05:59
@tzachi-runai
tzachi-runai force-pushed the dyn-4108/pagebroker-streamer-config branch 4 times, most recently from 7c664c8 to b89d50d Compare October 14, 2026 12:35
@tzachi-runai
tzachi-runai force-pushed the dyn-4108/pagebroker-k3s-smoke branch from 1e1e6d6 to 18ecd5a Compare October 14, 2026 13:21
@tzachi-runai
tzachi-runai force-pushed the dyn-4108/pagebroker-streamer-config branch from b89d50d to 598dc51 Compare October 14, 2026 13:45
@tzachi-runai
tzachi-runai force-pushed the dyn-4108/pagebroker-k3s-smoke branch from 18ecd5a to 8d33c6c Compare October 14, 2026 13:52
@tzachi-runai
tzachi-runai force-pushed the dyn-4108/pagebroker-streamer-config branch 2 times, most recently from 924bd9c to 3a37d3b Compare October 14, 2026 15:47
@tzachi-runai
tzachi-runai force-pushed the dyn-4108/pagebroker-k3s-smoke branch from 8d33c6c to 8c97592 Compare October 15, 2026 13:00
@tzachi-runai
tzachi-runai force-pushed the dyn-4108/pagebroker-streamer-config branch from 3a37d3b to 06f3958 Compare October 15, 2026 13:34
@tzachi-runai
tzachi-runai force-pushed the dyn-4108/pagebroker-k3s-smoke branch from 8c97592 to 4af859c Compare October 15, 2026 13:53
tzachi-runai pushed a commit that referenced this pull request Oct 19, 2026
Signed-off-by: Tzachi Lapidot <tlapdot@nvidia.com>
tzachi-runai pushed a commit that referenced this pull request Oct 19, 2026
Signed-off-by: Tzachi Lapidot <tlapdot@nvidia.com>
tzachi-runai pushed a commit that referenced this pull request Oct 20, 2026
Signed-off-by: Tzachi Lapidot <tlapdot@nvidia.com>
@tzachi-runai
tzachi-runai force-pushed the dyn-4108/pagebroker-k3s-smoke branch 2 times, most recently from 45ec8bb to fa63c46 Compare October 21, 2026 15:39
@tzachi-runai
tzachi-runai force-pushed the dyn-4108/pagebroker-streamer-config branch 2 times, most recently from 1136ebb to fb25f2a Compare October 21, 2026 17:08
@tzachi-runai
tzachi-runai force-pushed the dyn-4108/pagebroker-k3s-smoke branch from fa63c46 to c87155a Compare October 21, 2026 17:08
@tzachi-runai
tzachi-runai force-pushed the dyn-4108/pagebroker-streamer-config branch from fb25f2a to 24f412e Compare October 22, 2026 08:08
@tzachi-runai
tzachi-runai force-pushed the dyn-4108/pagebroker-k3s-smoke branch from c87155a to e23c2ea Compare October 22, 2026 08:08
@tzachi-runai
tzachi-runai force-pushed the dyn-4108/pagebroker-streamer-config branch from 24f412e to b4a4b55 Compare October 22, 2026 08:17
@tzachi-runai
tzachi-runai force-pushed the dyn-4108/pagebroker-k3s-smoke branch 2 times, most recently from a4c8d32 to f7baf06 Compare October 22, 2026 08:18
@tzachi-runai
tzachi-runai force-pushed the dyn-4108/pagebroker-streamer-config branch 2 times, most recently from 20b4cc4 to 9363e1f Compare October 22, 2026 12:58
@tzachi-runai
tzachi-runai force-pushed the dyn-4108/pagebroker-k3s-smoke branch from e918989 to 6bc2436 Compare October 22, 2026 12:58
@tzachi-runai
tzachi-runai force-pushed the dyn-4108/pagebroker-streamer-config branch from f57360a to c45833d Compare October 22, 2026 15:35
@tzachi-runai
tzachi-runai force-pushed the dyn-4108/pagebroker-k3s-smoke branch from 52cfbdb to 69b5e27 Compare October 22, 2026 15:35
@tzachi-runai
tzachi-runai force-pushed the dyn-4108/pagebroker-streamer-config branch from 4afdb9a to ced4394 Compare October 25, 2026 09:00
@tzachi-runai
tzachi-runai force-pushed the dyn-4108/pagebroker-k3s-smoke branch from 69b5e27 to 1bb0e76 Compare October 25, 2026 09:00
@tzachi-runai
tzachi-runai changed the base branch from dyn-4108/pagebroker-streamer-config to dyn-4108/pagebroker-integration-tests October 25, 2026 15:04
Tzachi Lapidot added 6 commits October 26, 2026 11:36
Signed-off-by: Tzachi Lapidot <tlapdot@nvidia.com>
Signed-off-by: Tzachi Lapidot <tlapdot@nvidia.com>
Signed-off-by: Tzachi Lapidot <tlapdot@nvidia.com>
Signed-off-by: Tzachi Lapidot <tlapdot@nvidia.com>
Signed-off-by: Tzachi Lapidot <tlapdot@nvidia.com>
Signed-off-by: Tzachi Lapidot <tlapdot@nvidia.com>
@tzachi-runai
tzachi-runai force-pushed the dyn-4108/pagebroker-k3s-smoke branch from 87d6ac1 to fccf369 Compare October 26, 2026 09:38
@tzachi-runai
tzachi-runai force-pushed the dyn-4108/pagebroker-integration-tests branch from e26affa to 6b57129 Compare October 26, 2026 09:38
Tzachi Lapidot added 3 commits October 26, 2026 12:40
Signed-off-by: Tzachi Lapidot <tlapdot@nvidia.com>
Signed-off-by: Tzachi Lapidot <tlapdot@nvidia.com>
Signed-off-by: Tzachi Lapidot <tlapdot@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant