From d83b15219b0a1e0d40c744f2d251f4571b013277 Mon Sep 17 00:00:00 2001 From: mosheabr <257371078+mosheabr@users.noreply.github.com> Date: Sun, 23 Aug 2026 02:32:26 +0000 Subject: [PATCH] chore: sync skills (NeMo MBridge,NeMo Relay) --- .../BENCHMARK.md | 18 +++---- .../SKILL.md | 48 ++++++++++++------ .../card.yaml | 1 + .../skill-card.md | 35 ++++++------- .../skill.oms.sig | 2 +- skills/nemo-relay-get-started/BENCHMARK.md | 31 ++++++++---- skills/nemo-relay-get-started/SKILL.md | 2 + skills/nemo-relay-get-started/skill-card.md | 49 +++++++++---------- skills/nemo-relay-get-started/skill.oms.sig | 2 +- 9 files changed, 108 insertions(+), 80 deletions(-) diff --git a/skills/nemo-mbridge-perf-sequence-packing/BENCHMARK.md b/skills/nemo-mbridge-perf-sequence-packing/BENCHMARK.md index fd76567f..6b3edd9f 100644 --- a/skills/nemo-mbridge-perf-sequence-packing/BENCHMARK.md +++ b/skills/nemo-mbridge-perf-sequence-packing/BENCHMARK.md @@ -9,16 +9,16 @@ Recommended for publication based on the completed evaluation evidence in this r ## Evaluation Metadata - Skill: `nemo-mbridge-perf-sequence-packing` -- Evaluation date: 2026-08-11 -- Evaluator version: `1.2.0` +- Evaluation date: 2026-08-17 +- Evaluator version: `1.2.7` - Agents: Claude Code (`aws/anthropic/bedrock-claude-opus-4-8`), Codex (`openai/openai/gpt-5.5`) - Tasks: 1 evaluation tasks (1 positive) - Dataset digest: `sha256:57d3c088dee48ee97547a666ee24dcac45c6cd5cbf699640c3c7d41710f629a1` (skill-evaluator-dataset-snapshot/1) - Attempts per task: 1 -- Environment: `k8s-sandbox` +- Environment: `local` - Tier 3 evidence: required for publication -Each task attempt ran in its own isolated sandbox pod. +Tasks ran on the trusted local host; local mode is not sandboxed. ## What This Report Answers @@ -34,12 +34,12 @@ The three-tier evaluation checks whether the skill: | Measure | Claude Code (Baseline → Skill Uplift) | Codex (Baseline → Skill Uplift) | |---|---:|---:| -| Overall | 37% → 91% (+54 points) | 68% → 86% (+18 points) | +| Overall | 60% → 98% (+38 points) | 56% → 96% (+40 points) | | Security | 100% → 100% (±0 points) | 100% → 100% (±0 points) | -| Correctness | 0% → 100% (+100 points) | 100% → 100% (±0 points) | -| Discoverability | 50% → 100% (+50 points) | 50% → 88% (+38 points) | -| Effectiveness | 0% → 66% (+66 points) | 74% → 45% (-29 points) | -| Efficiency | 35% → 90% (+55 points) | 17% → 100% (+83 points) | +| Correctness | 0% → 100% (+100 points) | 40% → 100% (+60 points) | +| Discoverability | 100% → 100% (±0 points) | 50% → 88% (+38 points) | +| Effectiveness | 0% → 88% (+88 points) | 40% → 94% (+54 points) | +| Efficiency | 100% → 100% (±0 points) | 50% → 100% (+50 points) | **How to read this table:** baseline is the same task attempted without the target skill. Uplift is `skill score - baseline score`, shown in percentage points. diff --git a/skills/nemo-mbridge-perf-sequence-packing/SKILL.md b/skills/nemo-mbridge-perf-sequence-packing/SKILL.md index c1f8f61c..2c57721f 100644 --- a/skills/nemo-mbridge-perf-sequence-packing/SKILL.md +++ b/skills/nemo-mbridge-perf-sequence-packing/SKILL.md @@ -104,13 +104,21 @@ the packed dataset (asserted in `src/megatron/bridge/data/datasets/sft.py`). Custom packed datasets that omit the metadata file will hit an assertion at dataset initialization. -In-batch packing for VLM finetuning: +In-batch packing for GPT SFT and supported VLM finetuning: ```python cfg.dataset.enable_in_batch_packing = True -cfg.train.micro_batch_size = 2 +cfg.dataset.dataloader_type = "single" +cfg.train.micro_batch_size = 4 ``` +For local or materialized GPT-SFT JSONL, this keeps the existing mmap-backed +dataset and performs tokenization lazily. Both prompt/completion +(`GPTSFTDataset`) and chat (`GPTSFTChatDataset`) preserve their loss-mask +semantics. Use `dataloader_type="single"` or `"cyclic"` so every DataLoader +yield is one logical microbatch; GPT-SFT in-batch packing does not support the +global-batch `"batch"` dataloader. + Energon online packing for Qwen-VL uses Energon's per-worker candidate buffer instead of limiting selection to one collator micro batch: @@ -232,6 +240,14 @@ def prepare_padded_or_packed_sequence_batch( return ``` +GPT-SFT direct-row packing: + +```627:671:src/megatron/bridge/data/datasets/gpt_sft.py +def _collate_in_batch(self, batch): + ... + return build_mcore_thd_sequence_batch_from_rows(...) +``` + Packed THD runtime constraint: ```94:108:src/megatron/bridge/training/gpt_step.py @@ -250,18 +266,19 @@ if cu_seqlens.dim() > 1 and cu_seqlens.size(0) != 1: ## Pitfalls -1. Offline packed SFT, collate-time VLM packing, and Energon online packing are different features. Offline and Energon packing use physical MBS1; collate-time packing uses MBS greater than one. -2. When CP is enabled, packed sequence lengths must respect `2 * context_parallel_size` divisibility. -3. For finetuning with CP, `calculate_per_token_loss=True` and `ddp.average_in_collective=False` are required. -4. `pad_cu_seqlens=True` also requires `pad_to_max_length=True`. -5. Packing support is model-family-specific. `Qwen3-Next`, `GLM-4.5`, and `Qwen3.5-VL` contain explicit opt-outs in different paths. -6. MTP finetuning is documented as incompatible with packed sequences. -7. Synthetic padding rows, including negative indices remapped through `samples_mapping`, must retain an all-zero loss mask. -8. `global_batch_size` must be divisible by and no smaller than data parallel size when offline packing uses MBS1. -9. Derive `pad_seq_to_mult` from CP/TP/SP for both SFT and PEFT; do not hardcode different values by workload type. -10. `pad_to_max_length` controls final pack width and is conditional on fixed-shape execution requirements. -11. Energon `packing_buffer_size` is per worker and also affects validation; global/eval batch counts refer to physical packs rather than source conversations. -12. Exact Energon loader resume requires unchanged shards/splits, DP world size, worker counts, shuffle settings/seed, processor, sequence length, topology, and packing-buffer size. +1. Offline packed SFT, runtime in-batch packing, and Energon online packing are different features. Offline and Energon packing use physical MBS1; runtime in-batch packing uses MBS greater than one. +2. GPT-SFT in-batch packing requires `dataloader_type="single"` or `"cyclic"`; it does not support `"batch"`. +3. When CP is enabled, packed sequence lengths must respect `2 * context_parallel_size` divisibility. +4. For finetuning with CP, `calculate_per_token_loss=True` and `ddp.average_in_collective=False` are required. +5. `pad_cu_seqlens=True` also requires `pad_to_max_length=True`. +6. Packing support is model-family-specific. `Qwen3-Next`, `GLM-4.5`, and `Qwen3.5-VL` contain explicit opt-outs in different paths. +7. MTP finetuning is documented as incompatible with packed sequences. +8. Synthetic padding rows, including negative indices remapped through `samples_mapping`, must retain an all-zero loss mask. +9. `global_batch_size` must be divisible by and no smaller than data parallel size when offline packing uses MBS1. +10. Derive `pad_seq_to_mult` from CP/TP/SP for both SFT and PEFT; do not hardcode different values by workload type. +11. `pad_to_max_length` controls final pack width and is conditional on fixed-shape execution requirements. +12. Energon `packing_buffer_size` is per worker and also affects validation; global/eval batch counts refer to physical packs rather than source conversations. +13. Exact Energon loader resume requires unchanged shards/splits, DP world size, worker counts, shuffle settings/seed, processor, sequence length, topology, and packing-buffer size. ## Verification @@ -271,6 +288,8 @@ Use the checked-in unit coverage: uv run python -m pytest tests/unit_tests/training/utils/test_packed_seq_utils.py -v && \ uv run python -m pytest tests/unit_tests/training/test_config.py -k "packed_sequence or enable_in_batch_packing or offline_and_in_batch_packing_are_mutually_exclusive or context_parallel_seq_length_divisibility or context_parallel_finetuning_validations" -v && \ uv run python -m pytest tests/unit_tests/data/packing/test_in_batch.py -v && \ +uv run python -m pytest tests/unit_tests/data/datasets/test_gpt_sft.py -k "in_batch_packing" -v && \ +uv run python -m pytest tests/unit_tests/data/builders/test_gpt_sft_config.py -v && \ uv run python -m pytest tests/unit_tests/training/test_vlm_step.py -k "deferred_in_batch_packing or packed_metadata" -v && \ uv run python -m pytest tests/unit_tests/models/qwen_vl/data/test_energon.py tests/unit_tests/data/builders/test_energon_builder.py -v && \ uv run python -m pytest tests/unit_tests/tutorials/test_multimodal_data_tutorials.py -k "native_packing_loader" -v && \ @@ -283,5 +302,6 @@ Success criteria: - all selected tests pass - offline and in-batch configuration validation remains mutually exclusive - packed metadata reaches the training step in MCore THD form +- GPT-SFT in-batch packing rejects the global-batch `"batch"` dataloader - native Energon packing restores pending groups exactly and flushes finite partial buffers without dropping samples - mapped padding rows do not contribute to loss diff --git a/skills/nemo-mbridge-perf-sequence-packing/card.yaml b/skills/nemo-mbridge-perf-sequence-packing/card.yaml index e19fdc93..4956a591 100644 --- a/skills/nemo-mbridge-perf-sequence-packing/card.yaml +++ b/skills/nemo-mbridge-perf-sequence-packing/card.yaml @@ -79,6 +79,7 @@ known_constraints: - pad_seq_to_mult is derived from CP/TP/SP and follows the same rule for SFT and PEFT. - pad_to_max_length is conditional on a fixed-width dispatcher, kernel, or CUDA-graph path. - VLM in-batch packing requires micro_batch_size > 1. + - GPT-SFT in-batch packing requires dataloader_type single or cyclic; batch is unsupported. - Energon online packing requires physical micro_batch_size == 1 and counts candidate samples per worker. - Energon online packing is mutually exclusive with collate-time in-batch packing and the deferred Qwen step. - Energon online packing does not support MTP, CUDA graphs, or pipeline parallelism. diff --git a/skills/nemo-mbridge-perf-sequence-packing/skill-card.md b/skills/nemo-mbridge-perf-sequence-packing/skill-card.md index b4da1212..9c14de98 100644 --- a/skills/nemo-mbridge-perf-sequence-packing/skill-card.md +++ b/skills/nemo-mbridge-perf-sequence-packing/skill-card.md @@ -9,14 +9,14 @@ NVIDIA
### License/Terms of Use:
Apache 2.0
## Use Case:
-Developers and engineers configuring sequence packing and long-context training in Megatron-Bridge for LLM and VLM finetuning workloads.
+Developers and ML engineers configuring sequence packing and long-context training for LLM and VLM fine-tuning workflows in Megatron-Bridge.
### Deployment Geography for Use:
Global
## Requirements / Dependencies:
-**Requires API Key or External Credential:** [Not Specified]
-**Credential Type(s):** [None identified]
+**Requires API Key or External Credential:** [No]
+**Credential Type(s):** [None]
Do not include secrets in prompts/logs/output; use least-privilege credentials; rotate keys as appropriate.
@@ -27,7 +27,7 @@ Mitigation: Review and scan skill before deployment.
## Reference(s):
- [Packed Sequences Documentation](docs/training/packed-sequences.md)
- [Performance Tuning Guide](docs/performance-guide.md)
-- [Megatron Bridge Documentation](https://docs.nvidia.com/nemo/megatron-bridge/latest/)
+- [Hierarchical Context Parallel](docs/training/hierarchical-context-parallel.md)
## Skill Output:
@@ -43,15 +43,15 @@ Mitigation: Review and scan skill before deployment.
## Evaluation Tasks:
-Evaluated against 1 task (1 positive) in isolated k8s-sandbox pods with dataset digest sha256:57d3c088.
+Evaluated against 1 task (1 positive) using evaluator version 1.2.7 in local environment with Tier 3 live agent evaluation.
## Evaluation Metrics Used:
Reported benchmark dimensions:
-- Security: Whether the skill is safe to use (no unsafe operations, secret leakage, or unauthorized access).
-- Correctness: Whether the skill produces correct answers against reference ground truth.
-- Discoverability: Whether the right skill is loaded and activated when needed.
-- Effectiveness: Whether the skill helps complete the user's goal and expected workflow.
-- Efficiency: Whether the skill avoids wasted tool or skill usage.
+- Security: Whether the skill is safe to use (unsafe operations, secret leakage, unauthorized access).
+- Correctness: Whether the answer produced is correct against the reference answer.
+- Discoverability: Whether the right skill was loaded and executed when needed.
+- Effectiveness: Whether the skill helped complete the user's goal and expected workflow (goal accuracy + behavior check).
+- Efficiency: Whether the skill avoided wasted tool or skill usage (routing quality and productive tool use).
Underlying evaluation signals used in this run:
- `security`: Unsafe operations, secret leakage, and unauthorized access.
@@ -66,17 +66,12 @@ Underlying evaluation signals used in this run:
## Evaluation Results:
| Measure | Claude Code (Baseline → Skill Uplift) | Codex (Baseline → Skill Uplift) | |---|---:|---:| -| Overall | 37% → 91% (+54 points) | 68% → 86% (+18 points) | +| Overall | 60% → 98% (+38 points) | 56% → 96% (+40 points) | | Security | 100% → 100% (±0 points) | 100% → 100% (±0 points) | -| Correctness | 0% → 100% (+100 points) | 100% → 100% (±0 points) | -| Discoverability | 50% → 100% (+50 points) | 50% → 88% (+38 points) | -| Effectiveness | 0% → 66% (+66 points) | 74% → 45% (-29 points) | -| Efficiency | 35% → 90% (+55 points) | 17% → 100% (+83 points) | - -## Testing Completed:
-**[x] Agent Red-Teaming**
-**[ ] Network Security**
-**[ ] Product Security**
+| Correctness | 0% → 100% (+100 points) | 40% → 100% (+60 points) | +| Discoverability | 100% → 100% (±0 points) | 50% → 88% (+38 points) | +| Effectiveness | 0% → 88% (+88 points) | 40% → 94% (+54 points) | +| Efficiency | 100% → 100% (±0 points) | 50% → 100% (+50 points) | ## Skill Version(s):
1.0.0+b7643bd (source: pyproject.toml)
diff --git a/skills/nemo-mbridge-perf-sequence-packing/skill.oms.sig b/skills/nemo-mbridge-perf-sequence-packing/skill.oms.sig index f927fd0a..63829267 100644 --- a/skills/nemo-mbridge-perf-sequence-packing/skill.oms.sig +++ b/skills/nemo-mbridge-perf-sequence-packing/skill.oms.sig @@ -1 +1 @@ -{"mediaType":"application/vnd.dev.sigstore.bundle.v0.3+json","verificationMaterial":{"x509CertificateChain":{"certificates":[{"rawBytes":"MIICgzCCAgmgAwIBAgIUKIyS7SxNteQIiWzK1dWj85E6520wCgYIKoZIzj0EAwMwVTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjEpMCcGA1UEAwwgTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBJQ0EgMDEwHhcNMjYwNDAxMDAwMDAwWhcNMjgwNDIyMTUzMzA5WjBUMQswCQYDVQQGEwJVUzEbMBkGA1UECgwSTlZJRElBIENvcnBvcmF0aW9uMSgwJgYDVQQDDB9OVklESUEgQWdlbnQgU2tpbGxzIFNpZ25pbmcgMDAxMHYwEAYHKoZIzj0CAQYFK4EEACIDYgAEYoRM9bQl/dGlwSRNi6bTpIJUXH8Nv9GciP6LSflJYYMLCc296kpyuTSsk5ddbAWiDcFX3C/ydX3jwc+qCLYP6uHy9XphyLjOQ27Yb2J6rBLVtRBS1mgGco/Gr7fL6ODco4GaMIGXMB0GA1UdDgQWBBRQ/5ZW3nJ6lmo9SVk7I15o7UGmpTAfBgNVHSMEGDAWgBRPGpILxMBBleJSsBGjrMKsby1CgjAMBgNVHRMBAf8EAjAAMA4GA1UdDwEB/wQEAwIHgDA3BggrBgEFBQcBAQQrMCkwJwYIKwYBBQUHMAGGG2h0dHA6Ly9vY3NwLm5kaXMubnZpZGlhLmNvbTAKBggqhkjOPQQDAwNoADBlAjAUygu/GiOCIXrgGr4SmLgeEVDcEitfFUv7ALbvLVGVyMysB3mxmO/uInZfXzWcJZsCMQDxuoxj4ZmO30jhkPIcCxGFCOvnUsnfU3TfGcouYm4M6iRpbKvtVnHPiy4bi6pcKf0="},{"rawBytes":"MIICiDCCAg6gAwIBAgIUZsIuSv9NkpJCNqtYEfCouVv5BzowCgYIKoZIzj0EAwMwUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTAgFw0yNjA0MDEwMDAwMDBaGA85OTk5MTIzMTIzNTk1OVowVTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjEpMCcGA1UEAwwgTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBJQ0EgMDEwdjAQBgcqhkjOPQIBBgUrgQQAIgNiAASI72cR3ctKGg4VWnB3bNja6g1Z2PnOmFEopkPof+QeIcPk9rT+g9MjJnq51EQXL93a7C2GJ9J985G4o2V85VD7wJ1RaXhluHW2rf3y8bQGeAYaKMr5s/hUgn+M3/9WlWejgaAwgZ0wHQYDVR0OBBYEFE8akgvEwEGV4lKwEaOswqxvLUKCMB8GA1UdIwQYMBaAFItnoAjjfuCEUvzyvWyI2vOGvwPjMBIGA1UdEwEB/wQIMAYBAf8CAQAwDgYDVR0PAQH/BAQDAgEGMDcGCCsGAQUFBwEBBCswKTAnBggrBgEFBQcwAYYbaHR0cDovL29jc3AubmRpcy5udmlkaWEuY29tMAoGCCqGSM49BAMDA2gAMGUCMQCeIMMfAbyzPDacw2MxG+Yt1cikrJX/DVxiGfXuHmkkXn6VgSzE79+lkqDErpVO2gYCMCNEColOyvUvkzZGUEI1hQ3PfMgi3FIo9tHoBKMw4/wGBLFpu/0ubtmbBXM6/UMOEw=="},{"rawBytes":"MIICRTCCAcygAwIBAgIUeJdY3rV86EdvFmG7L8LJBsyQFYkwCgYIKoZIzj0EAwMwUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTAgFw0yNjA0MDEwMDAwMDBaGA85OTk5MTIzMTIzNTk1OVowUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTB2MBAGByqGSM49AgEGBSuBBAAiA2IABAYpiXCDjJ9NT2eSDhyHJVSw1Tbze18cGG2F/578oWvHxg23eQAhNRYdq88i1iOshZSO6C29doKui5Xpmo/7Ctw9Sx4PP2RzOmIuOLCuTdNtKcTRwi4GEsd5BAFvWj42M6NjMGEwHQYDVR0OBBYEFItnoAjjfuCEUvzyvWyI2vOGvwPjMB8GA1UdIwQYMBaAFItnoAjjfuCEUvzyvWyI2vOGvwPjMA8GA1UdEwEB/wQFMAMBAf8wDgYDVR0PAQH/BAQDAgEGMAoGCCqGSM49BAMDA2cAMGQCMCwtAjWLaNwgGWNCgdyNoTyvNhqWRECRJV2r3+7w8g0PL6NHLOsbkgE09BH95h8XlgIwTaQmbbUh2ChAJ5TA1wRiVDnCcvbzHlZl2jM2FcwQQZlk19LOAbyGMRixbu2Ww/rj"}]},"tlogEntries":[]},"dsseEnvelope":{"payload":"ewogICJfdHlwZSI6ICJodHRwczovL2luLXRvdG8uaW8vU3RhdGVtZW50L3YxIiwKICAic3ViamVjdCI6IFsKICAgIHsKICAgICAgIm5hbWUiOiAibmVtby1tYnJpZGdlLXBlcmYtc2VxdWVuY2UtcGFja2luZyIsCiAgICAgICJkaWdlc3QiOiB7CiAgICAgICAgInNoYTI1NiI6ICJlZGI5NzJlMWViNzIzNzhjMzc5YmJmODkyM2VmYmIyMmMwOTI2ZTg3NTk5Nzc4MWY2NjBmZDkzOTZlZGM1MTZiIgogICAgICB9CiAgICB9CiAgXSwKICAicHJlZGljYXRlVHlwZSI6ICJodHRwczovL21vZGVsX3NpZ25pbmcvc2lnbmF0dXJlL3YxLjAiLAogICJwcmVkaWNhdGUiOiB7CiAgICAicmVzb3VyY2VzIjogWwogICAgICB7CiAgICAgICAgImFsZ29yaXRobSI6ICJzaGEyNTYiLAogICAgICAgICJuYW1lIjogIkJFTkNITUFSSy5tZCIsCiAgICAgICAgImRpZ2VzdCI6ICIxY2RkYzBjYTc0Y2RlZmY1MWU5MWRmMGVlNzk0ODQyOTk3YTg5YjBiYjQ5YTVlOGQ0ODgxYjI2ZjdhOGYwYzFiIgogICAgICB9LAogICAgICB7CiAgICAgICAgImFsZ29yaXRobSI6ICJzaGEyNTYiLAogICAgICAgICJuYW1lIjogIlNLSUxMLm1kIiwKICAgICAgICAiZGlnZXN0IjogIjQzZDllMDdkYmNkMmRjZDQ1NTVmOGQyZDZhYzhlODM5MGU4NGFmOTMyNmY1OGI3YzJkYzIxMmQ5ZmFhNzY0NWMiCiAgICAgIH0sCiAgICAgIHsKICAgICAgICAiYWxnb3JpdGhtIjogInNoYTI1NiIsCiAgICAgICAgIm5hbWUiOiAiY2FyZC55YW1sIiwKICAgICAgICAiZGlnZXN0IjogImY3MzMxOWVkMzJiMDQyZWEyNmZhMTAzZjRmYzBjY2Y3NzhiZWQyZTc1ODMxZDIzM2MxODJhMzk1OTIxYTA2OWYiCiAgICAgIH0sCiAgICAgIHsKICAgICAgICAiYWxnb3JpdGhtIjogInNoYTI1NiIsCiAgICAgICAgIm5hbWUiOiAiZXZhbHMvZXZhbHMuanNvbiIsCiAgICAgICAgImRpZ2VzdCI6ICI5NGY0MTQyMzEyZjhjZjZlMGNkNGQ3NTA2NjEzNmU1YTFmNDBjNDkyMjg2OTRkNTUxMGRiYjg3YTI4ZDJkZWNlIgogICAgICB9LAogICAgICB7CiAgICAgICAgImFsZ29yaXRobSI6ICJzaGEyNTYiLAogICAgICAgICJuYW1lIjogInNraWxsLWNhcmQubWQiLAogICAgICAgICJkaWdlc3QiOiAiYTNhZmY1ZmMzMWE3YTE0YWU0YWY3YWRhMGYwZjYzMTI1OWRkMzFjZGYxNDcwZGMyYjc3OGRkODFmN2Q2MGNkMiIKICAgICAgfQogICAgXSwKICAgICJzZXJpYWxpemF0aW9uIjogewogICAgICAiaWdub3JlX3BhdGhzIjogWwogICAgICAgICIuZ2l0aHViIiwKICAgICAgICAiLmdpdCIsCiAgICAgICAgIi5naXRpZ25vcmUiLAogICAgICAgICIuZ2l0YXR0cmlidXRlcyIKICAgICAgXSwKICAgICAgIm1ldGhvZCI6ICJmaWxlcyIsCiAgICAgICJhbGxvd19zeW1saW5rcyI6IGZhbHNlLAogICAgICAiaGFzaF90eXBlIjogInNoYTI1NiIKICAgIH0KICB9Cn0=","payloadType":"application/vnd.in-toto+json","signatures":[{"sig":"MGYCMQDcUXl0d6T2gmbXdyUsM1Rv5NhwWgLomaePuFz6CJk6LRwxqwLhVsd/6k5PU7JmWoUCMQDeA3ncsPASeOjkjU0EmKpJqPnHmOSwoh0zCtGaIizN3mPCXtq2i/n71u4I59Gn+F4=","keyid":""}]}} \ No newline at end of file +{"mediaType":"application/vnd.dev.sigstore.bundle.v0.3+json","verificationMaterial":{"x509CertificateChain":{"certificates":[{"rawBytes":"MIICgzCCAgmgAwIBAgIUKIyS7SxNteQIiWzK1dWj85E6520wCgYIKoZIzj0EAwMwVTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjEpMCcGA1UEAwwgTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBJQ0EgMDEwHhcNMjYwNDAxMDAwMDAwWhcNMjgwNDIyMTUzMzA5WjBUMQswCQYDVQQGEwJVUzEbMBkGA1UECgwSTlZJRElBIENvcnBvcmF0aW9uMSgwJgYDVQQDDB9OVklESUEgQWdlbnQgU2tpbGxzIFNpZ25pbmcgMDAxMHYwEAYHKoZIzj0CAQYFK4EEACIDYgAEYoRM9bQl/dGlwSRNi6bTpIJUXH8Nv9GciP6LSflJYYMLCc296kpyuTSsk5ddbAWiDcFX3C/ydX3jwc+qCLYP6uHy9XphyLjOQ27Yb2J6rBLVtRBS1mgGco/Gr7fL6ODco4GaMIGXMB0GA1UdDgQWBBRQ/5ZW3nJ6lmo9SVk7I15o7UGmpTAfBgNVHSMEGDAWgBRPGpILxMBBleJSsBGjrMKsby1CgjAMBgNVHRMBAf8EAjAAMA4GA1UdDwEB/wQEAwIHgDA3BggrBgEFBQcBAQQrMCkwJwYIKwYBBQUHMAGGG2h0dHA6Ly9vY3NwLm5kaXMubnZpZGlhLmNvbTAKBggqhkjOPQQDAwNoADBlAjAUygu/GiOCIXrgGr4SmLgeEVDcEitfFUv7ALbvLVGVyMysB3mxmO/uInZfXzWcJZsCMQDxuoxj4ZmO30jhkPIcCxGFCOvnUsnfU3TfGcouYm4M6iRpbKvtVnHPiy4bi6pcKf0="},{"rawBytes":"MIICiDCCAg6gAwIBAgIUZsIuSv9NkpJCNqtYEfCouVv5BzowCgYIKoZIzj0EAwMwUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTAgFw0yNjA0MDEwMDAwMDBaGA85OTk5MTIzMTIzNTk1OVowVTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjEpMCcGA1UEAwwgTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBJQ0EgMDEwdjAQBgcqhkjOPQIBBgUrgQQAIgNiAASI72cR3ctKGg4VWnB3bNja6g1Z2PnOmFEopkPof+QeIcPk9rT+g9MjJnq51EQXL93a7C2GJ9J985G4o2V85VD7wJ1RaXhluHW2rf3y8bQGeAYaKMr5s/hUgn+M3/9WlWejgaAwgZ0wHQYDVR0OBBYEFE8akgvEwEGV4lKwEaOswqxvLUKCMB8GA1UdIwQYMBaAFItnoAjjfuCEUvzyvWyI2vOGvwPjMBIGA1UdEwEB/wQIMAYBAf8CAQAwDgYDVR0PAQH/BAQDAgEGMDcGCCsGAQUFBwEBBCswKTAnBggrBgEFBQcwAYYbaHR0cDovL29jc3AubmRpcy5udmlkaWEuY29tMAoGCCqGSM49BAMDA2gAMGUCMQCeIMMfAbyzPDacw2MxG+Yt1cikrJX/DVxiGfXuHmkkXn6VgSzE79+lkqDErpVO2gYCMCNEColOyvUvkzZGUEI1hQ3PfMgi3FIo9tHoBKMw4/wGBLFpu/0ubtmbBXM6/UMOEw=="},{"rawBytes":"MIICRTCCAcygAwIBAgIUeJdY3rV86EdvFmG7L8LJBsyQFYkwCgYIKoZIzj0EAwMwUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTAgFw0yNjA0MDEwMDAwMDBaGA85OTk5MTIzMTIzNTk1OVowUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTB2MBAGByqGSM49AgEGBSuBBAAiA2IABAYpiXCDjJ9NT2eSDhyHJVSw1Tbze18cGG2F/578oWvHxg23eQAhNRYdq88i1iOshZSO6C29doKui5Xpmo/7Ctw9Sx4PP2RzOmIuOLCuTdNtKcTRwi4GEsd5BAFvWj42M6NjMGEwHQYDVR0OBBYEFItnoAjjfuCEUvzyvWyI2vOGvwPjMB8GA1UdIwQYMBaAFItnoAjjfuCEUvzyvWyI2vOGvwPjMA8GA1UdEwEB/wQFMAMBAf8wDgYDVR0PAQH/BAQDAgEGMAoGCCqGSM49BAMDA2cAMGQCMCwtAjWLaNwgGWNCgdyNoTyvNhqWRECRJV2r3+7w8g0PL6NHLOsbkgE09BH95h8XlgIwTaQmbbUh2ChAJ5TA1wRiVDnCcvbzHlZl2jM2FcwQQZlk19LOAbyGMRixbu2Ww/rj"}]},"tlogEntries":[]},"dsseEnvelope":{"payload":"ewogICJfdHlwZSI6ICJodHRwczovL2luLXRvdG8uaW8vU3RhdGVtZW50L3YxIiwKICAic3ViamVjdCI6IFsKICAgIHsKICAgICAgIm5hbWUiOiAibmVtby1tYnJpZGdlLXBlcmYtc2VxdWVuY2UtcGFja2luZyIsCiAgICAgICJkaWdlc3QiOiB7CiAgICAgICAgInNoYTI1NiI6ICI3MjVmODdmZWQ1MGU3YTJkODNkNjA2NjQxMmFhNGM4ODMwZGZhNTUxZjBlOTBiYmI3NWViOGU0MWUxMWQ4YWU0IgogICAgICB9CiAgICB9CiAgXSwKICAicHJlZGljYXRlVHlwZSI6ICJodHRwczovL21vZGVsX3NpZ25pbmcvc2lnbmF0dXJlL3YxLjAiLAogICJwcmVkaWNhdGUiOiB7CiAgICAicmVzb3VyY2VzIjogWwogICAgICB7CiAgICAgICAgIm5hbWUiOiAiQkVOQ0hNQVJLLm1kIiwKICAgICAgICAiYWxnb3JpdGhtIjogInNoYTI1NiIsCiAgICAgICAgImRpZ2VzdCI6ICI5Mzc2Y2NhYmU2OTdhYzMxOTE5ODg1NTBlYzBjMTMyNzQzY2ZkOGRkMGFkNzZjZjBmN2ZiMTA4ZmUxZDkyZjBjIgogICAgICB9LAogICAgICB7CiAgICAgICAgIm5hbWUiOiAiU0tJTEwubWQiLAogICAgICAgICJhbGdvcml0aG0iOiAic2hhMjU2IiwKICAgICAgICAiZGlnZXN0IjogImZlYjUwODhiZWU4MTljMTkwM2Y4ZDE5MzgyYTc5ZTJhODE1MTE3MTMyZDY2ZTRlMWUxOTBiM2ZhMGUyNzkxY2EiCiAgICAgIH0sCiAgICAgIHsKICAgICAgICAibmFtZSI6ICJjYXJkLnlhbWwiLAogICAgICAgICJhbGdvcml0aG0iOiAic2hhMjU2IiwKICAgICAgICAiZGlnZXN0IjogIjUyMGIzM2MyZjBkNzVkNTI0NGZiY2U2MGVkNzY2ZmJmMDczMjFmMDFkNjMzYWViYzJhMGNhYmU0ZDgwZjA5YjMiCiAgICAgIH0sCiAgICAgIHsKICAgICAgICAibmFtZSI6ICJldmFscy9ldmFscy5qc29uIiwKICAgICAgICAiYWxnb3JpdGhtIjogInNoYTI1NiIsCiAgICAgICAgImRpZ2VzdCI6ICI5NGY0MTQyMzEyZjhjZjZlMGNkNGQ3NTA2NjEzNmU1YTFmNDBjNDkyMjg2OTRkNTUxMGRiYjg3YTI4ZDJkZWNlIgogICAgICB9LAogICAgICB7CiAgICAgICAgIm5hbWUiOiAic2tpbGwtY2FyZC5tZCIsCiAgICAgICAgImFsZ29yaXRobSI6ICJzaGEyNTYiLAogICAgICAgICJkaWdlc3QiOiAiY2Y3NTc3ZDQ3YTc4MDE4Zjk2ODBhZmU2ZmE5MThiNjc0YTk3OWZmYmJhYzFjYmI0OTI3MmNjMmNhOTExNzQzMyIKICAgICAgfQogICAgXSwKICAgICJzZXJpYWxpemF0aW9uIjogewogICAgICAibWV0aG9kIjogImZpbGVzIiwKICAgICAgImFsbG93X3N5bWxpbmtzIjogZmFsc2UsCiAgICAgICJoYXNoX3R5cGUiOiAic2hhMjU2IiwKICAgICAgImlnbm9yZV9wYXRocyI6IFsKICAgICAgICAiLmdpdGF0dHJpYnV0ZXMiLAogICAgICAgICIuZ2l0aHViIiwKICAgICAgICAiLmdpdCIsCiAgICAgICAgIi5naXRpZ25vcmUiCiAgICAgIF0KICAgIH0KICB9Cn0=","payloadType":"application/vnd.in-toto+json","signatures":[{"sig":"MGUCMQD0nElbjvUyGrUlSlPPkhlz1oaqo7VITEoMop9dNSHygCRM3xaSPRqjaYDIS6dD3/oCMFKh+hOoTwagf1NCwbOFknpUq0BN+nskrznexmpYek8F7u5s2Z0GD28xzrq1iiXkdA==","keyid":""}]}} \ No newline at end of file diff --git a/skills/nemo-relay-get-started/BENCHMARK.md b/skills/nemo-relay-get-started/BENCHMARK.md index af9a9466..5429a29a 100644 --- a/skills/nemo-relay-get-started/BENCHMARK.md +++ b/skills/nemo-relay-get-started/BENCHMARK.md @@ -9,16 +9,27 @@ Recommended for publication based on the completed evaluation evidence in this r ## Evaluation Metadata - Skill: `nemo-relay-get-started` -- Evaluation date: 2026-08-12 -- Evaluator version: `1.2.4` +- Evaluation date: 2026-08-21 +- Evaluator version: `1.3.2` - Agents: Claude Code (`aws/anthropic/bedrock-claude-opus-4-8`), Codex (`openai/openai/gpt-5.5`) - Tasks: 15 evaluation tasks (14 positive, 1 negative) - Dataset digest: `sha256:d8b84368c53829b1ab95079b2968f8e667766a8d13577bc10331d59a90e243ae` (skill-evaluator-dataset-snapshot/1) - Attempts per task: 1 -- Environment: `k8s-sandbox` +- Environment: `local` - Tier 3 evidence: required for publication -Each task attempt ran in its own isolated sandbox pod. +Tasks ran on the trusted local host; local mode is not sandboxed. + +## Execution and Provenance + +- Validation status: `passed` +- Report generation: `complete` +- Evaluator version: `1.3.2` +- Git commit: `0117bc2e3e54da4244a656466526c5b1b5a559ea` +- Content type: requested `auto`, detected `skill` +- Container image: `gitlab-master.nvidia.com:5005/nvcarps/ci-group/nvcarps-ci/skillevaluator-ci:sha-0117bc2e3e54da4244a656466526c5b1b5a559ea` +- Container image digest: `not recorded` +- Tier 3: requested `true`, executed `true`, status `succeeded` ## What This Report Answers @@ -34,12 +45,12 @@ The three-tier evaluation checks whether the skill: | Measure | Claude Code (Baseline → Skill Uplift) | Codex (Baseline → Skill Uplift) | |---|---:|---:| -| Overall | 48% → 86% (+38 points) | 51% → 79% (+29 points) | -| Security | 100% → 93% (-7 points) | 73% → 80% (+7 points) | -| Correctness | 25% → 95% (+69 points) | 63% → 88% (+25 points) | -| Discoverability | 49% → 95% (+46 points) | 48% → 89% (+41 points) | -| Effectiveness | 28% → 72% (+44 points) | 45% → 65% (+20 points) | -| Efficiency | 39% → 76% (+37 points) | 25% → 75% (+50 points) | +| Overall | 49% → 88% (+39 points) | 51% → 83% (+32 points) | +| Security | 100% → 100% (±0 points) | 73% → 90% (+17 points) | +| Correctness | 23% → 92% (+69 points) | 68% → 89% (+21 points) | +| Discoverability | 51% → 95% (+44 points) | 44% → 90% (+45 points) | +| Effectiveness | 27% → 68% (+40 points) | 42% → 70% (+27 points) | +| Efficiency | 44% → 84% (+40 points) | 27% → 76% (+49 points) | **How to read this table:** baseline is the same task attempted without the target skill. Uplift is `skill score - baseline score`, shown in percentage points. diff --git a/skills/nemo-relay-get-started/SKILL.md b/skills/nemo-relay-get-started/SKILL.md index a85ea5e5..5ffdafd2 100644 --- a/skills/nemo-relay-get-started/SKILL.md +++ b/skills/nemo-relay-get-started/SKILL.md @@ -114,6 +114,8 @@ Use another handoff only after the user accepts or declines plugin progression, or when the demonstrated boundary does not yet cover the intended workflow: - Direct application expansion -> `nemo-relay-instrument-calls` +- Unsupported framework or harness integration -> + `nemo-relay-integrate-upstream` - Additional exporters or durable observability configuration -> `nemo-relay-plugin-observability` - A different package or supported integration -> `nemo-relay-install` diff --git a/skills/nemo-relay-get-started/skill-card.md b/skills/nemo-relay-get-started/skill-card.md index 0fa7f13e..ab673eff 100644 --- a/skills/nemo-relay-get-started/skill-card.md +++ b/skills/nemo-relay-get-started/skill-card.md @@ -9,7 +9,7 @@ NVIDIA
### License/Terms of Use:
Apache 2.0
## Use Case:
-Developers and engineers onboarding to NeMo Relay who want to trial the framework, choose the quickest supported path to visible value, and verify initial instrumentation before committing to production setup.
+Developers and engineers adopting NeMo Relay for the first time use this skill to select the simplest applicable quick-start path and verify observable Relay value before production deployment.
### Deployment Geography for Use:
Global
@@ -28,15 +28,14 @@ Mitigation: Review and scan skill before deployment.
- [CLI Try-Now Reference](references/cli-try-now.md)
- [Built-In Integrations Try-Now Reference](references/built-in-integrations-try-now.md)
- [Manual Language Try-Now Reference](references/manual-language-try-now.md)
-- [NeMo Relay CLI Overview](https://docs.nvidia.com/nemo/relay/dev/nemo-relay-cli/about)
-- [Supported Integrations](https://docs.nvidia.com/nemo/relay/dev/supported-integrations/about)
-- [Language Quick Starts](https://docs.nvidia.com/nemo/relay/dev/getting-started/quick-start)
-- [Plugin Configuration](https://docs.nvidia.com/nemo/relay/dev/configure-plugins/about)
+- [NeMo Relay Getting Started Quick Start](https://docs.nvidia.com/nemo/relay/dev/getting-started/quick-start)
+- [NeMo Relay Supported Integrations](https://docs.nvidia.com/nemo/relay/dev/supported-integrations/about)
+- [NeMo Relay Plugin Configuration](https://docs.nvidia.com/nemo/relay/dev/configure-plugins/about)
## Skill Output:
-**Output Type(s):** [Configuration instructions, Shell commands, Code]
-**Output Format:** [Markdown with inline bash code blocks]
+**Output Type(s):** [Shell commands, Configuration instructions, Code]
+**Output Format:** [Markdown with inline bash and language-specific code blocks]
**Output Parameters:** [1D]
**Other Properties Related to Output:** [None]
@@ -47,38 +46,38 @@ Mitigation: Review and scan skill before deployment.
## Evaluation Tasks:
-15 evaluation tasks (14 positive, 1 negative) in isolated sandbox pods.
+Evaluated against 15 tasks (14 positive, 1 negative) from the skill-evaluator dataset.
## Evaluation Metrics Used:
Reported benchmark dimensions:
- Security: Whether the skill avoids unsafe operations, secret leakage, and unauthorized access.
-- Correctness: Whether the skill produces correct answers against reference answers.
-- Discoverability: Whether the right skill was found and executed when needed.
-- Effectiveness: Whether the skill helps complete the user's goal and expected workflow.
-- Efficiency: Whether the skill avoids wasted tool or skill usage.
+- Correctness: Whether the final answer is correct against the reference answer.
+- Discoverability: Whether the expected skill was found and executed when needed.
+- Effectiveness: Whether the skill helps complete the user's goal and follows expected workflow behavior.
+- Efficiency: Whether the skill avoids wasted tool or skill usage through good routing and productive tool use.
Underlying evaluation signals used in this run:
-- `security`: Unsafe operations, secret leakage, and unauthorized access.
-- `skill_execution`: Whether the expected skill was found and executed.
-- `skill_efficiency`: Routing quality, workspace-aware skill reads, and productive tool use.
-- `accuracy`: Final-answer correctness against the reference answer.
-- `goal_accuracy`: Whether the user's goal was achieved.
-- `behavior_check`: Whether the expected workflow behavior was followed.
+- `security`: Verifies absence of unsafe operations, secret leakage, and unauthorized access.
+- `skill_execution`: Verifies that the expected skill was found and executed.
+- `skill_efficiency`: Verifies routing quality, workspace-aware skill reads, and productive tool use.
+- `accuracy`: Verifies final-answer correctness against the reference answer.
+- `goal_accuracy`: Verifies whether the user's goal was achieved.
+- `behavior_check`: Verifies whether the expected workflow behavior was followed.
## Evaluation Results:
| Measure | Claude Code (Baseline → Skill Uplift) | Codex (Baseline → Skill Uplift) | |---|---:|---:| -| Overall | 48% → 86% (+38 points) | 51% → 79% (+29 points) | -| Security | 100% → 93% (-7 points) | 73% → 80% (+7 points) | -| Correctness | 25% → 95% (+69 points) | 63% → 88% (+25 points) | -| Discoverability | 49% → 95% (+46 points) | 48% → 89% (+41 points) | -| Effectiveness | 28% → 72% (+44 points) | 45% → 65% (+20 points) | -| Efficiency | 39% → 76% (+37 points) | 25% → 75% (+50 points) | +| Overall | 49% → 88% (+39 points) | 51% → 83% (+32 points) | +| Security | 100% → 100% (±0 points) | 73% → 90% (+17 points) | +| Correctness | 23% → 92% (+69 points) | 68% → 89% (+21 points) | +| Discoverability | 51% → 95% (+44 points) | 44% → 90% (+45 points) | +| Effectiveness | 27% → 68% (+40 points) | 42% → 70% (+27 points) | +| Efficiency | 44% → 84% (+40 points) | 27% → 76% (+49 points) | ## Skill Version(s):
-db4ed2e (source: git SHA, committed 2026-08-12)
+7eb33b6 (source: git SHA, committed 2026-08-21)
## Ethical Considerations:
NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal team to ensure this skill meets requirements for the relevant industry and use case and addresses unforeseen product misuse.
diff --git a/skills/nemo-relay-get-started/skill.oms.sig b/skills/nemo-relay-get-started/skill.oms.sig index 7fab99af..43e58457 100644 --- a/skills/nemo-relay-get-started/skill.oms.sig +++ b/skills/nemo-relay-get-started/skill.oms.sig @@ -1 +1 @@ -{"mediaType":"application/vnd.dev.sigstore.bundle.v0.3+json","verificationMaterial":{"x509CertificateChain":{"certificates":[{"rawBytes":"MIICgzCCAgmgAwIBAgIUKIyS7SxNteQIiWzK1dWj85E6520wCgYIKoZIzj0EAwMwVTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjEpMCcGA1UEAwwgTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBJQ0EgMDEwHhcNMjYwNDAxMDAwMDAwWhcNMjgwNDIyMTUzMzA5WjBUMQswCQYDVQQGEwJVUzEbMBkGA1UECgwSTlZJRElBIENvcnBvcmF0aW9uMSgwJgYDVQQDDB9OVklESUEgQWdlbnQgU2tpbGxzIFNpZ25pbmcgMDAxMHYwEAYHKoZIzj0CAQYFK4EEACIDYgAEYoRM9bQl/dGlwSRNi6bTpIJUXH8Nv9GciP6LSflJYYMLCc296kpyuTSsk5ddbAWiDcFX3C/ydX3jwc+qCLYP6uHy9XphyLjOQ27Yb2J6rBLVtRBS1mgGco/Gr7fL6ODco4GaMIGXMB0GA1UdDgQWBBRQ/5ZW3nJ6lmo9SVk7I15o7UGmpTAfBgNVHSMEGDAWgBRPGpILxMBBleJSsBGjrMKsby1CgjAMBgNVHRMBAf8EAjAAMA4GA1UdDwEB/wQEAwIHgDA3BggrBgEFBQcBAQQrMCkwJwYIKwYBBQUHMAGGG2h0dHA6Ly9vY3NwLm5kaXMubnZpZGlhLmNvbTAKBggqhkjOPQQDAwNoADBlAjAUygu/GiOCIXrgGr4SmLgeEVDcEitfFUv7ALbvLVGVyMysB3mxmO/uInZfXzWcJZsCMQDxuoxj4ZmO30jhkPIcCxGFCOvnUsnfU3TfGcouYm4M6iRpbKvtVnHPiy4bi6pcKf0="},{"rawBytes":"MIICiDCCAg6gAwIBAgIUZsIuSv9NkpJCNqtYEfCouVv5BzowCgYIKoZIzj0EAwMwUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTAgFw0yNjA0MDEwMDAwMDBaGA85OTk5MTIzMTIzNTk1OVowVTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjEpMCcGA1UEAwwgTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBJQ0EgMDEwdjAQBgcqhkjOPQIBBgUrgQQAIgNiAASI72cR3ctKGg4VWnB3bNja6g1Z2PnOmFEopkPof+QeIcPk9rT+g9MjJnq51EQXL93a7C2GJ9J985G4o2V85VD7wJ1RaXhluHW2rf3y8bQGeAYaKMr5s/hUgn+M3/9WlWejgaAwgZ0wHQYDVR0OBBYEFE8akgvEwEGV4lKwEaOswqxvLUKCMB8GA1UdIwQYMBaAFItnoAjjfuCEUvzyvWyI2vOGvwPjMBIGA1UdEwEB/wQIMAYBAf8CAQAwDgYDVR0PAQH/BAQDAgEGMDcGCCsGAQUFBwEBBCswKTAnBggrBgEFBQcwAYYbaHR0cDovL29jc3AubmRpcy5udmlkaWEuY29tMAoGCCqGSM49BAMDA2gAMGUCMQCeIMMfAbyzPDacw2MxG+Yt1cikrJX/DVxiGfXuHmkkXn6VgSzE79+lkqDErpVO2gYCMCNEColOyvUvkzZGUEI1hQ3PfMgi3FIo9tHoBKMw4/wGBLFpu/0ubtmbBXM6/UMOEw=="},{"rawBytes":"MIICRTCCAcygAwIBAgIUeJdY3rV86EdvFmG7L8LJBsyQFYkwCgYIKoZIzj0EAwMwUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTAgFw0yNjA0MDEwMDAwMDBaGA85OTk5MTIzMTIzNTk1OVowUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTB2MBAGByqGSM49AgEGBSuBBAAiA2IABAYpiXCDjJ9NT2eSDhyHJVSw1Tbze18cGG2F/578oWvHxg23eQAhNRYdq88i1iOshZSO6C29doKui5Xpmo/7Ctw9Sx4PP2RzOmIuOLCuTdNtKcTRwi4GEsd5BAFvWj42M6NjMGEwHQYDVR0OBBYEFItnoAjjfuCEUvzyvWyI2vOGvwPjMB8GA1UdIwQYMBaAFItnoAjjfuCEUvzyvWyI2vOGvwPjMA8GA1UdEwEB/wQFMAMBAf8wDgYDVR0PAQH/BAQDAgEGMAoGCCqGSM49BAMDA2cAMGQCMCwtAjWLaNwgGWNCgdyNoTyvNhqWRECRJV2r3+7w8g0PL6NHLOsbkgE09BH95h8XlgIwTaQmbbUh2ChAJ5TA1wRiVDnCcvbzHlZl2jM2FcwQQZlk19LOAbyGMRixbu2Ww/rj"}]},"tlogEntries":[]},"dsseEnvelope":{"payload":"ewogICJfdHlwZSI6ICJodHRwczovL2luLXRvdG8uaW8vU3RhdGVtZW50L3YxIiwKICAic3ViamVjdCI6IFsKICAgIHsKICAgICAgIm5hbWUiOiAibmVtby1yZWxheS1nZXQtc3RhcnRlZCIsCiAgICAgICJkaWdlc3QiOiB7CiAgICAgICAgInNoYTI1NiI6ICI4MTY2YzY5MzcyNzU4ZjQwYzgwMmFkNWQxZWZiOWNiZDVkOTFiNDIzNTZlMWY4ODQyZGM5M2E5N2M5ZjhmN2NmIgogICAgICB9CiAgICB9CiAgXSwKICAicHJlZGljYXRlVHlwZSI6ICJodHRwczovL21vZGVsX3NpZ25pbmcvc2lnbmF0dXJlL3YxLjAiLAogICJwcmVkaWNhdGUiOiB7CiAgICAicmVzb3VyY2VzIjogWwogICAgICB7CiAgICAgICAgImFsZ29yaXRobSI6ICJzaGEyNTYiLAogICAgICAgICJkaWdlc3QiOiAiNTQ3MWIyYTEzOGVjZjBiNDg1NzRkNzI0MmJlNjQwMTJhNGMyNjBlY2M5OGE1YTE1NWVkZjVkYzc1NGMxYTE2YiIsCiAgICAgICAgIm5hbWUiOiAiQkVOQ0hNQVJLLm1kIgogICAgICB9LAogICAgICB7CiAgICAgICAgImFsZ29yaXRobSI6ICJzaGEyNTYiLAogICAgICAgICJkaWdlc3QiOiAiZjU2ZTFjZWIyMGJhZTA2MGFmZTBkNzFlZjdjYjUzZTlkNjk3ZmRiZjc5NTdlZDlkYjJlNGUyOWFhMzRlMzM4YSIsCiAgICAgICAgIm5hbWUiOiAiU0tJTEwubWQiCiAgICAgIH0sCiAgICAgIHsKICAgICAgICAiYWxnb3JpdGhtIjogInNoYTI1NiIsCiAgICAgICAgImRpZ2VzdCI6ICJiZmM1MzIzMDNhOGJjYTM0YmNiNDEzYWFlZWIyNDY5MGFmYzQ5N2M4NTRlMzgzZTFiNzU4MTQ1MTI3NmVhMGY3IiwKICAgICAgICAibmFtZSI6ICJldmFscy9ldmFscy5qc29uIgogICAgICB9LAogICAgICB7CiAgICAgICAgImFsZ29yaXRobSI6ICJzaGEyNTYiLAogICAgICAgICJkaWdlc3QiOiAiNjU4MTA1YTQyMWNiMTg2MWVjM2ZmNjQ0ZmEwZjRlYmRhNTYyZmQ2NjViMjJhNGQxZDAwYWFkNDliYTJlYzM0YyIsCiAgICAgICAgIm5hbWUiOiAicmVmZXJlbmNlcy9idWlsdC1pbi1pbnRlZ3JhdGlvbnMtdHJ5LW5vdy5tZCIKICAgICAgfSwKICAgICAgewogICAgICAgICJhbGdvcml0aG0iOiAic2hhMjU2IiwKICAgICAgICAiZGlnZXN0IjogIjkzZDZjMzA0YTU4NjYxZjA4MWQ2OTZkNTJiMmMzOTc0ZDJmYmQyYjE2MjMzODgwYTVhM2Y0ODhiYTUzMDNmYzEiLAogICAgICAgICJuYW1lIjogInJlZmVyZW5jZXMvY2xpLXRyeS1ub3cubWQiCiAgICAgIH0sCiAgICAgIHsKICAgICAgICAiYWxnb3JpdGhtIjogInNoYTI1NiIsCiAgICAgICAgImRpZ2VzdCI6ICI3N2QwMzRlMzBjNWI5ZWQ1ODlkYWJkNTgxN2VkNmI3NmFkYWUxZmY2OGUwN2Y1Yzk1NzQyNTM4YTQ2ZWQ1Zjk1IiwKICAgICAgICAibmFtZSI6ICJyZWZlcmVuY2VzL21hbnVhbC1sYW5ndWFnZS10cnktbm93Lm1kIgogICAgICB9LAogICAgICB7CiAgICAgICAgImFsZ29yaXRobSI6ICJzaGEyNTYiLAogICAgICAgICJkaWdlc3QiOiAiYzQ4ZjY5MWZkODkzYWYyNjRlYzA3ZjhlNzhhMmZkYTE1MmI3MTNlMzNmZmU2Y2E5ZjQwOTc1YjU0NWRmYzIzNiIsCiAgICAgICAgIm5hbWUiOiAic2tpbGwtY2FyZC5tZCIKICAgICAgfQogICAgXSwKICAgICJzZXJpYWxpemF0aW9uIjogewogICAgICAiaWdub3JlX3BhdGhzIjogWwogICAgICAgICIuZ2l0IiwKICAgICAgICAiLmdpdGlnbm9yZSIsCiAgICAgICAgIi5naXRhdHRyaWJ1dGVzIiwKICAgICAgICAiLmdpdGh1YiIKICAgICAgXSwKICAgICAgIm1ldGhvZCI6ICJmaWxlcyIsCiAgICAgICJoYXNoX3R5cGUiOiAic2hhMjU2IiwKICAgICAgImFsbG93X3N5bWxpbmtzIjogZmFsc2UKICAgIH0KICB9Cn0=","payloadType":"application/vnd.in-toto+json","signatures":[{"sig":"MGYCMQDsrpBYJmNLM15lTS5RSdFv2lIGmxuE7sBxZ05I8n4gzXFGQqI4MTdCKRN4yRYAUmgCMQCfXDJMgmRsEkOH/R3Og0h1kcPreWbr7e4DpgpnqfPk+y/5t72IGTQ4TJHV81ZbOT0=","keyid":""}]}} \ No newline at end of file +{"mediaType":"application/vnd.dev.sigstore.bundle.v0.3+json","verificationMaterial":{"x509CertificateChain":{"certificates":[{"rawBytes":"MIICgzCCAgmgAwIBAgIUKIyS7SxNteQIiWzK1dWj85E6520wCgYIKoZIzj0EAwMwVTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjEpMCcGA1UEAwwgTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBJQ0EgMDEwHhcNMjYwNDAxMDAwMDAwWhcNMjgwNDIyMTUzMzA5WjBUMQswCQYDVQQGEwJVUzEbMBkGA1UECgwSTlZJRElBIENvcnBvcmF0aW9uMSgwJgYDVQQDDB9OVklESUEgQWdlbnQgU2tpbGxzIFNpZ25pbmcgMDAxMHYwEAYHKoZIzj0CAQYFK4EEACIDYgAEYoRM9bQl/dGlwSRNi6bTpIJUXH8Nv9GciP6LSflJYYMLCc296kpyuTSsk5ddbAWiDcFX3C/ydX3jwc+qCLYP6uHy9XphyLjOQ27Yb2J6rBLVtRBS1mgGco/Gr7fL6ODco4GaMIGXMB0GA1UdDgQWBBRQ/5ZW3nJ6lmo9SVk7I15o7UGmpTAfBgNVHSMEGDAWgBRPGpILxMBBleJSsBGjrMKsby1CgjAMBgNVHRMBAf8EAjAAMA4GA1UdDwEB/wQEAwIHgDA3BggrBgEFBQcBAQQrMCkwJwYIKwYBBQUHMAGGG2h0dHA6Ly9vY3NwLm5kaXMubnZpZGlhLmNvbTAKBggqhkjOPQQDAwNoADBlAjAUygu/GiOCIXrgGr4SmLgeEVDcEitfFUv7ALbvLVGVyMysB3mxmO/uInZfXzWcJZsCMQDxuoxj4ZmO30jhkPIcCxGFCOvnUsnfU3TfGcouYm4M6iRpbKvtVnHPiy4bi6pcKf0="},{"rawBytes":"MIICiDCCAg6gAwIBAgIUZsIuSv9NkpJCNqtYEfCouVv5BzowCgYIKoZIzj0EAwMwUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTAgFw0yNjA0MDEwMDAwMDBaGA85OTk5MTIzMTIzNTk1OVowVTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjEpMCcGA1UEAwwgTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBJQ0EgMDEwdjAQBgcqhkjOPQIBBgUrgQQAIgNiAASI72cR3ctKGg4VWnB3bNja6g1Z2PnOmFEopkPof+QeIcPk9rT+g9MjJnq51EQXL93a7C2GJ9J985G4o2V85VD7wJ1RaXhluHW2rf3y8bQGeAYaKMr5s/hUgn+M3/9WlWejgaAwgZ0wHQYDVR0OBBYEFE8akgvEwEGV4lKwEaOswqxvLUKCMB8GA1UdIwQYMBaAFItnoAjjfuCEUvzyvWyI2vOGvwPjMBIGA1UdEwEB/wQIMAYBAf8CAQAwDgYDVR0PAQH/BAQDAgEGMDcGCCsGAQUFBwEBBCswKTAnBggrBgEFBQcwAYYbaHR0cDovL29jc3AubmRpcy5udmlkaWEuY29tMAoGCCqGSM49BAMDA2gAMGUCMQCeIMMfAbyzPDacw2MxG+Yt1cikrJX/DVxiGfXuHmkkXn6VgSzE79+lkqDErpVO2gYCMCNEColOyvUvkzZGUEI1hQ3PfMgi3FIo9tHoBKMw4/wGBLFpu/0ubtmbBXM6/UMOEw=="},{"rawBytes":"MIICRTCCAcygAwIBAgIUeJdY3rV86EdvFmG7L8LJBsyQFYkwCgYIKoZIzj0EAwMwUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTAgFw0yNjA0MDEwMDAwMDBaGA85OTk5MTIzMTIzNTk1OVowUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTB2MBAGByqGSM49AgEGBSuBBAAiA2IABAYpiXCDjJ9NT2eSDhyHJVSw1Tbze18cGG2F/578oWvHxg23eQAhNRYdq88i1iOshZSO6C29doKui5Xpmo/7Ctw9Sx4PP2RzOmIuOLCuTdNtKcTRwi4GEsd5BAFvWj42M6NjMGEwHQYDVR0OBBYEFItnoAjjfuCEUvzyvWyI2vOGvwPjMB8GA1UdIwQYMBaAFItnoAjjfuCEUvzyvWyI2vOGvwPjMA8GA1UdEwEB/wQFMAMBAf8wDgYDVR0PAQH/BAQDAgEGMAoGCCqGSM49BAMDA2cAMGQCMCwtAjWLaNwgGWNCgdyNoTyvNhqWRECRJV2r3+7w8g0PL6NHLOsbkgE09BH95h8XlgIwTaQmbbUh2ChAJ5TA1wRiVDnCcvbzHlZl2jM2FcwQQZlk19LOAbyGMRixbu2Ww/rj"}]},"tlogEntries":[]},"dsseEnvelope":{"payload":"ewogICJfdHlwZSI6ICJodHRwczovL2luLXRvdG8uaW8vU3RhdGVtZW50L3YxIiwKICAic3ViamVjdCI6IFsKICAgIHsKICAgICAgIm5hbWUiOiAibmVtby1yZWxheS1nZXQtc3RhcnRlZCIsCiAgICAgICJkaWdlc3QiOiB7CiAgICAgICAgInNoYTI1NiI6ICJmZWJhYThiYzg1ZmY1NzE4ZjIwMzcwMGQyMmM4MWI4ZTM4MjhhYTY5YWY4MzI4ZWUxNzIyZjhiY2M1YTIxZTUzIgogICAgICB9CiAgICB9CiAgXSwKICAicHJlZGljYXRlVHlwZSI6ICJodHRwczovL21vZGVsX3NpZ25pbmcvc2lnbmF0dXJlL3YxLjAiLAogICJwcmVkaWNhdGUiOiB7CiAgICAic2VyaWFsaXphdGlvbiI6IHsKICAgICAgIm1ldGhvZCI6ICJmaWxlcyIsCiAgICAgICJoYXNoX3R5cGUiOiAic2hhMjU2IiwKICAgICAgImlnbm9yZV9wYXRocyI6IFsKICAgICAgICAiLmdpdCIsCiAgICAgICAgIi5naXRhdHRyaWJ1dGVzIiwKICAgICAgICAiLmdpdGh1YiIsCiAgICAgICAgIi5naXRpZ25vcmUiCiAgICAgIF0sCiAgICAgICJhbGxvd19zeW1saW5rcyI6IGZhbHNlCiAgICB9LAogICAgInJlc291cmNlcyI6IFsKICAgICAgewogICAgICAgICJuYW1lIjogIkJFTkNITUFSSy5tZCIsCiAgICAgICAgImRpZ2VzdCI6ICI1OWZhYWQxYWVmYjRkODM3MzJkZGQ5MDNiMDA0NmJiZWQ3ZDBlNWQwYzk5ZGI1N2U5NWQyZTAwMjYwODA2OWE2IiwKICAgICAgICAiYWxnb3JpdGhtIjogInNoYTI1NiIKICAgICAgfSwKICAgICAgewogICAgICAgICJuYW1lIjogIlNLSUxMLm1kIiwKICAgICAgICAiZGlnZXN0IjogIjk2MzRmYzk3ODNlMDRhZDVlZGZmZjMyNzBmOTkyMDFhYzUyZjljNjhlOTc3ZWUxMjUzOGFmODI1OWEyYjcxMTciLAogICAgICAgICJhbGdvcml0aG0iOiAic2hhMjU2IgogICAgICB9LAogICAgICB7CiAgICAgICAgIm5hbWUiOiAiZXZhbHMvZXZhbHMuanNvbiIsCiAgICAgICAgImRpZ2VzdCI6ICJiZmM1MzIzMDNhOGJjYTM0YmNiNDEzYWFlZWIyNDY5MGFmYzQ5N2M4NTRlMzgzZTFiNzU4MTQ1MTI3NmVhMGY3IiwKICAgICAgICAiYWxnb3JpdGhtIjogInNoYTI1NiIKICAgICAgfSwKICAgICAgewogICAgICAgICJuYW1lIjogInJlZmVyZW5jZXMvYnVpbHQtaW4taW50ZWdyYXRpb25zLXRyeS1ub3cubWQiLAogICAgICAgICJkaWdlc3QiOiAiNjU4MTA1YTQyMWNiMTg2MWVjM2ZmNjQ0ZmEwZjRlYmRhNTYyZmQ2NjViMjJhNGQxZDAwYWFkNDliYTJlYzM0YyIsCiAgICAgICAgImFsZ29yaXRobSI6ICJzaGEyNTYiCiAgICAgIH0sCiAgICAgIHsKICAgICAgICAibmFtZSI6ICJyZWZlcmVuY2VzL2NsaS10cnktbm93Lm1kIiwKICAgICAgICAiZGlnZXN0IjogIjkzZDZjMzA0YTU4NjYxZjA4MWQ2OTZkNTJiMmMzOTc0ZDJmYmQyYjE2MjMzODgwYTVhM2Y0ODhiYTUzMDNmYzEiLAogICAgICAgICJhbGdvcml0aG0iOiAic2hhMjU2IgogICAgICB9LAogICAgICB7CiAgICAgICAgIm5hbWUiOiAicmVmZXJlbmNlcy9tYW51YWwtbGFuZ3VhZ2UtdHJ5LW5vdy5tZCIsCiAgICAgICAgImRpZ2VzdCI6ICI3N2QwMzRlMzBjNWI5ZWQ1ODlkYWJkNTgxN2VkNmI3NmFkYWUxZmY2OGUwN2Y1Yzk1NzQyNTM4YTQ2ZWQ1Zjk1IiwKICAgICAgICAiYWxnb3JpdGhtIjogInNoYTI1NiIKICAgICAgfSwKICAgICAgewogICAgICAgICJuYW1lIjogInNraWxsLWNhcmQubWQiLAogICAgICAgICJkaWdlc3QiOiAiMGY5ZjViOWI4NjQ1ZmY5ZGNlZDk5YjgzYzI4NzBmNzgxNjY3MGU5NDM0NTU5NzkxMmVhNTAxNDE5NjhlMTk4YiIsCiAgICAgICAgImFsZ29yaXRobSI6ICJzaGEyNTYiCiAgICAgIH0KICAgIF0KICB9Cn0=","payloadType":"application/vnd.in-toto+json","signatures":[{"sig":"MGQCMF3KRYmH0LrSod29lEej89KZqsS88MsHcmk6wRIPCyOgUomqMDjpQasIPP9Z2zr0qAIwRBCdroZvtBKTZXIlne8vnDO5Ow6F043Uaa8iEGdupvP2HneDobu6BKaodyl3iDxr","keyid":""}]}} \ No newline at end of file