NVIDIA · github-actions · Aug 23, 2026
@@ -9,16 +9,16 @@ Recommended for publication based on the completed evaluation evidence in this r
 ## Evaluation Metadata
 
 - Skill: `nemo-mbridge-perf-sequence-packing`
-- Evaluation date: 2026-08-11
-- Evaluator version: `1.2.0`
+- Evaluation date: 2026-08-17
+- Evaluator version: `1.2.7`
 - Agents: Claude Code (`aws/anthropic/bedrock-claude-opus-4-8`), Codex (`openai/openai/gpt-5.5`)
 - Tasks: 1 evaluation tasks (1 positive)
 - Dataset digest: `sha256:57d3c088dee48ee97547a666ee24dcac45c6cd5cbf699640c3c7d41710f629a1` (skill-evaluator-dataset-snapshot/1)
 - Attempts per task: 1
-- Environment: `k8s-sandbox`
+- Environment: `local`
 - Tier 3 evidence: required for publication
 
-Each task attempt ran in its own isolated sandbox pod.
+Tasks ran on the trusted local host; local mode is not sandboxed.
 
 ## What This Report Answers
 
@@ -34,12 +34,12 @@ The three-tier evaluation checks whether the skill:
 
 | Measure | Claude Code (Baseline → Skill Uplift) | Codex (Baseline → Skill Uplift) |
 |---|---:|---:|
-| Overall | 37% → 91% (+54 points) | 68% → 86% (+18 points) |
+| Overall | 60% → 98% (+38 points) | 56% → 96% (+40 points) |
 | Security | 100% → 100% (±0 points) | 100% → 100% (±0 points) |
-| Correctness | 0% → 100% (+100 points) | 100% → 100% (±0 points) |
-| Discoverability | 50% → 100% (+50 points) | 50% → 88% (+38 points) |
-| Effectiveness | 0% → 66% (+66 points) | 74% → 45% (-29 points) |
-| Efficiency | 35% → 90% (+55 points) | 17% → 100% (+83 points) |
+| Correctness | 0% → 100% (+100 points) | 40% → 100% (+60 points) |
+| Discoverability | 100% → 100% (±0 points) | 50% → 88% (+38 points) |
+| Effectiveness | 0% → 88% (+88 points) | 40% → 94% (+54 points) |
+| Efficiency | 100% → 100% (±0 points) | 50% → 100% (+50 points) |
 
 **How to read this table:** baseline is the same task attempted without the target skill. Uplift is `skill score - baseline score`, shown in percentage points.
 

@@ -104,13 +104,21 @@ the packed dataset (asserted in `src/megatron/bridge/data/datasets/sft.py`).
 Custom packed datasets that omit the metadata file will hit an assertion at
 dataset initialization.
 
-In-batch packing for VLM finetuning:
+In-batch packing for GPT SFT and supported VLM finetuning:
 
 ```python
 cfg.dataset.enable_in_batch_packing = True
-cfg.train.micro_batch_size = 2
+cfg.dataset.dataloader_type = "single"
+cfg.train.micro_batch_size = 4
 ```
 
+For local or materialized GPT-SFT JSONL, this keeps the existing mmap-backed
+dataset and performs tokenization lazily. Both prompt/completion
+(`GPTSFTDataset`) and chat (`GPTSFTChatDataset`) preserve their loss-mask
+semantics. Use `dataloader_type="single"` or `"cyclic"` so every DataLoader
+yield is one logical microbatch; GPT-SFT in-batch packing does not support the
+global-batch `"batch"` dataloader.
+
 Energon online packing for Qwen-VL uses Energon's per-worker candidate buffer
 instead of limiting selection to one collator micro batch:
 
@@ -232,6 +240,14 @@ def prepare_padded_or_packed_sequence_batch(
         return
 ```
 
+GPT-SFT direct-row packing:
+
+```627:671:src/megatron/bridge/data/datasets/gpt_sft.py
+def _collate_in_batch(self, batch):
+    ...
+    return build_mcore_thd_sequence_batch_from_rows(...)
+```
+
 Packed THD runtime constraint:
 
 ```94:108:src/megatron/bridge/training/gpt_step.py
@@ -250,18 +266,19 @@ if cu_seqlens.dim() > 1 and cu_seqlens.size(0) != 1:
 
 ## Pitfalls
 
-1. Offline packed SFT, collate-time VLM packing, and Energon online packing are different features. Offline and Energon packing use physical MBS1; collate-time packing uses MBS greater than one.
-2. When CP is enabled, packed sequence lengths must respect `2 * context_parallel_size` divisibility.
-3. For finetuning with CP, `calculate_per_token_loss=True` and `ddp.average_in_collective=False` are required.
-4. `pad_cu_seqlens=True` also requires `pad_to_max_length=True`.
-5. Packing support is model-family-specific. `Qwen3-Next`, `GLM-4.5`, and `Qwen3.5-VL` contain explicit opt-outs in different paths.
-6. MTP finetuning is documented as incompatible with packed sequences.
-7. Synthetic padding rows, including negative indices remapped through `samples_mapping`, must retain an all-zero loss mask.
-8. `global_batch_size` must be divisible by and no smaller than data parallel size when offline packing uses MBS1.
-9. Derive `pad_seq_to_mult` from CP/TP/SP for both SFT and PEFT; do not hardcode different values by workload type.
-10. `pad_to_max_length` controls final pack width and is conditional on fixed-shape execution requirements.
-11. Energon `packing_buffer_size` is per worker and also affects validation; global/eval batch counts refer to physical packs rather than source conversations.
-12. Exact Energon loader resume requires unchanged shards/splits, DP world size, worker counts, shuffle settings/seed, processor, sequence length, topology, and packing-buffer size.
+1. Offline packed SFT, runtime in-batch packing, and Energon online packing are different features. Offline and Energon packing use physical MBS1; runtime in-batch packing uses MBS greater than one.
+2. GPT-SFT in-batch packing requires `dataloader_type="single"` or `"cyclic"`; it does not support `"batch"`.
+3. When CP is enabled, packed sequence lengths must respect `2 * context_parallel_size` divisibility.
+4. For finetuning with CP, `calculate_per_token_loss=True` and `ddp.average_in_collective=False` are required.
+5. `pad_cu_seqlens=True` also requires `pad_to_max_length=True`.
+6. Packing support is model-family-specific. `Qwen3-Next`, `GLM-4.5`, and `Qwen3.5-VL` contain explicit opt-outs in different paths.
+7. MTP finetuning is documented as incompatible with packed sequences.
+8. Synthetic padding rows, including negative indices remapped through `samples_mapping`, must retain an all-zero loss mask.
+9. `global_batch_size` must be divisible by and no smaller than data parallel size when offline packing uses MBS1.
+10. Derive `pad_seq_to_mult` from CP/TP/SP for both SFT and PEFT; do not hardcode different values by workload type.
+11. `pad_to_max_length` controls final pack width and is conditional on fixed-shape execution requirements.
+12. Energon `packing_buffer_size` is per worker and also affects validation; global/eval batch counts refer to physical packs rather than source conversations.
+13. Exact Energon loader resume requires unchanged shards/splits, DP world size, worker counts, shuffle settings/seed, processor, sequence length, topology, and packing-buffer size.
 
 ## Verification
 
@@ -271,6 +288,8 @@ Use the checked-in unit coverage:
 uv run python -m pytest tests/unit_tests/training/utils/test_packed_seq_utils.py -v && \
 uv run python -m pytest tests/unit_tests/training/test_config.py -k "packed_sequence or enable_in_batch_packing or offline_and_in_batch_packing_are_mutually_exclusive or context_parallel_seq_length_divisibility or context_parallel_finetuning_validations" -v && \
 uv run python -m pytest tests/unit_tests/data/packing/test_in_batch.py -v && \
+uv run python -m pytest tests/unit_tests/data/datasets/test_gpt_sft.py -k "in_batch_packing" -v && \
+uv run python -m pytest tests/unit_tests/data/builders/test_gpt_sft_config.py -v && \
 uv run python -m pytest tests/unit_tests/training/test_vlm_step.py -k "deferred_in_batch_packing or packed_metadata" -v && \
 uv run python -m pytest tests/unit_tests/models/qwen_vl/data/test_energon.py tests/unit_tests/data/builders/test_energon_builder.py -v && \
 uv run python -m pytest tests/unit_tests/tutorials/test_multimodal_data_tutorials.py -k "native_packing_loader" -v && \
@@ -283,5 +302,6 @@ Success criteria:
 - all selected tests pass
 - offline and in-batch configuration validation remains mutually exclusive
 - packed metadata reaches the training step in MCore THD form
+- GPT-SFT in-batch packing rejects the global-batch `"batch"` dataloader
 - native Energon packing restores pending groups exactly and flushes finite partial buffers without dropping samples
 - mapped padding rows do not contribute to loss
@@ -79,6 +79,7 @@ known_constraints:
   - pad_seq_to_mult is derived from CP/TP/SP and follows the same rule for SFT and PEFT.
   - pad_to_max_length is conditional on a fixed-width dispatcher, kernel, or CUDA-graph path.
   - VLM in-batch packing requires micro_batch_size > 1.
+  - GPT-SFT in-batch packing requires dataloader_type single or cyclic; batch is unsupported.
   - Energon online packing requires physical micro_batch_size == 1 and counts candidate samples per worker.
   - Energon online packing is mutually exclusive with collate-time in-batch packing and the deferred Qwen step.
   - Energon online packing does not support MTP, CUDA graphs, or pipeline parallelism.

@@ -9,14 +9,14 @@ NVIDIA <br>
 ### License/Terms of Use: <br>
 Apache 2.0 <br>
 ## Use Case: <br>
-Developers and engineers configuring sequence packing and long-context training in Megatron-Bridge for LLM and VLM finetuning workloads. <br>
+Developers and ML engineers configuring sequence packing and long-context training for LLM and VLM fine-tuning workflows in Megatron-Bridge. <br>
 
 ### Deployment Geography for Use: <br>
 Global <br>
 
 ## Requirements / Dependencies: <br>
-**Requires API Key or External Credential:** [Not Specified] <br>
-**Credential Type(s):** [None identified] <br>
+**Requires API Key or External Credential:** [No] <br>
+**Credential Type(s):** [None] <br>
 
 Do not include secrets in prompts/logs/output; use least-privilege credentials; rotate keys as appropriate. <br>
 
@@ -27,7 +27,7 @@ Mitigation: Review and scan skill before deployment. <br>
 ## Reference(s): <br>
 - [Packed Sequences Documentation](docs/training/packed-sequences.md) <br>
 - [Performance Tuning Guide](docs/performance-guide.md) <br>
-- [Megatron Bridge Documentation](https://docs.nvidia.com/nemo/megatron-bridge/latest/) <br>
+- [Hierarchical Context Parallel](docs/training/hierarchical-context-parallel.md) <br>
 
 
 ## Skill Output: <br>
@@ -43,15 +43,15 @@ Mitigation: Review and scan skill before deployment. <br>
 
 
 ## Evaluation Tasks: <br>
-Evaluated against 1 task (1 positive) in isolated k8s-sandbox pods with dataset digest sha256:57d3c088. <br>
+Evaluated against 1 task (1 positive) using evaluator version 1.2.7 in local environment with Tier 3 live agent evaluation. <br>
 
 ## Evaluation Metrics Used: <br>
 Reported benchmark dimensions: <br>
-- Security: Whether the skill is safe to use (no unsafe operations, secret leakage, or unauthorized access). <br>
-- Correctness: Whether the skill produces correct answers against reference ground truth. <br>
-- Discoverability: Whether the right skill is loaded and activated when needed. <br>
-- Effectiveness: Whether the skill helps complete the user's goal and expected workflow. <br>
-- Efficiency: Whether the skill avoids wasted tool or skill usage. <br>
+- Security: Whether the skill is safe to use (unsafe operations, secret leakage, unauthorized access). <br>
+- Correctness: Whether the answer produced is correct against the reference answer. <br>
+- Discoverability: Whether the right skill was loaded and executed when needed. <br>
+- Effectiveness: Whether the skill helped complete the user's goal and expected workflow (goal accuracy + behavior check). <br>
+- Efficiency: Whether the skill avoided wasted tool or skill usage (routing quality and productive tool use). <br>
 
 Underlying evaluation signals used in this run: <br>
 - `security`: Unsafe operations, secret leakage, and unauthorized access. <br>
@@ -66,17 +66,12 @@ Underlying evaluation signals used in this run: <br>
 ## Evaluation Results: <br>
 | Measure | Claude Code (Baseline → Skill Uplift) | Codex (Baseline → Skill Uplift) |
 |---|---:|---:|
-| Overall | 37% → 91% (+54 points) | 68% → 86% (+18 points) |
+| Overall | 60% → 98% (+38 points) | 56% → 96% (+40 points) |
 | Security | 100% → 100% (±0 points) | 100% → 100% (±0 points) |
-| Correctness | 0% → 100% (+100 points) | 100% → 100% (±0 points) |
-| Discoverability | 50% → 100% (+50 points) | 50% → 88% (+38 points) |
-| Effectiveness | 0% → 66% (+66 points) | 74% → 45% (-29 points) |
-| Efficiency | 35% → 90% (+55 points) | 17% → 100% (+83 points) |
-
-## Testing Completed: <br>
-**[x] Agent Red-Teaming** <br>
-**[ ] Network Security** <br>
-**[ ] Product Security** <br>
+| Correctness | 0% → 100% (+100 points) | 40% → 100% (+60 points) |
+| Discoverability | 100% → 100% (±0 points) | 50% → 88% (+38 points) |
+| Effectiveness | 0% → 88% (+88 points) | 40% → 94% (+54 points) |
+| Efficiency | 100% → 100% (±0 points) | 50% → 100% (+50 points) |
 
 ## Skill Version(s): <br>
 1.0.0+b7643bd (source: pyproject.toml) <br>