[0z5a]

Purpose

Add Huginn-0125 to PPO, GRPO, Decoupled PPO and categorical Flow-DPPO. Actor/critic replay preserves per-completion latent seeds, the prelude/core/coda and sandwich norms; conversion and full policy publication preserve tied embeddings and FP32 RoPE.

Shared backend/objective wiring is inherited from #463 (45f2dd7). Huginn layer at the GPU-tested source: 20 files, +554/-38. Updated onto merged #460/#454 at 8c8b8e8e; current base diff: 57 files, +2772/-33.

Test Plan

Recipe. Tested 71860e5, official revision bb6621b, RLT d2a3589, two RTX 5090s, Torch 2.12.1/CUDA 13.0. Real Ray/MCore, FP32/native SGD (momentum 0), recomputation, full disk publication, fixed 32 recurrences; four prompts, up to 16 response tokens/completion, controlled binary rewards.

Compare three continuous rollouts against two saved rollouts plus one in fresh processes. This validates reproducibility; default FP16 training and model quality remain unmeasured.

Test Result

Algorithm	Steps/role	Complete final DCP + fresh resume	Max selected-logprob error
PPO	3	actor/critic 78 chunks each, exact	1.562e-5
GRPO	3	actor 78 chunks, exact	3.147e-5
DPPO	3	actor/critic 78 chunks each, exact	1.562e-5
Flow-DPPO	6	actor/critic 78 chunks each, exact	1.478e-5

Optimizer/scheduler/RNG, data cursor, tokens, latent traces and publication digests match; all cases exit 0. Earlier CPU suites: 70 provider/recipe + 110 shared regressions passed. Merge dff03a95 passes 131 HF conversion/collector/argument CPU regressions, 0 skipped, plus Ruff/Black/isort for all 16 merged Python files. Huginn provider, conversion, engine, loss and recipe blobs retain the GPU-tested versions. DCO/docs pass; automatic CI #1448 is queued.

PPO Graph comparison: three complete matched pairs (AB/BA/AB), same Triton backend, seeds and full publication; only Graph mode changes. Median of three run medians per arm; warm measurements exclude capture.

Metric	Eager (s)	Graph (s)	Eager/Graph	Duration change
Complete 3-iteration phase, including startup/save/publication/observer	483.075	484.960	0.996x	+0.39%
Warm rollout	5.584	5.171	1.080x	-7.39%

Warm rollout improves; complete training time is effectively unchanged. Graph fresh resume and cross-arm full-state/token comparisons pass; zero fallbacks or warm recaptures. Timings cover PPO only.

[read-the-docs-community]

Documentation build overview

📚 vime | 🛠️ Build #34966656 | 📁 Comparing dff03a9 against latest (791f101)

  🔍 Preview build  

51 files changed · + 6 added · ± 45 modified

+ Added

• advanced/draft-feature-collection.html
• _examples_synced/huginn/README.html
• _examples_synced/looped_grpo/README.html
• _examples_synced/looped_ppo/README.html
• _examples_synced/nanbeige/README.html
• _examples_synced/ouro/README.html

± Modified

• genindex.html
• index.html
• search.html
• advanced/arch-support-beyond-megatron.html
• advanced/delta-weight-sync.html
• advanced/external-rollout-engines.html
• advanced/fault-tolerance.html
• advanced/low-precision.html
• advanced/megatron-config.html
• advanced/observability.html
• and 35 more...

[gemini-code-assist]

Code Review

This pull request introduces a native vLLM-RLT rollout backend supporting recurrent policy training (PPO, GRPO, DPPO, Flow-DPPO) for Ouro, Nanbeige, and Huginn models, along with a categorical Flow-DPPO extension, robust vLLM response parsing, and improved NVLink detection. The review feedback highlights several critical runtime issues that need to be addressed, including a TypeError from an invalid enable_gqa argument in F.scaled_dot_product_attention, missing imports of inspect and copy causing NameErrors, a potential KeyError in the recipe launcher for unsupported model types, and a potential TypeError in the RLT engine if exit_depths is null. Additionally, the reviewer recommends using F.kl_div to avoid numerical instability (NaN values) during manual KL divergence calculation.

[gemini-code-assist]

F.scaled_dot_product_attention does not accept an enable_gqa argument in standard PyTorch. Passing enable_gqa=True will raise a TypeError and crash at runtime. Standard SDPA natively supports Grouped Query Attention (GQA) based on the input shapes, so this argument should be removed.

        attended = (\n            F.scaled_dot_product_attention(\n                q.transpose(0, 1).unsqueeze(0),\n                k.transpose(0, 1).unsqueeze(0),\n                v.transpose(0, 1).unsqueeze(0),\n                is_causal=True,\n            )\n        )

[gemini-code-assist]

inspect is used on line 102 but is not imported in this file, which will cause a NameError at runtime. We should import inspect locally or at the top of the file.

            import inspect\n            custom_model_provider = load_function(args.custom_model_provider_path)

[gemini-code-assist]

copy is used on line 393 but is not imported in this file, which will cause a NameError at runtime. We should import copy locally or at the top of the file.

    import copy\n    prepared_sample = copy.copy(sample)

[gemini-code-assist]

Calculating the KL divergence manually using old_log_probs.exp() * (old_log_probs - log_probs) can lead to NaN values if old_log_probs contains -inf (zero probability tokens), because 0.0 * -inf is NaN in PyTorch. Using F.kl_div with log_target=True is a much safer, cleaner, and standard way to compute the categorical KL divergence while avoiding numerical instability.

Suggested change

	divergence = (old_log_probs.exp() * (old_log_probs - log_probs)).sum(-1).clamp_min(0)
	divergence = F.kl_div(log_probs, old_log_probs, log_target=True, reduction="none").sum(-1).clamp_min(0)

[gemini-code-assist]

If output.exit_depths is None, calling len(output.exit_depths) will raise a TypeError instead of the intended RuntimeError. We should add a check to ensure output.exit_depths is not None before calling len on it.

                or output.exit_depths is None\n                or len(output.exit_depths) != response_length

[gemini-code-assist]

If family is not found in providers, a KeyError will be raised on line 50. We should validate family and raise a clean parser error with a list of supported model types.

    providers = {"ouro": "ouro", "nanbeige": "nanbeige", "huginn_raven": "huginn"}\n    if family not in providers:\n        parser.error(f"Unsupported model type: {family}. Supported types: {list(providers.keys())}")\n    provider = providers[family]