feat: super v3.5 posttraining with flexible length penalties

[arendu]

What does this PR do ?

Adds configurable length penalties/bonuses for rollouts (reward-mutation and GDPO reward-feature modes) to reduce verbosity in RLVR training — including group-relative length scaling, zMAD outlier penalties, profile-band multipliers with per-prompt or global {a, b, f} bands, and per-prompt profile-gated group-relative scaling — applied
in both sync and async GRPO paths.

Issues

None. This is the super-v3.5-posttraining port of the length-adjustment work targeting main in #3604, plus fixes made since (async rollout propagation, profile-band clamp at f, global profile-band defaults, unit tests).

Usage

# Reward-mutation mode: dense group-relative length pressure, gated per prompt
grpo:
  length_bonus:
    default:
      enabled: true
      length_type: tokens
      group_total_length_penalty_coeff: 0.1
      group_length_penalty_profile_gate: true          # penalty only when group mean length
      group_length_penalty_profile_gate_channel: total #   exceeds the prompt's profiled target
      group_length_penalty_profile_gate_field: a
    profile_band:            # global {a,b,f} fallback for datasets without per-prompt bands
      enabled: true
      defaults:
        total: {a: 18000, b: 24000, f: 0.9}
    agent_overrides:
      instruction_following_simple_agent:
        enabled: false

# GDPO feature mode: same signals as a separate reward feature, env reward untouched
grpo:
  adv_estimator:
    name: gdpo
    reward_features:
      default:
        env_reward: 1.0
        length_adjusted_reward:
          group_total_length_penalty_coeff: 0.1
        think_count_delta: 1.0

Full algorithm reference: nemo_rl/utils/length_adjustments.md.

Before your PR is "Ready for review"

Pre checks:
- [x] Make sure you read and followed Contributor guidelines
- [x] Did you write any new necessary tests?
- [x] Did you run the unit tests and functional tests locally? Visit our Testing Guide for how to run tests
- [x] Did you add or update any necessary documentation? Visit our Document Development Guide for how to write, build and test the docs.
# Additional Information
* ...

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[copy-pr-bot]

Auto-sync is disabled for ready for review pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[yfw]

Local review by a multi-agent pass (RL/config, reward-math, bug-scan, tests, design, plus an adversarial verification round). Every numeric claim below was executed, not inferred.

The feature's length_bonus half looks sound — I verified that group-relative scaling preserves the group mean exactly, that degenerate groups (size 1, all-equal lengths, all-zero or all-negative rewards, zero lengths) are all safe with no NaN created and no division by zero, that _band_multiplier's boundaries are correct including the past-b clamp from 25df5bfcc, that the zMAD statistic uses the correct Iglewicz–Hoaglin constant, and that a config which doesn't opt in is an exact no-op on rewards. The 661-line design doc is genuinely above the bar for this repo.

The GDPO feature half is where the problems are, and two of them need fixing before merge: the dispatch hijack (which affects existing configs with no opt-in) and the fact that the documented config shape produces no length pressure at all. Details inline.

Follow-ups — not asking for these in this PR:

• _ZMAD_MIN_MAD_REL = 0.015 gates the whole channel rather than suppressing noise: lengths [20000, 20100, 19900, 20050, 19950, 32768] give MAD/median = 0.0037 < 0.015, so a modified-Z of 114.6 is missed — precisely the tight-distribution-with-one-runaway case the channel exists for. The justifying comment cites flag_reasoning_length_outliers, which doesn't exist anywhere in the repo.
• profiled_length_penalty has no positive-reward gate, unlike every other channel, so an all-wrong group gains within-group variance (and a gradient) where GRPO previously produced none.
• _apply_profile_band_multipliers never checks enabled, unlike Phase 1 at L353. Currently masked — skipped groups leave lengths at 0 and _band_multiplier(0, ch) returns 1.0 — but it's a live guard gap. Same root as Phase 2 printing coefficients the profile gate had already zeroed: _resolve_agent_params is re-resolved independently in three places and the copies have drifted.
• sync_rollout_actor.py:262 passes reward_penalty_config but not length_adjustment_config — the same omission b8ca5c331 fixed for the async collector. No shipped config reaches it today (nothing sets data_plane.enabled together with nemo_gym), so no impact, but a parametrised "every rollout producer threads the config" test would stop this recurring a third time.
• scripts/build_profile_band_dataset.py: ZeroDivisionError on empty input (after the output file is written), TypeError on an explicit "pass_rate": null, and a comment promising a "tiny parser" fallback that doesn't exist wrapped around an unreachable except ImportError. Also tools/ is the established home for standalone utilities.
• _resolve_agent_params can never return None, so three if params is None guards and the verbose [skipped] branch are unreachable; and its print() warning fires once per group per step (128 unsuppressable prints/step at num_prompts_per_step: 128) while the module already has a logger.

Generated by Claude Code

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.