feat(sc): apply reward penalties to Gym rollouts

[macandro96]

What does this PR do ?

Support reward penalties for gym rollouts.

Issues

List issues that this PR closes (syntax):

Usage

# Reward-zeroing penalties applied to NeMo-Gym rollout results.
reward_penalties:
  penalize_duplicated_reasoning: false
  penalize_empty_final_answer: false
  penalize_unwanted_tokens: false
  penalize_malformed_think_tag: false
  # Optional model/tokenizer-specific IDs. Example:
  # token_ids: {unwanted: [2], think_open: 12, think_close: 13}

Before your PR is "Ready for review"

Pre checks:

• Make sure you read and followed Contributor guidelines
• Did you write any new necessary tests?
• Did you run the unit tests and functional tests locally? Visit our Testing Guide for how to run tests
• Did you add or update any necessary documentation? Visit our Document Development Guide for how to write, build and test the docs.

Additional Information

• ...

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[macandro96]

Nice work extending the reward-penalty feature to the SingleController Gym path. Overall the wiring is clean: resolving reward_penalty_config once at setup enables the fail-fast test, and the batched _results_to_completions refactor keeps the singleton hook. Tests for the new SC surface pass locally (16 + 35).

Findings from the parallel review team (bug-finder, rl-expert, test-agent, design-reviewer, devil-advocate):

• 1 pre-existing test regression that will break CI (inline).
• 3 DRY / config-conventions suggestions (inline).
• 1 test-coverage suggestion (inline).
• Body: the PR description is the unfilled auto-template — please add a one-line summary and link the non-SC parity motivation (existing feature at grpo.py:2960 / :3893 / trajectory_collector.py:936).

Pre-commit passed clean. Local pytest runs:

• tests/unit/experience/test_rollout_manager.py: 16 passed
• tests/unit/single_controller/test_single_controller_setup.py: 35 passed
• tests/unit/experience/test_rollouts.py::test_rollout_manager_consumes_stream_and_restores_input_order: FAILED (see inline)

Generated by Claude Code

[yuki-97]

Traced the reward-penalty wiring for SC end-to-end (RewardPenaltyConfig -> validate_single_controller_config -> resolve_reward_penalty_config -> RolloutManager -> apply_reward_penalties -> compute_reward_penalty_metrics) — logic is correct and the prior self-review threads all genuinely landed. Two minor items below: a dead private wrapper left over from the streaming refactor, and a heads-up that main has since added PPO to SingleController, so this needs a small parity follow-up after rebase.

[macandro96]

/ok to test 2fe8a47

[yuki-97]

Re-reviewed at 2fe8a47. Both open threads check out against the current head:

• Dead _result_to_completion wrapper — deleted, and both mask-gate tests now call _results_to_completions([result])[0][0]. A repo-wide grep at head finds no definition and no caller left.
• PPO parity after the rebase — reward_penalties sits on the new grpo/ppo-Optional MasterConfig, the PPO SC exemplar carries the block, and neither validate_single_controller_config nor resolve_reward_penalty_config reads master_config.grpo, so the PPO SC path genuinely gets the penalties rather than silently dropping them.

One nit below, nothing blocking.

L0_Unit_Tests_Other is red at this head, needs to be fixed before merge.

[macandro96]

/ok to test 5e48ef6