fix(sc): apply advantage clipping and reward scaling in the advantage stage

[tianyi-zhang-02]

What does this PR do?

Makes SingleController honor the existing GRPO advantage-clipping and reward-scaling settings.

• grpo.reward_scaling uses the same scale_rewards helper and runs before advantage estimation.
• grpo.advantage_clip_low/high uses _clip_grpo_advantages after pre-clip diagnostics are logged and before the data-plane write, matching the legacy and synchronous GRPO paths.
• clipping is GRPO-only; a similarly named extra field on PPO cannot activate it.

Both changes are no-ops at their shipped defaults. Reward shaping and message-level advantage penalties remain out of scope because SC does not yet carry the response-length/message fields they require.

Validation

Final SHA: e8686a6f5149b48898a3abe6b322d38516f9546c, based on upstream main at ccbcd4cc5.

2×H100 end to end

Runpod Secure Cloud, Qwen3-0.6B, Megatron policy/reference, real vLLM rollout, TransferQueue data plane, and NCCL_NVLS_ENABLE=0:

Run	Result
default knobs, 2 steps	train_steps=2, trainer_version=2; two refits succeeded; no stale/drop/abort events
scaling + clipping enabled, 1 step	reward scaling 0..1 → -1..1; pre-clip advantages [-0.8096, 1.2351]; clip bounds [-0.5, 0.5]; loss 0.1040; grad norm 2.6518

The enabled run used Reinforce++ without baseline subtraction so a uniform zero reward could not make the clipping path vacuous.

Unit

• PPO setup and SingleController actor suites: 108 passed
• covers enabled/disabled scaling, clip ordering, data-plane values, and the PPO guard

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[tianyi-zhang-02]

Heads-up on merge order, with the resolution written out.

Four of my open PRs edit the same twelve-line block — the unsupported knob list in _validate_algo_settings:

• fix(sc): preserve DatumSpec loss_multiplier through rollouts #3786 removes overlong_filtering from it (build_sample_mask now applies it)
• fix(sc): apply advantage clipping and reward scaling in the advantage stage #3787 removes reward_scaling from it (_advantage_stage now applies it)
• feat(sc): run the distillation teacher in the SingleController train pump #3846 / test(sc): add the distillation recipe and functional test for SingleController #3849 wrap the list in an is_distillation_run guard, because DistillationConfig defines none of those knobs and the comprehension would AttributeError

I checked every pair by actually merging them. #3786 × #3787 is clean now — they only ever conflicted on the comment wording and on both adding a test before the same anchor, and both are fixed. The remaining four pairs are a genuine textual conflict on that block, and no restructuring avoids it: extracting or reordering the list conflicts just as hard.

So rather than force a stack, here is the resolution. Whichever order they land in, this is the merged form:

    # An enabled one here describes shaping this run does not do. An entry
    # leaves this list when the SC path starts implementing it -- rejecting a
    # knob is only right while nobody honours it.
    #
    # DistillationConfig defines none of them: there is no reward to shape or
    # filter on, so the list cannot even be evaluated on that path.
    unsupported = (
        []
        if is_distillation_run(master_config)
        else [
            name
            for name, enabled in (
                ("use_dynamic_sampling", algo_cfg.use_dynamic_sampling),
                ("reward_scaling", algo_cfg.reward_scaling.enabled),
                ("reward_shaping", algo_cfg.reward_shaping.enabled),
            )
            if enabled
        ]
    )

I built that merge locally and ran the SC and config suites against it — 1084 passed. Happy to rebase whichever ones are left once the first lands; just say which order you want.