feat(sc): support PPO in single controller by yuki-97 · Pull Request #3773 · NVIDIA-NeMo/RL

feat(sc): add the driver-side TQValue for the PPO critic

yuki-97

committed

5b7f9dd

feat(sc): add the worker-side TQ value forward for the PPO critic, with unit tests

yuki-97

committed

b5ba0eb

refactor: move the PPO estimator config to a BaseModel, pin estimator names to Literals

yuki-97

committed

f7e90f3

feat(sc): add the ppo algorithm block to the SingleController config

yuki-97

committed

d597342

feat(sc): build the PPO value model in SingleController setup, validate the algo block first

yuki-97

committed

ec4ac13

refactor: name the SingleController test modules after what they test

yuki-97

committed

f60ec32

feat(sc): accept PPO runs in the SingleController entrypoint, make ppo.async_ppo nullable

yuki-97

committed

a638ccc

feat(sc): size the train cluster for the critic, rename the GAE logprob kwargs to match GRPO's

yuki-97

committed

55c73ac

feat(sc): run the PPO critic stages in the SingleController train pump

yuki-97

committed

596a4bd

feat(sc): skip the weight sync during PPO critic warmup, require the in_order sampler

yuki-97

committed

74738e5

feat(sc): run ppo.ppo_epochs optimizer steps per RL step, fix the stale weight-sync test assertions

yuki-97

committed

8edca3d

feat(sc): widen the sampler lookahead during PPO critic warmup, port the remaining warmup behaviours from ppo.py

yuki-97

committed

eab2fbe

feat(sc): widen the sampler lookahead during PPO critic warmup, port the remaining warmup behaviours from ppo.py

yuki-97

committed

7e9d9a4

test(sc): add an async PPO SingleController functional test and its Megatron exemplar

yuki-97

committed

a63dcc9

test(sc): add an async PPO SingleController nightly recipe

yuki-97

committed

42f7062

pyrefly

yuki-97

committed

e95ce03

test(sc): regroup the SingleController L1 functional test list, raise the nightly GPU-hour cap to 4190

yuki-97

committed

59b2845

fix(sc): mark returns/values token-aligned, migrate the data_plane configs to backend-specific blocks, correct the async SC nightly metric step indices

yuki-97

committed

b9bd64c

fix(sc): reject the shaping knobs the SingleController path does not implement, stop max_num_epochs<=0 from zeroing max_num_steps, correct the PPO recipe's buffer-size comment

yuki-97

committed

4025501

docs(sc): document PPO on the SingleController path, correct the stale checkpointing and reward-shaping feature gaps

yuki-97

committed

0ef34c8

fix(sc): re-enable the DDP forward pre-hook in finish_train_step

yuki-97

committed

aa06643

chore(sc): restore the L1 functional test rationale, put the SC-utils modules back in pyrefly project-includes, add the value stages to the data-flow diagram

yuki-97

committed

0b59fe4

fix(sc): require a Megatron critic and reject drop budgets under PPO, hoist the warmup-lookahead guard onto the GRPO path, unprefix shared validation messages, retune the nightly thresholds

yuki-97

committed

c253574

test(sc): remove four duplicated PPO unit tests, parametrize five near-identical groups

yuki-97

committed

018cc7e

test(sc): set the first-train-step pre-hook state on the fabricated worker, cover the finish-path re-enable

yuki-97

committed

a091a6f

docs(sc): document the PPO value stages and critic warmup, note the PPO-only drop-budget constraint

yuki-97

committed

842adb9

test(sc): cover the PPO warmup-lookahead, negative-KL, and gate-window guards, make the warmup top-k assertion observable, tighten the nightly GPU-hour cap to 4149

yuki-97

committed

91c7cc9

fix(sc): reject colocated generation on the SingleController path

yuki-97

committed

ba8357a

fix(sc): move the exactly-one-algorithm-block check onto MasterConfig, guard ppo.async_ppo=null in run_ppo.py, make the advantage-estimator logprob args keyword-only, document the colocated branch …

yuki-97

committed

7d1f222

fix(sc): park the policy before the critic stages when neither logprob is required

yuki-97

committed

6d303d3

Navigation Menu

Uh oh!

feat(sc): support PPO in single controller - #3773

feat(sc): support PPO in single controller#3773
yuki-97 merged 31 commits into
mainfrom
yukih/sc-ppo

Commits on Aug 25, 2026

feat(sc): add the driver-side TQValue for the PPO critic

feat(sc): add the worker-side TQ value forward for the PPO critic, with unit tests

refactor: move the PPO estimator config to a BaseModel, pin estimator names to Literals

feat(sc): add the ppo algorithm block to the SingleController config

feat(sc): build the PPO value model in SingleController setup, validate the algo block first

refactor: name the SingleController test modules after what they test

feat(sc): accept PPO runs in the SingleController entrypoint, make ppo.async_ppo nullable

feat(sc): size the train cluster for the critic, rename the GAE logprob kwargs to match GRPO's

feat(sc): run the PPO critic stages in the SingleController train pump

feat(sc): skip the weight sync during PPO critic warmup, require the in_order sampler

feat(sc): run ppo.ppo_epochs optimizer steps per RL step, fix the stale weight-sync test assertions

feat(sc): widen the sampler lookahead during PPO critic warmup, port the remaining warmup behaviours from ppo.py

feat(sc): widen the sampler lookahead during PPO critic warmup, port the remaining warmup behaviours from ppo.py

test(sc): add an async PPO SingleController functional test and its Megatron exemplar

test(sc): add an async PPO SingleController nightly recipe

pyrefly

test(sc): regroup the SingleController L1 functional test list, raise the nightly GPU-hour cap to 4190

fix(sc): mark returns/values token-aligned, migrate the data_plane configs to backend-specific blocks, correct the async SC nightly metric step indices

fix(sc): reject the shaping knobs the SingleController path does not implement, stop max_num_epochs<=0 from zeroing max_num_steps, correct the PPO recipe's buffer-size comment

docs(sc): document PPO on the SingleController path, correct the stale checkpointing and reward-shaping feature gaps

fix(sc): re-enable the DDP forward pre-hook in finish_train_step

chore(sc): restore the L1 functional test rationale, put the SC-utils modules back in pyrefly project-includes, add the value stages to the data-flow diagram

fix(sc): require a Megatron critic and reject drop budgets under PPO, hoist the warmup-lookahead guard onto the GRPO path, unprefix shared validation messages, retune the nightly thresholds

test(sc): remove four duplicated PPO unit tests, parametrize five near-identical groups

test(sc): set the first-train-step pre-hook state on the fabricated worker, cover the finish-path re-enable

docs(sc): document the PPO value stages and critic warmup, note the PPO-only drop-budget constraint

test(sc): cover the PPO warmup-lookahead, negative-KL, and gate-window guards, make the warmup top-k assertion observable, tighten the nightly GPU-hour cap to 4149

fix(sc): reject colocated generation on the SingleController path

fix(sc): move the exactly-one-algorithm-block check onto MasterConfig, guard ppo.async_ppo=null in run_ppo.py, make the advantage-estimator logprob args keyword-only, document the colocated branch …

fix(sc): park the policy before the critic stages when neither logprob is required

Commits on Aug 26, 2026

fix unit test