feat: forward rollout session IDs to Dynamo

[jthomson04]

What does this PR do ?

Forward stable rollout session IDs to Dynamo for NeMo-Gym and direct NeMo-RL rollouts.

• NeMo-Gym reuses its stable session UUID and sends it through the Dynamo token wrapper.
• Direct rollouts create one UUID per trajectory attempt and reuse it across turns and HTTP retries.
• Sibling trajectories and complete-trajectory retries receive different IDs.
• Only Dynamo receives X-Dynamo-Session-ID; other backends and Dynamo routing policy are unchanged.
• Pin Gym to NVIDIA-NeMo/Gym@cc8ec02281a590ff9e5111acae8d70fe1dd354ab for optional session-header support.

This PR is transport-only. Session affinity remains opt-in through the existing frontend_args.extra_cli_args escape hatch. It does not add an affinity default, tracing controls, or a new routing policy.

Issues

None.

Usage

No user configuration is required for header forwarding. Dynamo-backed Gym and direct rollouts add the session header automatically. Users who want session affinity can configure the existing Dynamo frontend CLI escape hatch.

Before your PR is "Ready for review"

Pre checks:

• Read and followed the contributor guidelines.
• Added focused tests for Gym configuration, wrapper forwarding, direct rollout identity, and Dynamo HTTP forwarding.
• Functional and GPU tests were not run. The focused CPU unit tests pass.
• No documentation update is required because this does not add a user-facing option.

Additional Information

Validation:

• Gym VLLM model test file: 126 passed.
• NeMo-RL affected unit-test files: 138 passed.
• Scoped Gym and NeMo-RL pre-commit checks passed, including pyrefly.
• uv sync --locked passed.
• The new Gym pin is a fast-forward from the current upstream NeMo-RL Gym pin.
• A fresh fetch of the exact Gym SHA from the official Gym URL passed.
• No GPU test was run.

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[github-actions]

✅ Submodule Fast-Forward Check Results

Check based on commit: 6ef3f99 (PR #3856 from jthomson04/dynamo-session-id-forwarding)

✅ Submodules that are properly updated:

Gym: ✅ PR branch is ahead of main branch (fast-forward)

All submodule changes look good! ✨

[jthomson04]

/ok to test 2af6e1b

[copy-pr-bot]

Auto-sync is disabled for ready for review pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.