Disallow truncate_history_thinking=true for NeMo-RL rollouts

[aroshanghias-nvd]

Summary

truncate_history_thinking=true does not do what it is supposed to do in NeMo-RL rollouts, and it should not be supported as a rollout configuration.

We cannot simply truncate reasoning traces from previous turns in the current cumulative trajectory representation. Generation has seen those reasoning tokens, so training must see them too. Removing them only when constructing the training sequence would make the learner evaluate sampled tokens under a different causal context from the behavior policy.

Current NeMo-RL therefore preserves the reasoning:

• Native rollouts concatenate the exact generated token history;
• NeMo-Gym model servers restore the exact generated prefix after chat-template rendering.

This is necessary for correct on-policy training, but it means truncate_history_thinking=true is incompatible with the rollout contract.

The flag should not be repurposed as a context-compaction mechanism. Proper context compaction must be configured and represented explicitly, as in PR #3910, with exact per-call evidence and separate physical training traces.

Why reasoning cannot simply be removed

Consider two policy calls:

Call 1:
prompt   = U1
response = R1 -> A1

Call 2 after removing reasoning history:
prompt   = U1 -> A1 -> U2
response = R2 -> A2

Training A1 must include R1 in its causal context because A1 was generated after and conditioned on R1.

If the second call genuinely uses the compacted prompt, R2 -> A2 must instead be trained without R1 in its causal context. These calls cannot be represented correctly as the single flat sequence:

U1 -> R1 -> A1 -> U2 -> R2 -> A2

Changing context between calls requires separate physical traces containing the exact prompt tokens, sampled response tokens, and behavior-policy logprobs for each call. A model-specific Jinja chat-template option is not sufficient to declare or transport that training contract.

Current behavior

Native NeMo-RL rollout

Native multi-turn rollout does not reapply the chat template between policy calls. It stores the exact generated token IDs, appends environment tokens, and concatenates the complete token history for the next generation. Reasoning generated during an earlier call therefore remains in subsequent model inputs regardless of truncate_history_thinking=true.

Relevant code:

• nemo_rl/experience/rollouts.py:542
• nemo_rl/experience/rollouts.py:931
• nemo_rl/experience/rollout_manager.py:587

NeMo-Gym rollout

NeMo-Gym may logically rerender the conversation without historical reasoning. NeMo-RL's model-serving layer then restores the exact previous prompt and generated-token prefix before the next generation. This prefix restoration preserves the on-policy cumulative trajectory, but also means reasoning removed by the template is added back to the actual model input.

Relevant code:

• nemo_rl/models/generation/vllm/vllm_worker_async.py:411
• nemo_rl/models/generation/vllm/vllm_worker_async.py:508
• nemo_rl/models/generation/vllm/vllm_worker_async.py:818
• nemo_rl/models/generation/openai_server_utils.py:69
• nemo_rl/models/generation/openai_server_utils.py:104

Equivalent prefix preservation exists in the TRT-LLM and Dynamo serving paths.

Why this is a bug

NeMo-RL accepts truncate_history_thinking=true without explaining that the requested behavior is incompatible with cumulative RL rollouts.

A user can reasonably expect previous-turn reasoning to be absent from later model calls. In practice, native rollout retains it and NeMo-Gym serving restores it. The requested behavior is therefore silently unavailable.

This is not currently a training-correctness failure: NeMo-RL preserves the generated prefix specifically to maintain correct on-policy training. It is a configuration-semantics and observability bug.

Expected behavior

For NeMo-RL rollout and training paths:

1. Reject an explicit truncate_history_thinking=true during configuration or startup.
2. Ensure cumulative rollouts preserve reasoning history, including when a model template would otherwise default to truncation.
3. Remove truncate_history_thinking as a user-selectable rollout option from NeMo-RL recipes once preservation is enforced internally.
4. Do not interpret this chat-template flag as a request for context compaction.
5. Direct users who need context removal to the explicit context-compaction rollout path and trace contract.
6. Add tests confirming that rollout configurations containing true fail with a clear error.

A possible error is:

truncate_history_thinking=true is not supported for NeMo-RL rollouts.

NeMo-RL must preserve the exact causal context used to generate trainable
tokens. Use the explicit context-compaction rollout path when context must
change between model calls.

Scope

This does not necessarily require deleting generic chat-template keyword support from tokenizers or offline independently rendered training pipelines.

The restriction is that truncate_history_thinking=true must not be accepted as NeMo-RL rollout behavior. Context compaction should retain its own explicit configuration, transport contract, validation, and physical-trace training representation.