Skip to content

Document which chat template the released RL Zero checkpoints used - #1929

Open
JOhnsonKC201 wants to merge 2 commits into
allenai:mainfrom
JOhnsonKC201:rlzero-template-note
Open

JOhnsonKC201 wants to merge 2 commits into
allenai:mainfrom
JOhnsonKC201:rlzero-template-note

Conversation

@JOhnsonKC201

Copy link
Copy Markdown

Refs #1899.

Following up on @abhishekraok's reply in the issue, this documents which chat template the released RL Zero checkpoints were trained with, since the scripts on main now select a different one.

What I checked before writing it down:

  • olmo_thinker in dataset_transformation.py at d928a7c (the commit the README table lists for the RL Zero runs) is byte-identical to the chat_template.jinja on the Hub for Olmo-3-7B-RL-Zero-Code, -IF, -General and -Mix. All four Hub files are the same 1652 bytes.
  • olmo_thinker_rlzero and olmo_thinker_code_rlzero don't exist at d928a7c. They came in with rlzero template and fixes to rlzero scripts #1216 about six weeks later.
  • olmo_thinker on main is no longer the same string as at d928a7c, so switching the flag back on main would not give the released template either.
  • Olmo-3-7B-RL-Zero-Math is different: its Hub template is a simple prompt, the same as olmo_thinker_rlzero except it says "math problem".

The claim that the Code and IF wandb runs record chat_template_name=olmo_thinker is taken from the issue thread, I can't see those runs.

Changes:

  • scripts/train/olmo3/README.md: new "RL Zero chat templates" section with the above.
  • 7b_rlzero_code.sh and 7b_rlzero_instruction_following.sh: a three line comment at the top pointing to that section.

I left the --chat_template_name flags alone. Whether the scripts should go back to the release recipe or stay on the simple templates seems like your call, and the README now says which is which either way. Happy to change the flags or add the same comment to the math and mix scripts if you'd prefer.

No changes under open_instruct/, so no changelog entry.

The RL Zero Code and IF scripts on main select the simple rlzero templates added in allenai#1216, but the released checkpoints were trained with olmo_thinker at d928a7c and ship that template on the Hub. Say so in the olmo3 README and at the top of the two scripts.
Codex · codex://threads/01a0d61a-a285-7931-ab3a-39b0134c4a25

Replace invalid checkout-and-launch advice with historical code and script
sources. Distinguish recorded runs, shipped templates and unverified
checkpoint/evaluation provenance; identify the original Math template
and later 3.1 recipe. Add General and Math script warnings.

@abhishekraok abhishekraok left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Codex · codex://threads/01a0d61a-a285-7931-ab3a-39b0134c4a25

Reviewed corrected head 0da772f9e3e906f3ee774e1641d66c33154a8016, including an independent Claude review with no remaining findings.

The update removes the invalid checkout-and-launch example, separates recorded Code/IF/General configurations from Mix's shipped-template evidence, identifies the original Math olmo_thinker_dapo template, and links historical script snapshots while preserving the remaining reproduction limits. It also warns that General's template definition and the current Math recipe have changed.

Validation: 54 CPU tests pass; style, quality and type checks pass; the four edited shell scripts parse; documentation builds; the five public release templates match the historical definitions stated in the note. Fresh GitHub CI is still pending on this head. No training run was launched, and the exact paper-evaluation provenance remains open in template-mismatch (#1899).

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants