Skip to content

[Umbrella] Long-context SFT fixes (do not merge) - #4184

Draft
qgallouedec wants to merge 6 commits into
mainfrom
long-context-umbrella
Draft

[Umbrella] Long-context SFT fixes (do not merge)#4184
qgallouedec wants to merge 6 commits into
mainfrom
long-context-umbrella

Conversation

@qgallouedec

Copy link
Copy Markdown
Member

Install-only umbrella for training on 1M+ token sequences: merges

so colleagues can install one branch. Not meant to merge.

#4175 is what makes Qwen3-8B fit at 1,048,576 tokens on one 8xH100 node: without it the run OOMs. Paired with huggingface/transformers#48163 the step time is 373 s at 56.2 GB per GPU, against 582 s for Qwen3-4B (the largest that fits) on unmodified main.

Used by the 1M-token context-parallelism example in huggingface/trl#6846.

@HuggingFaceDocBuilderDev

Copy link
Copy Markdown

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants