[WIP] Document 1M-token training and add a context-parallelism example

[qgallouedec]

Two things, both about training on sequences far longer than the usual few thousand tokens:

• a "Training at 1M tokens and beyond" section in docs/source/distributing_training.md, with the configurations I verified and the levers in the order you hit them;
• examples/sft_long_context/, a runnable example that trains a book-length sequence per step on one 8xH100 node, plus examples/accelerate_configs/context_parallel_8gpu.yaml.

Measured

One 8xH100 node, bf16, per_device_train_batch_size=1, loss_type="chunked_nll" (the default):

Model	Sequence length	Step time	Peak GPU memory
Qwen3-0.6B	1M	137 s	27.9 GB
Qwen3-8B	1M	364 s	56.2 GB
Qwen3-30B-A3B (MoE)	1M	483 s	46.0 GB
Qwen3-32B	1M	1295 s	60.8 GB
Qwen3-0.6B	4M	2135 s	73.7 GB

Context length scales with node count (Qwen3-8B: 1M on 1 node at 364 s, 2M on 2 nodes at 696 s, 4M on 4 nodes at 1346 s), because each GPU keeps the same shard of the sequence.

It is real training, not just "it fits": Qwen3-30B-A3B with YaRN x32 on PG-19 books at 1M context goes from loss 4.88 to 2.37 in six steps.

Assumptions

Everything here is written as if the rest of the series has landed. Each of these is a separate PR, and the text or the example changes if one of them does not merge:

assumption	where it shows up	if it does not land
huggingface/accelerate#4175 — FSDP2 activation_checkpointing_offload	context_parallel_8gpu.yaml sets fsdp_activation_checkpointing_offload: true; the guide lists it as lever 3	drop the line from the YAML; models at or above 8B no longer fit at 1M
huggingface/transformers#48163 — cuDNN SDPA preference on Hopper	the step times above; neither the guide nor the example selects a backend	both need an SFTTrainer subclass wrapping training_step in sdpa_kernel([SDPBackend.CUDNN_ATTENTION], set_priority=True) — covering the backward too, since checkpoint recompute must pick the same backend — and every step time above is ~1.7x larger
huggingface/accelerate#4177 — refuse sliding-window models under CP	the constraints section says accelerate rejects them	they are silently trained with full causal attention instead, and the section has to say so
#6843 — refuse packing under CP	the constraints section says TRL raises	the "Do not use packing" bullet becomes advice rather than a guarantee
transformers cp-eval-loss-scaling — apply CP to the evaluation path	not mentioned, but any user who evaluates during a long-context run gets eval_loss multiplied by cp_size	worth a warning in the guide

Two more that are not assumptions, just constraints of what exists today, both stated in the example:

• MLP sequence tiling (lever 6) is a user-side monkeypatch, not a feature. It is what makes Qwen3-32B fit.
• Context parallelism expresses only full causal attention, so gpt-oss, Gemma 3/4, Mistral and Qwen3.5 and later cannot be used. Qwen3.5+ is the interesting one: three quarters of its layers are linear attention, which is a different problem from the sliding-window case and is not covered by Refuse context parallelism for models with sliding-window or chunked attention layers accelerate#4177's fix, only by its refusal.

---

Note

Low Risk
Documentation and example scripts only; no trainer or library behavior changes in this diff.

Overview
Adds a Training at 1M tokens and beyond section to the distributed-training guide: verified step-time and memory tables (up to 4M on one node, multi-node scaling), ordered memory/speed levers (chunked_nll, FSDP activation offload, CP alltoall + bf16 merge, param offload, MLP sequence tiling, YaRN via rope_parameters), optional selective checkpointing that keeps attention outputs, and explicit CP limits (full causal attention only; no packing).

Ships a runnable Qwen3-8B @ 1,048,576 tokens example under examples/sft_qwen_3_8_1M_context/ with an 8-GPU FSDP2 context-parallel Accelerate config (activation checkpointing + host offload), PG-19 concatenation (not packing=True), and Transformers v5 RoPE overrides.

^{Reviewed by Cursor Bugbot for commit 3132df9. Bugbot is set up for automated code reviews on this repo. Configure here.}

[bot-ci-comment]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: bae6acb881

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".