[tengomucho]

What does this PR do?

This PR is to move vLLM from 0.23.0 to 0.29.0. The reason is to close the security advisories raised by dependabot and prepare a release with less CVE risks.
Since vLLM 0.23 was the last one accepting transformers 4.x, and update to transformers v5 was also necessary, together with the update to huggingface_hub v1.

Even if recent vLLM declares a dependency to torch==2.13.0, those seem to be metadata only, since vLLM only tests with one version of pytorch and supports only that version, but that can be overridden and that is what is done here, because the latest torch supported by SDK 2.31 is 2.9.1.

All the changes were done with an agent. The Claude description details for this PR follows.

Dependencies

package	before	after
transformers	4.57.1	5.17.0
vllm	0.23.0	0.29.0
huggingface_hub	>= 0.35.3	>= 1.0.0
peft	0.17.0	0.18.1
diffusers	0.35.*	0.37.*
compel	2.1.1	2.4.0

peft, diffusers and compel move to their first releases that work with transformers v5 —
the previous pins fail at import.

The neuronx_distributed shim

neuronx_distributed/pipeline/trace.py:43 imports transformers.utils.fx, deleted in
transformers v5 along with the rest of torch.fx support, so import neuronx_distributed
fails outright. The newest published wheel (0.19.28492) still has it.

_optimum_neuron_fx_shim.py appends a finder to sys.meta_path that serves a minimal
stand-in module. Being appended, it is consulted only once the regular finders have failed,
so transformers 4.x keeps its real module and the shim retires itself the day AWS guards
that import.

It is installed at interpreter startup by optimum-neuron-fx-shim.pth, because the
processes parallel_model_trace spawns import neuronx_distributed while unpickling their
work item, before any optimum module: tensor parallel export of encoder-decoder models dies
without it. Nothing is imported until something actually asks for transformers.utils.fx,
so the hook costs a few milliseconds and pulls in neither torch nor transformers.

Note that an editable install does not get the path configuration file, so on a development
checkout the shim only covers the current process.

Two bugs this surfaced

• SmolVLM vision positions. The patch position ids were bucketed from
  i/N * (1 - 1e-6), copied from transformers 4.57. The bucket boundaries sit exactly at
  multiples of 1/N, so that factor pushes every coordinate just below its own boundary:
  31 of 32 positions were shifted by one, in both dimensions. transformers v5 replaced the
  scaling with a clamp, which is what made the copy visibly wrong — the vision features
  differed from the reference by 58.8 against a magnitude of 114, and agree to 8e-05 now.
• --data_parallel_size reached vLLM. Every other option of optimum-cli neuron serve
  is declared in both spellings; this one only in the dashed form, so the underscore
  spelling fell through to vLLM, which served the model from its own data parallel engines,
  all of them claiming every visible core.

API changes

• RoPE: config.rope_theta and config.rope_scaling collapsed into a single
  config.rope_parameters dict, which Gemma3 nests per layer type. All call sites go
  through one helper. The deprecated config.rope_scaling now always returns a dict, so
  every is not None check on it had to move to the rope type — silently, had it been left.
• Traced outputs: transformers v4 turned its model outputs into a tuple while tracing
  (through config.torchscript), and the neuron models read them positionally. v5 dropped
  that, so the traced models returned dicts.
• Mixture of experts: the expert weights are now a single fused tensor per projection
  instead of one linear layer per expert (mixtral, qwen3_moe).
• Beam search is removed from the legacy seq2seq generation mixin: it was vendored
  transformers v4 code built on BeamSearchScorer. generate() rejects num_beams > 1
  explicitly. Exporting with num_beams > 1 still works, but the result can no longer
  generate.
• The question-answering pipeline is dropped, following its removal from transformers.
  NeuronModelForQuestionAnswering itself is untouched.
• Smaller replacements: download_url/is_remote_url, is_safetensors_available,
  HfFolder, get_preprocessor, find_pruneable_heads_and_indices,
  EncoderDecoderCache.from_legacy_cache, CLIPFeatureExtractor, the pipeline framework
  attribute, the GreedySearch*/Sample* output aliases, and configs no longer carrying
  pad_token_id or resolved generation parameters.

vLLM 0.29

WorkerBase.load_model gained a load_dummy_weights keyword, the entrypoint modules moved
under vllm.entrypoints.launchers, and the engine core now asks every worker which KV cache
layouts it supports — answered through vLLM attention backends, of which a neuron worker has
none. The sampler override and the picklable ModelConfig patch still match 0.29 exactly.

For the record: vLLM 0.29 deprecates Model Runner V1 with removal targeted at 0.32. Our
runner is our own class so nothing breaks now, but it will need revisiting.

Training is out of scope

The training layer is not migrated and not tested. One guard was needed: neuron
configuration deserialization registered its classes by importing both the inference and
the training configurations, so a broken training import made loading any inference
configuration fail.

Before submitting

• Did you write any new necessary tests?

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[JingyaHuang]

Thanks for the fix!

[JingyaHuang]

why do you want to enable both flags here? Is it related to the fix?

[tengomucho]

because some users expects underscores and others expect dashes, and this is an easy way to support both

[github-actions]

This PR is stale because it has been open 15 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[github-actions]

This PR was closed because it has been stalled for 5 days with no activity.