Manual merge of PR stack#20602
Merged
Merged
Conversation
added 4 commits
June 26, 2026 12:52
Pull Request resolved: #20382 Adds Vulkan support for `aten.bitwise_or.Tensor` and `aten.logical_or.default`, mirroring the existing `bitwise_and` / `logical_and` implementation. This is the first of two ops needed to collapse the Llama4-mini TISO en_US backbone export to a single Vulkan partition: the discrete-speech mask OR-s several bool tensors via `bitwise_or`, which previously had no Vulkan implementation and forced a CPU fallback that split the delegated graph. Implementation mirrors `bitwise_and`: a `X | Y` uint8 shader variant in `binary_op_buffer.yaml` / `binary_op_texture.yaml`, a `DEFINE_BINARY_OP_FN(bitwise_or)` dispatch with `VK_REGISTER_OP` for both `aten.bitwise_or.Tensor` and `aten.logical_or.default` in `BinaryOp.cpp`, and `register_bitwise_or` / `register_logical_or` `OpFeatures` (bool inputs) in `op_registry.py`. This change was authored with Claude. ghstack-source-id: 397529322 @exported-using-ghexport Differential Revision: [D108457794](https://our.internmc.facebook.com/intern/diff/D108457794/)
Pull Request resolved: #20383 Adds Vulkan support for `aten.eq.Scalar`. This is the second of two ops needed to collapse the Llama4-mini TISO en_US backbone export to a single Vulkan partition after `bitwise_or`: the discrete-speech mask compares the int token-id tensor against scalar constants via `aten.eq.Scalar`, which previously had no Vulkan implementation and forced a CPU fallback that split the delegated graph. Implemented by extending the existing tensor-scalar binary-op path with a comparison-output variant: `binary_scalar_buffer.glsl` / `binary_scalar_texture.glsl` gain an `IS_COMPARISON_OP` code path that writes a `uint8` (bool) output while leaving the existing arithmetic path unchanged; `binary_scalar_buffer.yaml` / `binary_scalar_texture.yaml` generate variants for the `half`/`float`, `float`/`float`, `int32`/`int32`, and `int32`/`float` tensor/scalar dtype pairs. Mixed tensor/scalar dtypes are computed in a promoted type rather than narrowing the scalar to the tensor dtype: a new `get_higher_precision_dtype` helper in `gen_vulkan_spv.py` picks the higher-precision of the two dtypes, and `binary_op_defs.glslh` evaluates operands in that promoted `COMPUTE_T` before narrowing to the output dtype (so e.g. an `int32` tensor compared against a `float` scalar compares in `float`). `BinaryScalarOp.cpp` adds an `eq_tensor_scalar` dispatch, registers `aten.eq.Scalar`, and coerces the scalar extract dtype to match a generated variant; `op_registry.py` registers `aten.eq.Scalar` features with FP/INT tensor input and bool output. The generated op-test graph builders now preserve `at::Scalar` tags when adding graph scalars, so integer, boolean, and floating scalar literals exercise the correct graph scalar type instead of all being converted through `double`. The int64 token tensor is serialized to int32 via the existing `downcast_64_bit` path, so the dispatch resolves to the int32 shader variant; no dtype-conversion pass is added. This change was authored with Claude. ghstack-source-id: 397529325 @exported-using-ghexport Differential Revision: [D108457791](https://our.internmc.facebook.com/intern/diff/D108457791/)
…type Pull Request resolved: #20491 The per-token dynamic-activation-quant (`dq8ca`) zero-point image must be bound in the shader with the same dtype the tensor was allocated with; a binding-vs-allocation dtype mismatch corrupts the per-token zero-point. The allocation dtype differs by export path: standard `export_llama -qmode 8da4w` models (e.g. Qwen3-0.6B) serialize the zero-point as `int8`, while the Llama4-mini TISO backbone (torchao `per_token_dynamic_quant` / `Int8DynamicActivationIntxWeightConfig` with an explicit fp32 `zero_point_dtype`) serializes it as float, which `vulkan_graph_builder.get_effective_dtype` downcasts to `half` under `force_fp16`. A single fixed binding dtype cannot satisfy both paths. Binding the zero-point as `int8` (`rgba8i`) corrupts the float-allocated TISO zero-point on ARM Mali (Valhall) -- negative values come back as garbage, garbling the 8da4w TTS backbone. Conversely, binding it as the codegen `DTYPE` (matching the scale's float dtype) corrupts the int8-allocated zero-point: under fp16 inference the `rgba8i` image is read and written as `rgba16f`, saturating the per-token zero-point to the int8 floor/ceiling and garbling standard fp16 8da4w models such as Qwen3-0.6B. This change makes the zero-point binding a codegen variant so it always matches the tensor's allocation. A new `ZP_DTYPE_MODE` axis emits two variants of every dq8ca shader that binds the per-token zero-point: `zpint8` (binding declared `int8`, an `rgba8i` integer image) and `zpinherit` (binding declared with the codegen `DTYPE`, inheriting the inference float dtype to match the scale -- `rgba32f`, or `rgba16f` under `USE_VULKAN_FP16_INFERENCE`). The C++ shader pickers select the variant from `graph.dtype_of(zero_point)` (`kChar` -> `zpint8`; `kHalf` / `kFloat` -> `zpinherit`), so the shader binding matches the tensor's allocation regardless of how the model was exported. The shared read helper is unchanged: `ivec4(texelFetch(t_int8_input_zps, ...))` already reads both an integer image (identity) and a float image (exact truncation of the integer-valued zero-point in `[-128, 127]`). Affected shaders: `choose_qparams_per_row` (writes the zero-point, storing `ivec4` or `VEC4_T` per variant), `quantize_and_pack_4h4w_with_group_sums`, `linear_dq8ca_q4gsw_tiled`, and the dq8ca `linear_q4gsw_coop` variants (read the zero-point). This fixes the fp16 8da4w regression for standard int8 zero-point exports while preserving the float zero-point path that the TISO backbone and the original Mali fix depend on. Only the runtime shader binding changes, so existing `.pte` files are handled correctly with no re-export. Authored with Claude Code. ghstack-source-id: 397529329 @exported-using-ghexport Differential Revision: [D109595977](https://our.internmc.facebook.com/intern/diff/D109595977/)
…_q4gsw Pull Request resolved: #20381 TISO and other torchao-quantized models emit a `torchao.dequantize_affine -> aten.embedding` subgraph for their weight-only int4 quantized embedding. The existing `QuantizedEmbeddingMatch` only matches the `quantized_decomposed.embedding_4bit.dtype` fused op, so the torchao embedding never fused: its `dequantize_affine` const-folded to an fp32 weight, the resulting `aten.embedding` exceeded the buffer-element limit and fell back to CPU, and the fp32 constant bloated the serialized model. This adds a separate `TorchAOQuantizedEmbeddingMatch` matcher that recognizes the torchao int4 `dequantize_affine -> aten.embedding` shape (qmin=-8/qmax=7, per-row group block_size `[1, G]`) and rewrites it to the existing `et_vk.embedding_q4gsw.default` op, repacking the unpacked int8 weight into the packed 4-bit layout. It asserts symmetric quantization (zero_point == 0, which the shader assumes) and guards against repacking a shared/tied weight more than once by recording the repack against the weight's state-dict FQN via `register_param_mutation` (a per-ExportedProgram registry on `ep._et_vk_param_modification_tags`); a second match on the same tied weight sees the recorded tag and skips the in-place repack. It is kept as a separate class from `QuantizedEmbeddingMatch` because the two dialects produce different graph shapes (one fused op vs a split dequant+gather), so a single class would only co-locate two disjoint parse paths. On the en_US TISO backbone the embedding now delegates to Vulkan instead of falling back to CPU, and the serialized `.pte` drops from 418 MiB to 348 MiB. This change was authored with Claude. ghstack-source-id: 397529341 @exported-using-ghexport Differential Revision: [D108457797](https://our.internmc.facebook.com/intern/diff/D108457797/)
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/20602
Note: Links to docs will display an error until the docs builds have been completed. This comment was automatically generated by Dr. CI and updates every 15 minutes. |
meta-cla
Bot
added
the
CLA Signed
|
This PR needs a
|
SS-JIA
changed the title
psiddh
approved these changes
Jun 29, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TSIA: #20381