Phonon: run the Mimi transposed convolutions as convolutions on Vulkan, MLX and CUDA (#25020) by psiddh · Pull Request #25020 · pytorch/executorch

psiddh · 2026-11-13T05:56:32Z

Summary:

Applies the Mimi rewrite from the previous diff (conv_transpose1d_as_conv1d plus depthwise_conv_transpose_as_mul) to Vulkan, MLX and CUDA as well. Vulkan previously ran the decoder's transposed convolutions through conv_transpose1d_as_2d, which is now used only by --webgpu-compat-ops. The CUDA backend has no Triton kernel for transposed convolution and runs each one as a convolution over a zero-stuffed input, which does stride times the work; the rewrite avoids that.

DepthwiseConvTranspose1dAsMul now creates its zeros with x.new_zeros, so they follow the input to the GPU (torch.zeros put them on the CPU and broke the CUDA export), and the two swapped modules are in WEIGHT_MODULES, so --dtype bf16 still computes them in bf16.

This diff was written with the help of Claude Code (Claude Opus 5.5).

Reviewed By: Gasoonjia, shoumikhin

Differential Revision: D131642733

pytorch-bot · 2026-11-13T05:56:37Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/25020

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure

As of commit 6eff255 with merge base 3e85cae ():

NEW FAILURE - The following job has failed:

Build Presets / apple (llm) / build (gh)
pytorch_tokenizers

This comment was automatically generated by Dr. CI and updates every 15 minutes.

Copilot

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

meta-codesync · 2026-11-13T05:56:59Z

@psiddh has exported this pull request. If you are a Meta employee, you can view the originating Diff in D131642733.

github-actions · 2026-11-13T05:57:43Z

This PR needs a `release notes:` label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

…n, MLX and CUDA (pytorch#25020) Summary: Pull Request resolved: pytorch#25020 Applies the Mimi rewrite from the previous diff (conv_transpose1d_as_conv1d plus depthwise_conv_transpose_as_mul) to Vulkan, MLX and CUDA as well. Vulkan previously ran the decoder's transposed convolutions through conv_transpose1d_as_2d, which is now used only by --webgpu-compat-ops. The CUDA backend has no Triton kernel for transposed convolution and runs each one as a convolution over a zero-stuffed input, which does stride times the work; the rewrite avoids that. DepthwiseConvTranspose1dAsMul now creates its zeros with x.new_zeros, so they follow the input to the GPU (torch.zeros put them on the CPU and broke the CUDA export), and the two swapped modules are in WEIGHT_MODULES, so --dtype bf16 still computes them in bf16. This diff was written with the help of Claude Code (Claude Opus 5.5). Differential Revision: D131642733

Copilot

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

…n, MLX and CUDA (pytorch#25020) Summary: Pull Request resolved: pytorch#25020 Applies the Mimi rewrite from the previous diff (conv_transpose1d_as_conv1d plus depthwise_conv_transpose_as_mul) to Vulkan, MLX and CUDA as well. Vulkan previously ran the decoder's transposed convolutions through conv_transpose1d_as_2d, which is now used only by --webgpu-compat-ops. The CUDA backend has no Triton kernel for transposed convolution and runs each one as a convolution over a zero-stuffed input, which does stride times the work; the rewrite avoids that. DepthwiseConvTranspose1dAsMul now creates its zeros with x.new_zeros, so they follow the input to the GPU (torch.zeros put them on the CPU and broke the CUDA export), and the two swapped modules are in WEIGHT_MODULES, so --dtype bf16 still computes them in bf16. This diff was written with the help of Claude Code (Claude Opus 5.5). Differential Revision: D131642733

Copilot

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

…n, MLX and CUDA (pytorch#25020) Summary: Pull Request resolved: pytorch#25020 Applies the Mimi rewrite from the previous diff (conv_transpose1d_as_conv1d plus depthwise_conv_transpose_as_mul) to Vulkan, MLX and CUDA as well. Vulkan previously ran the decoder's transposed convolutions through conv_transpose1d_as_2d, which is now used only by --webgpu-compat-ops. The CUDA backend has no Triton kernel for transposed convolution and runs each one as a convolution over a zero-stuffed input, which does stride times the work; the rewrite avoids that. DepthwiseConvTranspose1dAsMul now creates its zeros with x.new_zeros, so they follow the input to the GPU (torch.zeros put them on the CPU and broke the CUDA export), and the two swapped modules are in WEIGHT_MODULES, so --dtype bf16 still computes them in bf16. This diff was written with the help of Claude Code (Claude Opus 5.5). Differential Revision: D131642733

Gasoonjia

Review automatically exported from Phabricator review in Meta.

…n, MLX and CUDA (pytorch#25020) Summary: Pull Request resolved: pytorch#25020 Applies the Mimi rewrite from the previous diff (conv_transpose1d_as_conv1d plus depthwise_conv_transpose_as_mul) to Vulkan, MLX and CUDA as well. Vulkan previously ran the decoder's transposed convolutions through conv_transpose1d_as_2d, which is now used only by --webgpu-compat-ops. The CUDA backend has no Triton kernel for transposed convolution and runs each one as a convolution over a zero-stuffed input, which does stride times the work; the rewrite avoids that. DepthwiseConvTranspose1dAsMul now creates its zeros with x.new_zeros, so they follow the input to the GPU (torch.zeros put them on the CPU and broke the CUDA export), and the two swapped modules are in WEIGHT_MODULES, so --dtype bf16 still computes them in bf16. This diff was written with the help of Claude Code (Claude Opus 5.5). Reviewed By: Gasoonjia Differential Revision: D131642733

Copilot

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

…n, MLX and CUDA (pytorch#25020) Summary: Pull Request resolved: pytorch#25020 Applies the Mimi rewrite from the previous diff (conv_transpose1d_as_conv1d plus depthwise_conv_transpose_as_mul) to Vulkan, MLX and CUDA as well. Vulkan previously ran the decoder's transposed convolutions through conv_transpose1d_as_2d, which is now used only by --webgpu-compat-ops. The CUDA backend has no Triton kernel for transposed convolution and runs each one as a convolution over a zero-stuffed input, which does stride times the work; the rewrite avoids that. DepthwiseConvTranspose1dAsMul now creates its zeros with x.new_zeros, so they follow the input to the GPU (torch.zeros put them on the CPU and broke the CUDA export), and the two swapped modules are in WEIGHT_MODULES, so --dtype bf16 still computes them in bf16. This diff was written with the help of Claude Code (Claude Opus 5.5). Reviewed By: Gasoonjia Differential Revision: D131642733

…n, MLX and CUDA (pytorch#25020) Summary: Applies the Mimi rewrite from the previous diff (conv_transpose1d_as_conv1d plus depthwise_conv_transpose_as_mul) to Vulkan, MLX and CUDA as well. Vulkan previously ran the decoder's transposed convolutions through conv_transpose1d_as_2d, which is now used only by --webgpu-compat-ops. The CUDA backend has no Triton kernel for transposed convolution and runs each one as a convolution over a zero-stuffed input, which does stride times the work; the rewrite avoids that. DepthwiseConvTranspose1dAsMul now creates its zeros with x.new_zeros, so they follow the input to the GPU (torch.zeros put them on the CPU and broke the CUDA export), and the two swapped modules are in WEIGHT_MODULES, so --dtype bf16 still computes them in bf16. This diff was written with the help of Claude Code (Claude Opus 5.5). Reviewed By: Gasoonjia, shoumikhin Differential Revision: D131642733

Copilot

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

…s on XNNPACK (pytorch#25004) Summary: XNNPACK's deconvolution is the most expensive op in Phonon's Mimi decoder: about 45% of each frame on x86 and 38% on arm64. Every transposed convolution in Mimi has kernel = 2 * stride, so each one is exactly a kernel-2 convolution with stride * out_channels outputs, one per output phase, whose phases are then interleaved into the output steps. The XNNPACK export now applies that rewrite (conv_transpose1d_as_conv1d) to the three decoder transposed convolutions, and the existing depthwise_conv_transpose_as_mul to the upsampler. The output is the same up to float rounding: the exported 31M program writes audio within 1 LSB (16-bit PCM) of the program without the rewrite, with the same length and frames. Running the transposed convolutions as 2-D convolutions with the sequence on the width axis (conv_transpose1d_as_2d) does not help: Mimi stays at 8.2 ms per frame, so the gain comes from the convolution kernel rather than the tiling axis. This diff was written with the help of Claude Code (Claude Opus 5.5). Reviewed By: shoumikhin, JakeStevens Differential Revision: D131605676

…n, MLX and CUDA (pytorch#25020) Summary: Applies the Mimi rewrite from the previous diff (conv_transpose1d_as_conv1d plus depthwise_conv_transpose_as_mul) to Vulkan, MLX and CUDA as well. Vulkan previously ran the decoder's transposed convolutions through conv_transpose1d_as_2d, which is now used only by --webgpu-compat-ops. The CUDA backend has no Triton kernel for transposed convolution and runs each one as a convolution over a zero-stuffed input, which does stride times the work; the rewrite avoids that. DepthwiseConvTranspose1dAsMul now creates its zeros with x.new_zeros, so they follow the input to the GPU (torch.zeros put them on the CPU and broke the CUDA export), and the two swapped modules are in WEIGHT_MODULES, so --dtype bf16 still computes them in bf16. This diff was written with the help of Claude Code (Claude Opus 5.5). Reviewed By: Gasoonjia, shoumikhin Differential Revision: D131642733

Copilot

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

psiddh requested review from SS-JIA, digantdesai and larryliu0820 as code owners November 13, 2026 05:56

Copilot AI balanced review requested due to automatic review settings November 13, 2026 05:56

pytorch-bot Bot added the module: vulkan Issues related to the Vulkan delegate and code under backends/vulkan/ label Nov 13, 2026

meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Nov 13, 2026

Copilot AI reviewed Nov 13, 2026

meta-codesync Bot added the meta-exported label Nov 13, 2026

meta-codesync Bot changed the title ~~Phonon: run the Mimi transposed convolutions as convolutions on Vulkan, MLX and CUDA~~ Phonon: run the Mimi transposed convolutions as convolutions on Vulkan, MLX and CUDA (#25020) Nov 13, 2026

psiddh force-pushed the export-D131642733 branch from ca106de to 75fbfe9 Compare November 13, 2026 08:50

Copilot AI balanced review requested due to automatic review settings November 13, 2026 08:50

Copilot AI reviewed Nov 13, 2026

JakeStevens mentioned this pull request Nov 13, 2026

[XNNPACK] Lower dense transposed convs as one polyphase conv + depth to space #25037

Draft

psiddh force-pushed the export-D131642733 branch from 75fbfe9 to 141726b Compare November 13, 2026 17:28

Copilot AI balanced review requested due to automatic review settings November 13, 2026 17:28

Copilot AI reviewed Nov 13, 2026

Gasoonjia approved these changes Nov 13, 2026

View reviewed changes

psiddh force-pushed the export-D131642733 branch from 141726b to 3602271 Compare November 13, 2026 19:07

Copilot AI balanced review requested due to automatic review settings November 13, 2026 19:07

Copilot AI reviewed Nov 13, 2026

psiddh force-pushed the export-D131642733 branch from 3602271 to 0876f42 Compare November 14, 2026 03:19

Copilot AI balanced review requested due to automatic review settings November 14, 2026 03:19

Copilot AI reviewed Nov 14, 2026

psiddh added 2 commits November 13, 2026 23:50

psiddh force-pushed the export-D131642733 branch from 0876f42 to 6eff255 Compare November 14, 2026 07:50

Copilot AI balanced review requested due to automatic review settings November 14, 2026 07:50

Copilot AI reviewed Nov 14, 2026

Navigation Menu

Repository navigation

Phonon: run the Mimi transposed convolutions as convolutions on Vulkan, MLX and CUDA (#25020) - #25020

Phonon: run the Mimi transposed convolutions as convolutions on Vulkan, MLX and CUDA (#25020)#25020
psiddh wants to merge 2 commits into
pytorch:mainfrom
psiddh:export-D131642733

psiddh commented Nov 13, 2026 •

edited by meta-codesync Bot

Loading

Uh oh!

pytorch-bot Bot commented Nov 13, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

meta-codesync Bot commented Nov 13, 2026

Uh oh!

github-actions Bot commented Nov 13, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Gasoonjia left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

psiddh commented Nov 13, 2026 • edited by meta-codesync Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot Bot commented Nov 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/25020

❌ 1 New Failure

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

meta-codesync Bot commented Nov 13, 2026

Uh oh!

github-actions Bot commented Nov 13, 2026

This PR needs a release notes: label

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Gasoonjia left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

psiddh commented Nov 13, 2026 •

edited by meta-codesync Bot

Loading

pytorch-bot Bot commented Nov 13, 2026 •

edited

Loading

This PR needs a `release notes:` label