Skip to content

Fit the Muse Glimmer CUDA batching e2e export in the CI timeout - #25018

Closed
Gasoonjia wants to merge 1 commit into
mainfrom
muse-glimmer-ci-export-time
Closed

Gasoonjia wants to merge 1 commit into
mainfrom
muse-glimmer-ci-export-time

Conversation

@Gasoonjia

Copy link
Copy Markdown
Contributor

test-muse-glimmer-cuda-e2e-kquant-17gb-dflash-image-batching has been cancelled at the 240-minute job timeout on every run since it was added in #24066 (for example job 130041481693: the export started at 19:12 and was still compiling at 22:50).

Why

The export is CPU-bound. All measurements below are on an A100 devserver with cold Inductor and Triton caches:

44 CPUs 11-CPU cgroup quota, 125 GiB (CI runner)
forward_1 5m47s 6m24s
forward_8 6m16s 7m00s
forward_16 7m10s 9m26s
forward_32 8m22s 11m45s
whole export 2h27m about 2h50m

Memory and swap stay flat under the quota. The quantized-GEMM compile pool is sized by os.cpu_count(), which inside a container reports the whole host, so it starts 32 compile workers on 11 vCPU. CI was slower still (about 2h53m to reach forward_others, against 1h26m on the devserver).

Change

  • Size the quantized-GEMM compile pool by the CPUs the process may use: the affinity mask and the cgroup v2 cpu.max quota. On the CI runner that is 11 workers instead of 32. Hosts without a quota are unchanged (capped at 32).
  • Export only forward_{1,4,16,64} for the CI e2e, instead of all seven static sizes. The CUDA executor picks from whatever forward methods the program has and pads each step to the next exported width (test_step_plan.cpp already covers sparse sets like {1,4}), so the runtime path under test is the same.

Test plan

  • The pool sizing returns 11 for a cpu.max of 1100000 100000, and 32 for max or no cgroup file.
  • CI: this PR touches .ci/scripts/export_model_artifact.sh, so the job should run here; it needs to finish within 240 minutes.

test-muse-glimmer-cuda-e2e-kquant-17gb-dflash-image-batching has been
cancelled at the 240-minute job timeout on every run since it was added: the
export alone was still compiling at 3h38m.

The export is CPU-bound. On an A100 devserver with cold caches it takes 2h27m
with 44 CPUs, and under the CI runner's 11-vCPU cgroup quota each static
forward slows down more the wider it is (forward_32: 8m22s -> 11m45s), with
memory and swap flat.

- Size the quantized-GEMM compile pool by the CPUs the process may use (the
  affinity mask and the cgroup v2 cpu.max quota) instead of os.cpu_count(),
  which reports the whole host inside a container. On the CI runner that is
  11 workers instead of 32.
- Export forward_{1,4,16,64} instead of the default seven static sizes for
  the CI e2e; the CUDA executor pads each step to the next exported width,
  so the runtime path is unchanged.
@pytorch-bot

pytorch-bot Bot commented Nov 13, 2026 •

Copy link
Copy Markdown

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/25018

Note: Links to docs will display an error until the docs builds have been completed.

❌ 2 New Failures, 4 Unrelated Failures, 3 Unclassified Failures

As of commit 814619c with merge base d01021b (image):

NEW FAILURES - The following jobs have failed:

UNCLASSIFIED FAILURES - DrCI could not classify the following jobs because the workflow did not run on the merge base. The failures may be pre-existing on trunk or introduced by this PR:

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

BROKEN TRUNK - The following jobs failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Nov 13, 2026
@github-actions

Copy link
Copy Markdown

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

@Gasoonjia
Gasoonjia deployed to upload-benchmark-results November 13, 2026 04:04 — with GitHub Actions Active
Gasoonjia added a commit that referenced this pull request Nov 13, 2026
…hing e2e

Each forward_{N} is its own AOTI compile and autotune. On the 11-vCPU A100
runner the full 1..64 set does not fit 240 minutes (#25018 still timed out
with this alone, so the 480-minute timeout stays); 1,4,16,64 exercises the
same CUDA executor padding path with three fewer compiles.
@Gasoonjia

Copy link
Copy Markdown
Contributor Author

Closing in favor of #24889. Measurement from this PR: with the cgroup-aware compile pool and --forward-sizes 1,4,16,64, the kquant-17gb dflash-image-batching export started 22 minutes into the job and was still compiling the draft methods when the 240-minute timeout hit, so the 480-minute timeout in #24889 is still needed. The forward-sizes change is now commit 8731344 on #24889.

@Gasoonjia Gasoonjia closed this Nov 13, 2026
@Gasoonjia
Gasoonjia deleted the muse-glimmer-ci-export-time branch November 13, 2026 06:52
Gasoonjia added a commit that referenced this pull request Nov 13, 2026
Export forward_{1,4} and a single draft owner in CI, and drop the job's
480-minute timeout back to the shared 240. Even with the CPU-quota-aware
compile pool and forward_{1,4,16,64}, the export was still compiling the
draft methods at the 240-minute mark (#25018); forward_16/64 and
draft_forward_{2,3,4} account for about two of its hours. The run still
covers static and dynamic target steps, vision, and DFlash drafting on the
batching runtime.

This branch was successfully deployed

1 active and 1 inactive deployments
upload-benchmark-results — 814619c1 Deployed Nov 13, 2026 by Gasoonjia via upload-benchmark-results #22426
cadence — 814619c1 Deployed Nov 13, 2026 by Gasoonjia via hifi-op-test / hifi4 #38174
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant