Repository navigation
Conversation
test-muse-glimmer-cuda-e2e-kquant-17gb-dflash-image-batching has been
cancelled at the 240-minute job timeout on every run since it was added: the
export alone was still compiling at 3h38m.
The export is CPU-bound. On an A100 devserver with cold caches it takes 2h27m
with 44 CPUs, and under the CI runner's 11-vCPU cgroup quota each static
forward slows down more the wider it is (forward_32: 8m22s -> 11m45s), with
memory and swap flat.
- Size the quantized-GEMM compile pool by the CPUs the process may use (the
affinity mask and the cgroup v2 cpu.max quota) instead of os.cpu_count(),
which reports the whole host inside a container. On the CI runner that is
11 workers instead of 32.
- Export forward_{1,4,16,64} instead of the default seven static sizes for
the CI e2e; the CUDA executor pads each step to the next exported width,
so the runtime path is unchanged.
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/25018
Note: Links to docs will display an error until the docs builds have been completed. ❌ 2 New Failures, 4 Unrelated Failures, 3 Unclassified FailuresAs of commit 814619c with merge base d01021b ( NEW FAILURES - The following jobs have failed:
UNCLASSIFIED FAILURES - DrCI could not classify the following jobs because the workflow did not run on the merge base. The failures may be pre-existing on trunk or introduced by this PR:
FLAKY - The following jobs failed but were likely due to flakiness present on trunk:
BROKEN TRUNK - The following jobs failed but were present on the merge base:👉 Rebase onto the `viable/strict` branch to avoid these failures
This comment was automatically generated by Dr. CI and updates every 15 minutes. |
This PR needs a
|
…hing e2e
Each forward_{N} is its own AOTI compile and autotune. On the 11-vCPU A100
runner the full 1..64 set does not fit 240 minutes (#25018 still timed out
with this alone, so the 480-minute timeout stays); 1,4,16,64 exercises the
same CUDA executor padding path with three fewer compiles.
|
Closing in favor of #24889. Measurement from this PR: with the cgroup-aware compile pool and |
Export forward_{1,4} and a single draft owner in CI, and drop the job's
480-minute timeout back to the shared 240. Even with the CPU-quota-aware
compile pool and forward_{1,4,16,64}, the export was still compiling the
draft methods at the 240-minute mark (#25018); forward_16/64 and
draft_forward_{2,3,4} account for about two of its hours. The run still
covers static and dynamic target steps, vision, and DFlash drafting on the
batching runtime.
test-muse-glimmer-cuda-e2e-kquant-17gb-dflash-image-batchinghas been cancelled at the 240-minute job timeout on every run since it was added in #24066 (for example job 130041481693: the export started at 19:12 and was still compiling at 22:50).Why
The export is CPU-bound. All measurements below are on an A100 devserver with cold Inductor and Triton caches:
Memory and swap stay flat under the quota. The quantized-GEMM compile pool is sized by
os.cpu_count(), which inside a container reports the whole host, so it starts 32 compile workers on 11 vCPU. CI was slower still (about 2h53m to reachforward_others, against 1h26m on the devserver).Change
cpu.maxquota. On the CI runner that is 11 workers instead of 32. Hosts without a quota are unchanged (capped at 32).forward_{1,4,16,64}for the CI e2e, instead of all seven static sizes. The CUDA executor picks from whatever forward methods the program has and pads each step to the next exported width (test_step_plan.cppalready covers sparse sets like{1,4}), so the runtime path under test is the same.Test plan
cpu.maxof1100000 100000, and 32 formaxor no cgroup file..ci/scripts/export_model_artifact.sh, so the job should run here; it needs to finish within 240 minutes.