Harden Android instrumentation CI and isolate performance checks (#24650) by psiddh · Pull Request #24650 · pytorch/executorch

psiddh · 2026-11-05T08:51:03Z

Summary:

Android instrumentation can report success without a completed passing suite, and unbounded device waits can prevent useful diagnostics. Require a nonempty passing summary and one successful terminal result, bound transport and execution phases, and retain diagnostics without masking the original test status.

Keep functional instrumentation blocking ordinary changes while running LLM performance checks in a separate nightly lane. Release pushes, release-targeted pull requests, manual runs, and default reusable-workflow calls retain full coverage. Required check names and branch protection are unchanged.

Keep the default LLM configuration test bounded by caller cancellation, while requiring real generated output and valid statistics. If a test timeout leaves native generation active, abort the instrumentation process instead of letting subsequent tests reuse its native state. Original per-test timeout durations are unchanged.

Differential Revision: D129513640

pytorch-bot · 2026-11-05T08:51:07Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/24650

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure, 7 Unrelated Failures, 3 Unclassified Failures

As of commit f7fbd58 with merge base 0b776dd ():

NEW FAILURE - The following job has failed:

pull / android / run-emulator (gh)
11-13 07:24:42.131 3002 3020 E ExecuTorchTests: Native test teardown refused while a method is executing; aborting test process

UNCLASSIFIED FAILURES - DrCI could not classify the following jobs because the workflow did not run on the merge base. The failures may be pre-existing on trunk or introduced by this PR:

trunk / test-models-macos-cpu (emformer_join, xnnpack-quantization-delegation) / macos-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
trunk / test-models-macos-cpu (ic4, xnnpack-quantization-delegation) / macos-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
trunk / test-models-macos-cpu (vit, xnnpack-quantization-delegation) / macos-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1

BROKEN TRUNK - The following jobs failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

pull / unittest / linux / linux-job (gh) (trunk failure)
examples/models/muse-glimmer/tests/test_cuda_sequence_export_inputs.py::SequenceExportInputsTest::test_small_logits_example_preserves_token_and_embedding_capacity
pull / unittest-editable / linux / linux-job (gh) (trunk failure)
examples/models/muse-glimmer/tests/test_cuda_sequence_export_inputs.py::SequenceExportInputsTest::test_small_logits_example_preserves_token_and_embedding_capacity
trunk / Arm suite / test-arm-backend-ethos-u (test_pytest_models_ethos_u55) / linux-job (gh) (trunk failure)
RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 2
trunk / Arm suite / test-arm-backend-ethos-u (test_pytest_models_ethos_u85) / linux-job (gh) (trunk failure)
RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 2
trunk / Arm suite / test-arm-backend-ethos-u (test_pytest_ops_ethos_u55) / linux-job (gh) (trunk failure)
RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 2
trunk / Arm suite / test-arm-backend-ethos-u (test_pytest_ops_ethos_u85) / linux-job (gh) (trunk failure)
RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 2
trunk / unittest-release / linux / linux-job (gh) (trunk failure)
examples/models/muse-glimmer/tests/test_cuda_sequence_export_inputs.py::SequenceExportInputsTest::test_small_logits_example_preserves_token_and_embedding_capacity

This comment was automatically generated by Dr. CI and updates every 15 minutes.

meta-codesync · 2026-11-05T08:51:13Z

@psiddh has exported this pull request. If you are a Meta employee, you can view the originating Diff in D129513640.

github-actions · 2026-11-05T08:52:06Z

This PR needs a `release notes:` label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

Copilot

🟢 Approval recommended

No active review issues are identified for the Android instrumentation CI hardening and functional/performance suite separation.

No open issues 🎉

What changed in this PR

This PR hardens ExecuTorch’s Android instrumentation CI and separates routine functional coverage from nightly LLM performance checks.

Changes:

Require a nonempty passing suite and exactly one successful terminal result, with bounded execution and diagnostics that preserve the test status.
Run functional tests for ordinary changes and performance tests nightly, while retaining full coverage for release, manual, and default reusable-workflow calls.
Add harness and workflow regression tests without changing required check names.

File	Description
`scripts/run_android_emulator.sh`	Adds suite selection, timeouts, strict result validation, and exit diagnostics.
`.github/workflows/pull.yml`	Selects functional or full coverage based on the triggering event and branch.
`.github/workflows/nightly.yml`	Adds a separate Android performance lane.
`.github/workflows/_android.yml`	Adds suite inputs, harness tests, execution limits, and diagnostic uploads.
`.ci/scripts/tests/test_run_android_emulator.py`	Covers harness outcomes, timeout behavior, diagnostics, and workflow routing.

_{Review effort: Balanced}

…orch#24650) Summary: Android instrumentation can report success without a completed passing suite, and unbounded device waits can prevent useful diagnostics. Require a nonempty passing summary and one successful terminal result, bound transport and execution phases, and retain diagnostics without masking the original test status. Keep functional instrumentation blocking ordinary changes while running LLM performance checks in a separate nightly lane. Release pushes, release-targeted pull requests, manual runs, and default reusable-workflow calls retain full coverage. Required check names and branch protection are unchanged. Differential Revision: D129513640

Copilot

🟢 Approval recommended

No active review issues block the Android CI hardening and separation of functional and nightly performance tests.

No open issues 🎉

_{Review effort: Balanced}

…orch#24650) Summary: Android instrumentation can report success without a completed passing suite, and unbounded device waits can prevent useful diagnostics. Require a nonempty passing summary and one successful terminal result, bound transport and execution phases, and retain diagnostics without masking the original test status. Keep functional instrumentation blocking ordinary changes while running LLM performance checks in a separate nightly lane. Release pushes, release-targeted pull requests, manual runs, and default reusable-workflow calls retain full coverage. Required check names and branch protection are unchanged. Differential Revision: D129513640

Copilot

🟢 Approval recommended

The review reports no active issues with the Android CI hardening and functional/performance test separation.

No open issues 🎉

_{Review effort: Balanced}

…orch#24650) Summary: Android instrumentation can report success without a completed passing suite, and unbounded device waits can prevent useful diagnostics. Require a nonempty passing summary and one successful terminal result, bound transport and execution phases, and retain diagnostics without masking the original test status. Keep functional instrumentation blocking ordinary changes while running LLM performance checks in a separate nightly lane. Release pushes, release-targeted pull requests, manual runs, and default reusable-workflow calls retain full coverage. Required check names and branch protection are unchanged. Differential Revision: D129513640

Copilot

🟡 Changes recommended

The readiness loop ignores the script deadline, allowing retries beyond the configured timeout and returning exit code 1 instead of 124 when the script budget expires.

1 open issue

Enforce script deadline across readiness and pre-install waits · New

_{Review effort: Balanced}

…orch#24650) Summary: Android instrumentation can report success without a completed passing suite, and unbounded device waits can prevent useful diagnostics. Require a nonempty passing summary and one successful terminal result, bound transport and execution phases, and retain diagnostics without masking the original test status. Keep functional instrumentation blocking ordinary changes while running LLM performance checks in a separate nightly lane. Release pushes, release-targeted pull requests, manual runs, and default reusable-workflow calls retain full coverage. Required check names and branch protection are unchanged. Differential Revision: D129513640

Copilot

🟡 Changes recommended

A critical TERM-handling flaw can delay exit and diagnostics for up to an hour, while readiness operations can exceed their budget and accept late success.

2 open issues

1 resolved since last review

Enforce script deadline across readiness and pre-install waits

_{Review effort: Balanced}

+trap 'exit 130' INT
+trap 'exit 131' QUIT
+trap 'exit 141' PIPE
+trap 'exit 143' TERM


+until adb_with_timeout shell \
+  'test "$(getprop sys.boot_completed)" = "1" && cmd package list packages >/dev/null && sm list-volumes all >/dev/null'; do
+  if ((SECONDS >= ready_deadline)); then
+    echo "Emulator framework did not become ready before the deadline" >&2
+    exit 1
+  fi
+  sleep_with_deadline 5
+  adb_with_timeout wait-for-device || true
+done


…orch#24650) Summary: Android instrumentation can report success without a completed passing suite, and unbounded device waits can prevent useful diagnostics. Require a nonempty passing summary and one successful terminal result, bound transport and execution phases, and retain diagnostics without masking the original test status. Keep functional instrumentation blocking ordinary changes while running LLM performance checks in a separate nightly lane. Release pushes, release-targeted pull requests, manual runs, and default reusable-workflow calls retain full coverage. Required check names and branch protection are unchanged. Differential Revision: D129513640

Copilot

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

…orch#24650) Summary: Android instrumentation can report success without a completed passing suite, and unbounded device waits can prevent useful diagnostics. Require a nonempty passing summary and one successful terminal result, bound transport and execution phases, and retain diagnostics without masking the original test status. Keep functional instrumentation blocking ordinary changes while running LLM performance checks in a separate nightly lane. Release pushes, release-targeted pull requests, manual runs, and default reusable-workflow calls retain full coverage. Required check names and branch protection are unchanged. Differential Revision: D129513640

Copilot

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

…orch#24650) Summary: Android instrumentation can report success without a completed passing suite, and unbounded device waits can prevent useful diagnostics. Require a nonempty passing summary and one successful terminal result, bound transport and execution phases, and retain diagnostics without masking the original test status. Keep functional instrumentation blocking ordinary changes while running LLM performance checks in a separate nightly lane. Release pushes, release-targeted pull requests, manual runs, and default reusable-workflow calls retain full coverage. Required check names and branch protection are unchanged. Keep the default LLM configuration test bounded by caller cancellation, while requiring real generated output and valid statistics. If a test timeout leaves native generation active, abort the instrumentation process instead of letting subsequent tests reuse its native state. Original per-test timeout durations are unchanged. Differential Revision: D129513640

Copilot

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

…orch#24650) Summary: Android instrumentation can report success without a completed passing suite, and unbounded device waits can prevent useful diagnostics. Require a nonempty passing summary and one successful terminal result, bound transport and execution phases, and retain diagnostics without masking the original test status. Keep functional instrumentation blocking ordinary changes while running LLM performance checks in a separate nightly lane. Release pushes, release-targeted pull requests, manual runs, and default reusable-workflow calls retain full coverage. Required check names and branch protection are unchanged. Keep the default LLM configuration test bounded by caller cancellation, while requiring real generated output and valid statistics. If a test timeout leaves native generation active, abort the instrumentation process instead of letting subsequent tests reuse its native state. Original per-test timeout durations are unchanged. Differential Revision: D129513640

Copilot

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

Copilot AI balanced review requested due to automatic review settings November 5, 2026 08:51

meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Nov 5, 2026

meta-codesync Bot added the meta-exported label Nov 5, 2026

Copilot started reviewing on behalf of psiddh November 5, 2026 08:51

View session