[launch] Report one actionable diagnostic when distributed ranks fail (and make --quiet work for multi-GPU)

[mosafariuk]

What does this PR do?

When a rank dies during accelerate launch --multi_gpu (or FSDP / single-node DeepSpeed), torch.distributed.elastic hands accelerate a fully structured ChildFailedError — one ProcessFailure per rank with exitcode, signal, pid, timestamp and error-file path. Today the launchers catch it with a generic except Exception and re-raise unchanged, so that structure reaches the user as raw traceback text underneath the per-rank log flood — and torchelastic's own "Root Cause (first observed failure)" heuristic can name a victim rank (one that timed out waiting) while the true culprit is the rank that logged nothing.

This PR reads that object and prints one bordered diagnostic block: the first observed failure (rank, pid, exitcode, signal) explicitly labelled as torchelastic's earliest-timestamp heuristic, a signal-class breakdown separating watchdog-style aborts from ranks the elastic agent tore down afterwards ("collateral, not causes"), pointers to every preserved per-rank artifact, and the exact re-run command for per-rank capture and Flight Recorder. It also:

• forwards torchrun's existing --local_ranks_filter / --redirects (accelerate already forwards --tee / --log_dir through the same _filter_args path) — and refuses the lossy configuration: filtering without file capture auto-enables --tee 3 --log_dir with a warning, so muted console output is demoted to files, never destroyed;
• makes --quiet apply to the multi-GPU path (its help text currently disclaims it, which is part of what how to cleanly exit when using accelerate launch #1089 reports); in quiet mode the exit code becomes 128 + signum (a watchdog SIGABRT yields 134 instead of a lossy 1);
• catches KeyboardInterrupt in simple_launcher, so Ctrl-C exits 130 instead of dumping accelerate's own traceback — the direct ask in how to cleanly exit when using accelerate launch #1089;
• keeps the classifier/formatter in a new pure-function module (accelerate.utils.error_reporting) with no torch import, unit-testable without GPUs.

The three modes, stated precisely: the default interprets the failure (strictly additive — nothing is removed); --quiet silences the Python-side launcher traceback; --local_ranks_filter mutes the C++-side console flood with preservation (torchrun's OS-level redirection; per-rank files always written). No mode deletes diagnostic data.

Verification (real hardware)

• Hardware validation: tested end-to-end on a 2x NVIDIA L40 instance (PyTorch 2.12.1+cu130, NCCL 2.27.7) with a genuine NCCL watchdog timeout provoked by freezing one rank while its peer entered an all-reduce.
• Default mode: the watchdog killed rank 0 — the victim waiting on frozen rank 1 — and the diagnostic block classified the SIGABRT as a watchdog abort, displayed the victim caveat over exactly the case it exists for, then re-raised ChildFailedError unchanged (no logs masked).
• --quiet: exit code 134 (128+SIGABRT), zero launcher tracebacks, single diagnostic report.
• --local_ranks_filter: eliminated the C++ console flood (169 -> 81 lines, zero watchdog lines on console) while the guard auto-enabled --tee/--log_dir; the full C++ stack traces were verified preserved in per-rank stderr.log files.
• Regression safety: healthy runs byte-identical (exit 0, zero new output); ordinary Python exceptions keep their full traceback; tests/test_cli.py and tests/test_utils.py pass.

Before — raw waterfall on the L40s (excerpt of 169 lines)

[rank0]:[E728 18:15:39.660703688 ProcessGroupNCCL.cpp:2197] [PG ID 0 PG GUID 0(default_pg) Rank 0] Process group watchdog thread terminated with exception: NCCL error in: /pytorch/torch/csrc/distributed/c10d/NCCLUtils.cpp:461, Unknown, NCCL version 2.27.7
ncclInternalError: Internal check failed.
Last error:
Unknown
... (~50 more lines of C++ watchdog / NCCL / teardown output) ...
terminate called after throwing an instance of 'c10::DistBackendError'
  what():  [PG ID 0 PG GUID 0(default_pg) Rank 0] Process group watchdog thread terminated with exception: NCCL error in: /pytorch/torch/csrc/distributed/c10d/NCCLUtils.cpp:461, Unknown, NCCL version 2.27.7
ncclInternalError: Internal check failed.
terminate called recursively
W0728 18:15:41.375000 528 torch/distributed/elastic/multiprocessing/api.py:1014] Sending process 676 closing signal SIGTERM

After — the diagnostic block this PR adds (default mode, printed before the unchanged re-raise)

accelerate launch: `repro_gpu_crash.py` failed on 2 rank(s)
------------------------------------------------------------------------------
First observed failure (earliest timestamp -- HEURISTIC, not a verdict):
  rank      : 0 (local_rank 0)
  exitcode  : -6 (SIGABRT), pid 675
  error_file: <none written -- see @record note below>

  A rank exiting on SIGABRT is the signature of PyTorch's NCCL watchdog or heartbeat monitor tearing the process down after a collective exceeded its timeout. This is by design and is not catchable in Python: the abort is raised on a C++ watchdog thread, not on the thread that issued the collective.

  The rank that reports a timeout is frequently a VICTIM waiting on a peer. The root-cause rank is often the one that produced no output at all (it hung), so treat 'first observed failure' as a heuristic, not a verdict.

Rank breakdown:
  aborted (watchdog / hard fault)     : 1  -> ranks [0]
  torn down by the launcher           : 1  -> ranks [1]
  raised a Python exception / exited  : 0

  Ranks torn down by SIGTERM/SIGKILL were killed by the elastic agent AFTER
  the first failure. Their tracebacks are collateral, not causes.

Preserved per-rank detail:
  none -- worker entrypoints were not decorated with
  `torch.distributed.elastic.multiprocessing.errors.record`, so no error
  files were written (exit codes and signals only).
  Per-rank stdout/stderr was NOT captured to disk for this run.
  Re-run with:  accelerate launch --tee 3 --log_dir ./accelerate_logs ...

Suggested next steps:
  1. Re-run with per-rank capture and a quiet console:
       accelerate launch --tee 3 --log_dir ./accelerate_logs \
                         --local_ranks_filter 0 ...
  2. Enable Flight Recorder to see which collective each rank was in:
       TORCH_NCCL_TRACE_BUFFER_SIZE=2000 TORCH_NCCL_DUMP_ON_TIMEOUT=1
  3. Look first at the rank that produced NO output.
==============================================================================
Traceback (most recent call last):
  File "/home/ubuntu/.local/bin/accelerate", line 6, in <module>
    sys.exit(main())

What this deliberately does NOT do

• No recovery, no retry. PyTorch core's position (Better nccl timeout handling pytorch/pytorch#163546) is that ProcessGroupNCCL is not built for recovery. This PR improves presentation of a crash the launcher already handles.
• No claim to catch NCCL watchdog timeouts in Python. The watchdog aborts from a C++ thread via std::terminate (RFC [RFC] Asynchronous Error Handling for Distributed Training with NCCL pytorch/pytorch#46874, by design); this PR classifies the resulting SIGABRT at the launcher, where it is observable.
• No exception swallowing, no data deletion. The default is strictly additive: print one block, then re-raise the original ChildFailedError (same type, same exit semantics). ACCELERATE_DISTRIBUTED_ERROR_SUMMARY=0 disables the block entirely. If the formatter itself fails for any reason it returns None and the original exception propagates exactly as before — a bug in the diagnostics can never hide the real crash.
• No collectives, barriers or state rescue in the crash path. (Lightning shipped the opposite once — Trainer Error Handling Fix Lightning-AI/pytorch-lightning#6842 — and removed it after it deadlocked.) The handler runs in the launcher process, formats, and exits or re-raises.

Scope limitation: DeepSpeed multi-node

DeepSpeed's PDSH/OpenMPI multi-node launchers run through subprocess.Popen and expose only the child's return code, so there is no ChildFailedError on that path (unless elastic training is enabled); behavior there is unchanged.

Tests

tests/test_error_reporting.py — classifier/formatter units (signal bucketing, minimum-timestamp selection, -6 -> 134 mapping, empty-input safety, the heuristic caveat, the None-fallback guard), plus an in-process harness that reaches the real except ChildFailedError clause on zero-GPU CI by monkeypatching torch.distributed.run.run — chosen deliberately because a --cpu subprocess test dispatches to simple_launcher and never reaches the clause. A real watchdog firing is not CI-inducible on hosted runners; the hardware validation above covers it.

Backwards compatibility

New code executes only on the crash path or when a new flag is passed. Exit codes and exception types are unchanged by default.

Refs #1089. Related reports of this failure mode: #3861, #2183, #314, #223. Precedent for launcher-layer mitigation of a distributed footgun: #2195.

Who can review?

@muellerzr @SunMarc @BenjaminBossan

[mosafariuk]

@muellerzr @SunMarc @BenjaminBossan — gentle ping on this one. It's been open since Jul 28 with no reviewer assigned, so I suspect it just didn't land in anyone's queue rather than anyone deciding against it. The 5 workflows are also still awaiting maintainer approval, so there's no CI signal yet either — if someone could approve the run, quality and the unit tests would at least give you something to look at.

Happy to split it if the scope is the blocker: the error_reporting module plus the diagnostic block is the substantive half, and the --quiet / --local_ranks_filter / KeyboardInterrupt changes for #1089 could go separately. Also glad to drop anything that looks like scope creep.