[ICE]: Found unstable fingerprints for evaluate_obligation(...): Ok(EvaluatedToOk)

[danderson]

This may be a dupe with #155740 and/or #154857, but filing anew because it seems to be a consistent reproducer for me with a fairly small source diff, although still on a fairly tall stack of crates and cargo work.

Reproduces consistently on two machines, with rustc 1.95.0 and 1.96.0. It does not reproduce with current nightly (2026-06-22).

Code

The ICE happens on incremental recompile between the two top commits of branch https://github.com/tailscale/tailscale-rs/tree/rust-ice . Initially found with our giant CI check run (in bin/check), but I narrowed it down to a pair of cargo build commands on a specific crate.

git clone https://github.com/tailscale/tailscale-rs rust-ice
cd rust-ice/ts_devtools
cargo clean
git checkout b0f8ce26024bcf8ba7f925636521c2fddf82fa34
cargo build
git checkout 1771fe1d2ffa443217496d66f88c97178d5479b2
cargo build

The second cargo build triggers a rustc panic.

Meta

Versions: reproduces consistently on 1.95.0, 1.96.0:

rustc --version --verbose:

rustc 1.95.0 (59807616e 2026-04-14)
binary: rustc
commit-hash: 59807616e1fa2540724bfbac14d7976d7e4a3860
commit-date: 2026-04-14
host: x86_64-unknown-linux-gnu
release: 1.95.0
LLVM version: 22.1.2

rustc 1.96.0 (ac68faa20 2026-05-25)
binary: rustc
commit-hash: ac68faa20c58cbccd01ee7208bf3b6e93a7d7f96
commit-date: 2026-05-25
host: x86_64-unknown-linux-gnu
release: 1.96.0
LLVM version: 22.1.2

It does not reproduce on current nightly at time of writing:

rustc 1.98.0-nightly (4429659e4 2026-06-22)
binary: rustc
commit-hash: 4429659e4745016bd3f26a4a421843edc7fbc422
commit-date: 2026-06-22
host: x86_64-unknown-linux-gnu
release: 1.98.0-nightly
LLVM version: 22.1.7

All toolchains come from rustup.

Error output

Error output and backtrace is from 1.96.0.

   Compiling ts_control v0.3.3 (/home/dave/tail/rust-ice/ts_control)
   Compiling ts_netcheck v0.3.3 (/home/dave/tail/rust-ice/ts_netcheck)
   Compiling ts_runtime v0.3.3 (/home/dave/tail/rust-ice/ts_runtime)
   Compiling tailscale v0.3.3 (/home/dave/tail/rust-ice)
   Compiling ts_cli_util v0.3.3 (/home/dave/tail/rust-ice/ts_cli_util)
   Compiling ts_devtools v0.3.3 (/home/dave/tail/rust-ice/ts_devtools)
error: internal compiler error: encountered incremental compilation error with evaluate_obligation(91ec3250ae59823b-a2c85d98a904b519)
  |
  = note: please follow the instructions below to create a bug report with the provided information
  = note: for incremental compilation bugs, having a reproduction is vital
  = note: an ideal reproduction consists of the code before and some patch that then triggers the bug when applied and compiled again
  = note: as a workaround, you can run `cargo clean -p pftail` or `cargo clean` to allow your project to compile

Backtrace

thread 'rustc' (403248) panicked at /rustc-dev/ac68faa20c58cbccd01ee7208bf3b6e93a7d7f96/compiler/rustc_middle/src/verify_ich.rs:80:9:
Found unstable fingerprints for evaluate_obligation(91ec3250ae59823b-a2c85d98a904b519): Ok(EvaluatedToOk)
stack backtrace:
   0:     0x7c3411a278db - <<std[52919eca6bce4da3]::sys::backtrace::BacktraceLock>::print::DisplayBacktrace as core[18c8dd30382e7099]::fmt::Display>::fmt
   1:     0x7c341201e548 - core[18c8dd30382e7099]::fmt::write
   2:     0x7c34133e5d76 - <std[52919eca6bce4da3]::sys::stdio::unix::Stderr as std[52919eca6bce4da3]::io::Write>::write_fmt
   3:     0x7c34119fdbe8 - std[52919eca6bce4da3]::panicking::default_hook::{closure#0}
   4:     0x7c3411a1af43 - std[52919eca6bce4da3]::panicking::default_hook
   5:     0x7c34109f32bc - std[52919eca6bce4da3]::panicking::update_hook::<alloc[c54872da263eaafc]::boxed::Box<rustc_driver_impl[6663a291b147a63]::install_ice_hook::{closure#1}>>::{closure#0}
   6:     0x7c3411a1b222 - std[52919eca6bce4da3]::panicking::panic_with_hook
   7:     0x7c34119fdca8 - std[52919eca6bce4da3]::panicking::panic_handler::{closure#0}
   8:     0x7c34119f2209 - std[52919eca6bce4da3]::sys::backtrace::__rust_end_short_backtrace::<std[52919eca6bce4da3]::panicking::panic_handler::{closure#0}, !>
   9:     0x7c34119ff70d - __rustc[8068f81614cfe5c]::rust_begin_unwind
  10:     0x7c340e57083c - core[18c8dd30382e7099]::panicking::panic_fmt
  11:     0x7c34110582d7 - rustc_middle[a5b6b88feb5609e]::verify_ich::incremental_verify_ich_failed
  12:     0x7c3412bbb016 - rustc_middle[a5b6b88feb5609e]::verify_ich::incremental_verify_ich::<rustc_middle[a5b6b88feb5609e]::query::erase::ErasedData<[u8; 2usize]>>
  13:     0x7c3412608c5a - rustc_query_impl[5e8495fc2c24c830]::execution::try_execute_query::<rustc_middle[a5b6b88feb5609e]::query::caches::DefaultCache<rustc_type_ir[142733f82a324bd8]::canonical::CanonicalQueryInput<rustc_middle[a5b6b88feb5609e]::ty::context::TyCtxt, rustc_middle[a5b6b88feb5609e]::ty::ParamEnvAnd<rustc_middle[a5b6b88feb5609e]::ty::predicate::Predicate>>, rustc_middle[a5b6b88feb5609e]::query::erase::ErasedData<[u8; 2usize]>>, true>
  14:     0x7c341260788a - rustc_query_impl[5e8495fc2c24c830]::query_impl::evaluate_obligation::execute_query_incr::__rust_end_short_backtrace
  15:     0x7c3412760a1d - <rustc_trait_selection[de5a181da859feaf]::traits::fulfill::FulfillProcessor as rustc_data_structures[b25e30df202a677b]::obligation_forest::ObligationProcessor>::process_obligation
  16:     0x7c3412004716 - <rustc_data_structures[b25e30df202a677b]::obligation_forest::ObligationForest<rustc_trait_selection[de5a181da859feaf]::traits::fulfill::PendingPredicateObligation>>::process_obligations::<rustc_trait_selection[de5a181da859feaf]::traits::fulfill::FulfillProcessor>
  17:     0x7c341287c9f3 - <rustc_trait_selection[de5a181da859feaf]::traits::engine::ObligationCtxt>::make_canonicalized_query_response::<()>
  18:     0x7c341287b6f9 - rustc_traits[3975eb9380e6e8fa]::type_op::type_op_prove_predicate
  19:     0x7c3412e19d27 - rustc_query_impl[5e8495fc2c24c830]::execution::try_execute_query::<rustc_middle[a5b6b88feb5609e]::query::caches::DefaultCache<rustc_type_ir[142733f82a324bd8]::canonical::CanonicalQueryInput<rustc_middle[a5b6b88feb5609e]::ty::context::TyCtxt, rustc_middle[a5b6b88feb5609e]::ty::ParamEnvAnd<rustc_middle[a5b6b88feb5609e]::traits::query::type_op::ProvePredicate>>, rustc_middle[a5b6b88feb5609e]::query::erase::ErasedData<[u8; 8usize]>>, true>
  20:     0x7c3412e18c3f - rustc_query_impl[5e8495fc2c24c830]::query_impl::type_op_prove_predicate::execute_query_incr::__rust_end_short_backtrace
  21:     0x7c3412655250 - <rustc_borrowck[66c3034ef1d4feb6]::type_check::TypeChecker>::normalize_and_prove_instantiated_predicates
  22:     0x7c340f3c2cc4 - <rustc_borrowck[66c3034ef1d4feb6]::type_check::TypeChecker as rustc_middle[a5b6b88feb5609e]::mir::visit::Visitor>::visit_operand
  23:     0x7c3412e627d5 - <rustc_borrowck[66c3034ef1d4feb6]::type_check::TypeChecker as rustc_middle[a5b6b88feb5609e]::mir::visit::Visitor>::visit_body
  24:     0x7c3412e419b6 - rustc_borrowck[66c3034ef1d4feb6]::type_check::type_check
  25:     0x7c3412ba138e - <rustc_borrowck[66c3034ef1d4feb6]::root_cx::BorrowCheckRootCtxt>::do_mir_borrowck
  26:     0x7c3412b9cd2f - rustc_borrowck[66c3034ef1d4feb6]::mir_borrowck
  27:     0x7c3412b9cb65 - rustc_query_impl[5e8495fc2c24c830]::query_impl::mir_borrowck::invoke_provider_fn::__rust_begin_short_backtrace
  28:     0x7c341218b110 - rustc_query_impl[5e8495fc2c24c830]::execution::try_execute_query::<rustc_data_structures[b25e30df202a677b]::vec_cache::VecCache<rustc_span[fab24ddf1f607803]::def_id::LocalDefId, rustc_middle[a5b6b88feb5609e]::query::erase::ErasedData<[u8; 8usize]>, rustc_middle[a5b6b88feb5609e]::dep_graph::graph::DepNodeIndex>, true>
  29:     0x7c3412182b6e - rustc_query_impl[5e8495fc2c24c830]::query_impl::mir_borrowck::execute_query_incr::__rust_end_short_backtrace
  30:     0x7c3412311238 - rustc_mir_transform[f41c4d82acfd6942]::mir_drops_elaborated_and_const_checked
  31:     0x7c341231086b - rustc_query_impl[5e8495fc2c24c830]::query_impl::mir_drops_elaborated_and_const_checked::invoke_provider_fn::__rust_begin_short_backtrace
  32:     0x7c341218a729 - rustc_query_impl[5e8495fc2c24c830]::execution::try_execute_query::<rustc_data_structures[b25e30df202a677b]::vec_cache::VecCache<rustc_span[fab24ddf1f607803]::def_id::LocalDefId, rustc_middle[a5b6b88feb5609e]::query::erase::ErasedData<[u8; 8usize]>, rustc_middle[a5b6b88feb5609e]::dep_graph::graph::DepNodeIndex>, true>
  33:     0x7c3412182a4a - rustc_query_impl[5e8495fc2c24c830]::query_impl::mir_drops_elaborated_and_const_checked::execute_query_incr::__rust_end_short_backtrace
  34:     0x7c3412f340d6 - <rustc_middle[a5b6b88feb5609e]::ty::context::TyCtxt>::par_hir_body_owners::<rustc_interface[fdd38ffd3247134f]::passes::run_required_analyses::{closure#2}::{closure#0}>::{closure#0}
  35:     0x7c3412f33397 - rustc_interface[fdd38ffd3247134f]::passes::analysis
  36:     0x7c341322fe87 - rustc_query_impl[5e8495fc2c24c830]::execution::try_execute_query::<rustc_middle[a5b6b88feb5609e]::query::caches::SingleCache<rustc_middle[a5b6b88feb5609e]::query::erase::ErasedData<[u8; 0usize]>>, true>
  37:     0x7c341322f7ea - rustc_query_impl[5e8495fc2c24c830]::query_impl::analysis::execute_query_incr::__rust_end_short_backtrace
  38:     0x7c3413183642 - rustc_interface[fdd38ffd3247134f]::interface::run_compiler::<(), rustc_driver_impl[6663a291b147a63]::run_compiler::{closure#0}>::{closure#1}
  39:     0x7c341316d67e - std[52919eca6bce4da3]::sys::backtrace::__rust_begin_short_backtrace::<rustc_interface[fdd38ffd3247134f]::util::run_in_thread_with_globals<rustc_interface[fdd38ffd3247134f]::util::run_in_thread_pool_with_globals<rustc_interface[fdd38ffd3247134f]::interface::run_compiler<(), rustc_driver_impl[6663a291b147a63]::run_compiler::{closure#0}>::{closure#1}, ()>::{closure#0}, ()>::{closure#0}::{closure#0}, ()>
  40:     0x7c341316df20 - <std[52919eca6bce4da3]::thread::lifecycle::spawn_unchecked<rustc_interface[fdd38ffd3247134f]::util::run_in_thread_with_globals<rustc_interface[fdd38ffd3247134f]::util::run_in_thread_pool_with_globals<rustc_interface[fdd38ffd3247134f]::interface::run_compiler<(), rustc_driver_impl[6663a291b147a63]::run_compiler::{closure#0}>::{closure#1}, ()>::{closure#0}, ()>::{closure#0}::{closure#0}, ()>::{closure#1} as core[18c8dd30382e7099]::ops::function::FnOnce<()>>::call_once::{shim:vtable#0}
  41:     0x7c341316edac - <std[52919eca6bce4da3]::sys::thread::unix::Thread>::new::thread_start
  42:     0x7c340caa40da - start_thread
                               at ./nptl/pthread_create.c:454:8
  43:     0x7c340cb377ac - __GI___clone3
                               at ./misc/../sysdeps/unix/sysv/linux/x86_64/clone3.S:78:0
  44:                0x0 - <unknown>

error: the compiler unexpectedly panicked. This is a bug

note: we would appreciate a bug report: https://github.com/rust-lang/rust/issues/new?labels=C-bug%2C+I-ICE%2C+T-compiler&template=ice.md

note: rustc 1.96.0 (ac68faa20 2026-05-25) running on x86_64-unknown-linux-gnu

note: compiler flags: --crate-type bin -C embed-bitcode=no -C debuginfo=2 -C incremental=[REDACTED]

note: some of the compiler flags provided by cargo are hidden

query stack during panic:
#0 [evaluate_obligation] evaluating trait selection obligation `futures_util::stream::stream::filter_map::FilterMap<ts_cli_util::CommonArgs::connect_control::{opaque#0}::{opaque#0}, {async closure body@ts_devtools/src/bin/pftail.rs:30:31: 30:53}, {async closure@ts_devtools/src/bin/pftail.rs:30:21: 30:30}>: futures_core::stream::Stream`
#1 [type_op_prove_predicate] evaluating `type_op_prove_predicate` `ProvePredicate { predicate: Binder { value: TraitPredicate(<futures_util::stream::stream::filter_map::FilterMap<ts_cli_util::CommonArgs::connect_control::{opaque#0}::{opaque#0}, {async closure body@ts_devtools/src/bin/pftail.rs:30:31: 30:53}, {async closure@ts_devtools/src/bin/pftail.rs:30:21: 30:30}> as futures_core::stream::Stream>, polarity:Positive), bound_vars: [] } }`
#2 [mir_borrowck] borrow-checking `main`
#3 [mir_drops_elaborated_and_const_checked] elaborating drops for `main`
#4 [analysis] running analysis passes on crate `pftail`
end of query stack

[danderson]

I bisected with cargo-bisect-rustc, and it reports that this was first fixed in nightly 2026-06-05, by 76dfce2 (PR #145477):

searched nightlies: from nightly-2026-05-25 to nightly-2026-06-23
regressed nightly: nightly-2026-06-05
searched commit range: https://github.com/rust-lang/rust/compare/b354133fb126352871bea4b40795a45739adff2b...e7815e522ecc746592fee32f50478f521333b503
regressed commit: https://github.com/rust-lang/rust/commit/76dfce2cb2d3f7b7f34d62e6ffe044f7e7d76948

(Note, it reports the nightly/commit as "regressed" because I had to invert bisect-rustc's failure condition, since I had a range that starts out broken then gets fixed. So, the version/commit it flags is in fact the first one that doesn't have this ICE.)