Fix multiplex No-points crash when removing a shared-conditioning-frame owner#573
Open
xsilphf wants to merge 1 commit into
Open
Fix multiplex No-points crash when removing a shared-conditioning-frame owner#573xsilphf wants to merge 1 commit into
xsilphf wants to merge 1 commit into
Conversation
…me owner In the multiplex video tracker, several objects in a bucket can share a single conditioning frame: one "owner" object's input created it, while others were added on already-tracked frames and never created their own conditioning frame, relying on the bucket's shared anchor plus their own non-conditioning memory. Removing the owner downgrades that shared conditioning frame inside clear_all_points_in_frame, emptying output_dict["cond_frame_outputs"], which then calls _reset_tracking_results and wipes every object's memory -- including the surviving objects' -- later raising "No points are provided" in propagate_in_video. Fix: when conditioning frames would become empty but other objects still have tracked memory, promote the earliest non-conditioning frame containing a surviving object to be the new conditioning anchor instead of resetting; fall back to the full reset only when no other object has memory left. consolidated_frame_inds is left untouched (it must equal the set of input frames). Adds a model-free regression test (test/test_multiplex_shared_cond_reset.py).
meta-cla
Bot
added
the
CLA Signed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #572
Problem
In the SAM 3.1 multiplex video tracker, several objects in a bucket can share a single conditioning frame: one object (the "owner") supplied the input that created it, while the others were added on already-tracked frames (
is_init_cond_frame=False) and never created their own conditioning frame — they rely on the bucket having a conditioning frame plus their own non-conditioning memory.Removing the owner downgrades that shared conditioning frame inside
clear_all_points_in_frame, emptyingoutput_dict["cond_frame_outputs"]. The existing code then calls_reset_tracking_results, which wipes every object's memory — including the surviving objects'. A laterpropagate_in_videoraisesRuntimeError: No points are provided; please add points first(also asserted in_prepare_memory_conditioned_featuresviaassert len(output_dict["cond_frame_outputs"]) > 0).Fix
In
clear_all_points_in_frame, when the conditioning frames would become empty but other objects still have tracked memory, promote the earliest non-conditioning frame that contains a surviving object to be the new conditioning anchor (keeping the usual "early anchor + forward memory" layout), instead of resetting. Fall back to the full reset only when no other object has any memory left (e.g. when removing the only object).output_dict["cond_frame_outputs"]is changed (the gate checked bypropagate_in_video, and the source read by memory attention); the per-object slices are kept consistent.consolidated_frame_indsis intentionally not touched — it must equal the set of input frames, and the promoted frame has no user input. This does meanoutput_dict["cond_frame_outputs"]andconsolidated_frame_indsdiverge for the promoted frame; I'm happy to discuss the cleanest contract here, since the freeloader objects never had a user-input conditioning frame and so any recovery anchor is by definition input-less.Test
Adds
test/test_multiplex_shared_cond_reset.py— model-free, data-free unit tests (arbitrary synthetic ids) that fail on currentmainand pass with the fix: they confirm the other objects' memory is preserved, the first-participant frame is promoted, the non-conditioning chain is kept, and a full reset still happens when removing the only object.Validation
Verified end-to-end on a real multiplex tracking run: it crosses the exact frame where the unpatched code raises
No points, completes the full run with 0 crashes, and produces the same number of unique tracks as my previous local workaround.Heads up: I used Claude Code to help write and debug this fix, but I've gone through it carefully myself and verified it end-to-end on a real run (it cleanly passes the exact frame where the unpatched code used to crash). If anything looks off, or you'd prefer a different recovery contract, just ping me — happy to iterate.