fix(batching): stop shorter clips in a batch from failing to transcribe#49
Open
richiejp wants to merge 1 commit into
Open
fix(batching): stop shorter clips in a batch from failing to transcribe#49richiejp wants to merge 1 commit into
richiejp wants to merge 1 commit into
Conversation
…ion masks
In a heterogeneous-length batch on a chunked_limited / local-attention
model (e.g. nemotron-3.5-asr-streaming-0.6b), any clip whose padding tail
exceeded the attention left context (~4.5 s of length spread at 8x
subsampling) decoded to an EMPTY transcript. Mechanism: a deeply padded
query row has its ENTIRE additive mask row at -inf, softmax over all--inf
yields NaN, the post-softmax query-row mask keeps it (NaN*0 == NaN), and
the NaN key columns then poison every softmax row of the next layer --
24 layers later the whole item is NaN. Full attention was unaffected
(padded query rows can still see valid keys), which is why only
limited-context models in batches with enough length spread were hit; the
batch JSON entry points the LocalAI backend calls were silently affected.
Leave fully-masked padded query rows unmasked instead: their finite
garbage output is zeroed by the existing post-softmax query mask (output
reduces to linear_out.bias, exactly as before), and valid rows' masks are
untouched. Applied to every band/window mask builder (single + batched,
full/chunked-limited + banded local + chunked kernel).
Verified on nemotron-3.5-asr-streaming-0.6b q8_0 (att_context [56,3]): in
a {16.4 s, 7.4 s} batch the short item's encoder output was 100% NaN
before the fix and now byte-matches its solo decode; transcripts match
solo at every tested T_max spread (8.4-16.4 s). Also verified on
parakeet_realtime_eou_120m-v1 f16 (att_context [70,1]): pre-fix the 7.4 s
fixture decoded empty in any batch with T_max >= 14.4 s, post-fix it
byte-matches solo at every spread. Cache-aware streaming and single-clip
offline transcripts are byte-identical between pre-fix and post-fix
builds (those paths never form padded batches), so the fix changes
nothing outside the broken case.
Assisted-by: Claude:claude-fable-5 [Claude Code]
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Basically on streaming models with a short attention window the padding
that is applied to the shorter clips can cause a scascade of Infs/Nans
that causes transcription to fail and return empty.
In a heterogeneous-length batch on a chunked_limited / local-attention
model (e.g. nemotron-3.5-asr-streaming-0.6b), any clip whose padding tail
exceeded the attention left context (~4.5 s of length spread at 8x
subsampling) decoded to an EMPTY transcript. Mechanism: a deeply padded
query row has its ENTIRE additive mask row at -inf, softmax over all--inf
yields NaN, the post-softmax query-row mask keeps it (NaN*0 == NaN), and
the NaN key columns then poison every softmax row of the next layer --
24 layers later the whole item is NaN. Full attention was unaffected
(padded query rows can still see valid keys), which is why only
limited-context models in batches with enough length spread were hit; the
batch JSON entry points the LocalAI backend calls were silently affected.
Leave fully-masked padded query rows unmasked instead: their finite
garbage output is zeroed by the existing post-softmax query mask (output
reduces to linear_out.bias, exactly as before), and valid rows' masks are
untouched. Applied to every band/window mask builder (single + batched,
full/chunked-limited + banded local + chunked kernel).
Verified on nemotron-3.5-asr-streaming-0.6b q8_0 (att_context [56,3]): in
a {16.4 s, 7.4 s} batch the short item's encoder output was 100% NaN
before the fix and now byte-matches its solo decode; transcripts match
solo at every tested T_max spread (8.4-16.4 s). Also verified on
parakeet_realtime_eou_120m-v1 f16 (att_context [70,1]): pre-fix the 7.4 s
fixture decoded empty in any batch with T_max >= 14.4 s, post-fix it
byte-matches solo at every spread. Cache-aware streaming and single-clip
offline transcripts are byte-identical between pre-fix and post-fix
builds (those paths never form padded batches), so the fix changes
nothing outside the broken case.
Assisted-by: Claude:claude-fable-5 [Claude Code]