Skip to content

fix(batching): stop shorter clips in a batch from failing to transcribe#49

Open
richiejp wants to merge 1 commit into
mudler:masterfrom
richiejp:fix-batched-attn-nan-padding
Open

fix(batching): stop shorter clips in a batch from failing to transcribe#49
richiejp wants to merge 1 commit into
mudler:masterfrom
richiejp:fix-batched-attn-nan-padding

Conversation

@richiejp

@richiejp richiejp commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

Basically on streaming models with a short attention window the padding
that is applied to the shorter clips can cause a scascade of Infs/Nans
that causes transcription to fail and return empty.


In a heterogeneous-length batch on a chunked_limited / local-attention
model (e.g. nemotron-3.5-asr-streaming-0.6b), any clip whose padding tail
exceeded the attention left context (~4.5 s of length spread at 8x
subsampling) decoded to an EMPTY transcript. Mechanism: a deeply padded
query row has its ENTIRE additive mask row at -inf, softmax over all--inf
yields NaN, the post-softmax query-row mask keeps it (NaN*0 == NaN), and
the NaN key columns then poison every softmax row of the next layer --
24 layers later the whole item is NaN. Full attention was unaffected
(padded query rows can still see valid keys), which is why only
limited-context models in batches with enough length spread were hit; the
batch JSON entry points the LocalAI backend calls were silently affected.

Leave fully-masked padded query rows unmasked instead: their finite
garbage output is zeroed by the existing post-softmax query mask (output
reduces to linear_out.bias, exactly as before), and valid rows' masks are
untouched. Applied to every band/window mask builder (single + batched,
full/chunked-limited + banded local + chunked kernel).

Verified on nemotron-3.5-asr-streaming-0.6b q8_0 (att_context [56,3]): in
a {16.4 s, 7.4 s} batch the short item's encoder output was 100% NaN
before the fix and now byte-matches its solo decode; transcripts match
solo at every tested T_max spread (8.4-16.4 s). Also verified on
parakeet_realtime_eou_120m-v1 f16 (att_context [70,1]): pre-fix the 7.4 s
fixture decoded empty in any batch with T_max >= 14.4 s, post-fix it
byte-matches solo at every spread. Cache-aware streaming and single-clip
offline transcripts are byte-identical between pre-fix and post-fix
builds (those paths never form padded batches), so the fix changes
nothing outside the broken case.

Assisted-by: Claude:claude-fable-5 [Claude Code]

…ion masks

In a heterogeneous-length batch on a chunked_limited / local-attention
model (e.g. nemotron-3.5-asr-streaming-0.6b), any clip whose padding tail
exceeded the attention left context (~4.5 s of length spread at 8x
subsampling) decoded to an EMPTY transcript. Mechanism: a deeply padded
query row has its ENTIRE additive mask row at -inf, softmax over all--inf
yields NaN, the post-softmax query-row mask keeps it (NaN*0 == NaN), and
the NaN key columns then poison every softmax row of the next layer --
24 layers later the whole item is NaN. Full attention was unaffected
(padded query rows can still see valid keys), which is why only
limited-context models in batches with enough length spread were hit; the
batch JSON entry points the LocalAI backend calls were silently affected.

Leave fully-masked padded query rows unmasked instead: their finite
garbage output is zeroed by the existing post-softmax query mask (output
reduces to linear_out.bias, exactly as before), and valid rows' masks are
untouched. Applied to every band/window mask builder (single + batched,
full/chunked-limited + banded local + chunked kernel).

Verified on nemotron-3.5-asr-streaming-0.6b q8_0 (att_context [56,3]): in
a {16.4 s, 7.4 s} batch the short item's encoder output was 100% NaN
before the fix and now byte-matches its solo decode; transcripts match
solo at every tested T_max spread (8.4-16.4 s). Also verified on
parakeet_realtime_eou_120m-v1 f16 (att_context [70,1]): pre-fix the 7.4 s
fixture decoded empty in any batch with T_max >= 14.4 s, post-fix it
byte-matches solo at every spread. Cache-aware streaming and single-clip
offline transcripts are byte-identical between pre-fix and post-fix
builds (those paths never form padded batches), so the fix changes
nothing outside the broken case.

Assisted-by: Claude:claude-fable-5 [Claude Code]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant