Skip to content

contrib/avcodec: whole-track audio decode to PCM — load_pcm's producer - #1483

Merged
paul-hammant merged 4 commits into
mainfrom
feat/avcodec-audio-pcm
Aug 10, 2026
Merged

contrib/avcodec: whole-track audio decode to PCM — load_pcm's producer#1483
paul-hammant merged 4 commits into
mainfrom
feat/avcodec-audio-pcm

Conversation

@paul-hammant

Copy link
Copy Markdown
Collaborator

The producer half of asks/pcm-please.md, completing the loop with the audio.load_pcm consumer that landed in 0.512.0 (#1481). Together they mean an MP4's audio reaches the speakers with no hand-extracted sidecar WAV — the last hand-cranked step in an otherwise in-process pipeline.

Three commits: the ask itself, the implementation (Paul's), and a CHANGELOG entry.

What it adds

pcm, n, rate, ch, err = avcodec.audio_pcm(path)
src, e = audio.load_pcm(pcm, n, rate, ch, audio.FORMAT_S16)

avc_audio_decode_raw demuxes and decodes a file's audio stream to interleaved s16 stereo at the source rate in one shot. libswresample does the conversion in the same pass, so a 5.1 float-planar AAC track comes back as plain stereo s16 without the caller arranging anything. The whole-buffer shape deliberately mirrors load_pcm's, so the two compose directly.

Verified end to end, not just asserted

I rebuilt the pipeline here before writing the CHANGELOG rather than transcribing the commit's numbers — a 2s AAC clip, all the way to a playable source:

decoded n=356352 rate=44100 ch=2
load_pcm ok: duration_ms=2020    (expect ~2000)

FFmpeg demux → s16 stereo → std.audio → playable. The ~1% over 352800 is AAC encoder padding, not a defect. (The original commit's own evidence is a 117.3s clip → 22,523,904 bytes at 48 kHz stereo = 117.3s exactly, with position_ms advancing in real time.)

make contrib-check passes all seven contrib tests including the extended avcodec one.

Dependency change

contrib/avcodec now requires libswresample alongside the other four FFmpeg libraries. Both the contrib-check table and the build probe were updated together; all five are required as a set, and a partial install stays a clean SKIP rather than a build failure.

Shaping up

Three things the commits needed before this could go out:

Verification

  • make ci — C suite 230/230; .ae 978/979. The one failure is integration_http_server_h2, pre-existing and diagnosed in pesky_bug.md.
  • gcc -Werror -Wall -Wextra on the new C with the FFmpeg cflags, checked locally before pushing.
  • End-to-end run above.

One ergonomic note

Building this by hand hit the ae build --extra shim.c link-flag limitation — it compiles a C shim but cannot pass -l flags, so an aether.toml workspace is required. That is the same gap the avcodec contrib-check fix worked around, and it is what the TODO.md "contrib runtime coverage is split across two mechanisms" entry is about. Not addressed here.

🤖 Generated with Claude Code

paul-hammant and others added 3 commits August 9, 2026 22:15
From the aether-ui video line. contrib/avcodec removed the intermediate
file for VIDEO -- frames go straight from FFmpeg into a vg raster region
-- but audio still cannot make the same trip, so video_frame requires a
hand-extracted sidecar WAV: 20 MB for a 21 MB source, a manual pre-step
before playback, and no workaround at all for a live source.

Measured the gap rather than assuming it, and it is not where the API
name suggests. load_wav is ma_decoder_init_memory, which sniffs the
format, so it ALREADY accepts MP3 (duration_ms=6013 on a test file). It
rejects MP4 and raw AAC. So the missing thing is not "formats beyond
WAV" -- it is that every entry point takes an encoded container
miniaudio can demux itself, with no way in for samples a DIFFERENT
decoder produced. Noted separately that the load_wav name understates
what it does.

Asks for load_pcm(data, length, rate, channels, format) as the simple
shape, and sketches a streaming push variant for live sources, flagging
that position_ms must then report the DEVICE position since that is the
clock video chases.

Deliberately marked not-urgent: A/V sync is proven and correct today
(video tracks audio.position_ms within 3 ms on a real 720p/5.1 clip),
and nothing about the clock relationship changes with where samples come
from. This is packaging, not architecture -- the sidecar is just the
last hand-cranked step in an otherwise in-process pipeline.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The other half of the pcm-please ask, now that std.audio's load_pcm
(e0738b4) landed. avc_audio_decode_raw demuxes and decodes a file's
audio stream to interleaved s16 STEREO at the source rate in one shot --
libswresample converts whatever the source is (Big Buck Bunny's 5.1
float-planar AAC included) in the same pass. Exposed as
avcodec.audio_pcm(url) -> (pcm, n, rate, ch, err), the whole-buffer
shape matching load_pcm's.

Proven C-only first (the video noise bug taught that order): 117.3s clip
decodes to 22,523,904 bytes at 48 kHz stereo = 117.3s exactly, real
samples mid-track. Then end-to-end in Aether: load_pcm reports
duration_ms=117312 and position_ms advances in real time.

A transient SIGKILL during first end-to-end testing was chased and
did not reproduce -- co-resident make -j8 memory pressure, not a leak;
the identical test passes with 18 GB free.

Probe and contrib-check gain libswresample (all five FFmpeg libs
required together; partial install stays a SKIP). Test gains the
no-audio case (the video-only clip must report an error, not garbage)
and a decoded-size range assertion whose LOWER bound is the real guard.
Falsified via the downmix contract: reporting 6 channels instead of 2
gives "FAIL: channels 6 want 2"; restored, PASS.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The avcodec audio-decode commit landed with no CHANGELOG entry and there was no
[current] section, so per this file's own workflow the next release would have
tagged a version with whole-track audio decode unmentioned — the same way
0.506.0 through 0.509.0 shipped empty (backfilled in #1474, root cause in
#1477).

Written from the shipped code rather than the commit subject, and pairs the
entry with load_pcm (0.512.0) since the two only make sense together: this is
the producer, that is the consumer, and asks/pcm-please.md wanted both.

Verified end-to-end here before writing it, rather than transcribing the
commit's numbers: a 2s AAC clip decodes to 356352 bytes at 44100 Hz stereo and
audio.load_pcm reports duration_ms=2020. (The ~1% over 352800 is AAC encoder
padding, not a defect.)

[skip actions] deliberately NOT used here, despite this being the docs commit:
it is the branch HEAD, so the token would skip CI for the whole branch —
including 244 lines of new C in contrib/avcodec that wants exercising on the
Windows and macOS runners. The token is for branches that are docs-only end to
end, not for the docs commit of a branch that ships code.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@paul-hammant
paul-hammant force-pushed the feat/avcodec-audio-pcm branch from cd3ef4c to fabc485 Compare August 9, 2026 23:06
The CHANGELOG commit carried the skip token in its subject and was the branch
HEAD, so GitHub suppressed the workflow for the whole PR — including 244 lines
of new C in contrib/avcodec that wants exercising on Windows and macOS.

Amending the token away did not retrigger, because the push event had already
been skipped; an empty commit produces the fresh event that does. The subject
here deliberately avoids the token text itself — an earlier attempt explained
it in the subject line and GitHub matched THAT, skipping again.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@paul-hammant
paul-hammant force-pushed the feat/avcodec-audio-pcm branch from 34e9b45 to c528671 Compare August 9, 2026 23:09
@paul-hammant
paul-hammant merged commit dfccb75 into main Aug 10, 2026
23 checks passed
@paul-hammant
paul-hammant deleted the feat/avcodec-audio-pcm branch August 10, 2026 05:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant