[Inference Providers] image-text-to-video: reference images, videos and audio on fal and wavespeed

[hanouticelina]

Context

MiniMaxAI/MiniMax-H3 can be driven by more than one image and a prompt: extra subject or style images, motion video clips, an audio track to sync to — each addressed from the prompt by position ("Image 1", "Video 1", "Audio 1"). Today the client can send none of that.

Three providers serve it, with three different vocabularies. This PR wires the two that are live on the Hub; replicate follows separately.

	fal	wavespeed	replicate (follow-up)
structure	6 endpoints (3 + 3 lora)	6 endpoints (3 + 3 lora)	one model, all modalities
images / videos / audio	reference_image_urls reference_video_urls reference_audio_urls	reference_images reference_videos reference_audios	reference_image_urls reference_video_urls reference_audio_urls
caps	9 / 3 / 3, 12 total	9 / 3 / 3	none published
first frame	image_url, separate endpoint	separate endpoint	first_frame_image, same schema

The task owns the concept

ImageTextToVideoParameters gains three optional fields, typed and generated for every language:

reference_images?: string[];   // "Image 1", "Image 2", …
reference_videos?: string[];   // "Video 1", …
reference_audio?:  string[];   // "Audio 1", …

Callers write those names whoever serves the model — so anything built on the task, the Hub's ImageTextToVideoWidget included, needs no provider-specific code:

await client.imageTextToVideo({
  model: "MiniMaxAI/MiniMax-H3",
  inputs: subjectImage,
  parameters: {
    prompt: "Image 1 moves like Video 1, synced to Audio 1",
    reference_images: [styleFile, "https://example.com/style.png"],
    reference_videos: [motionFile],
    reference_audio:  [voiceFile],
  },
});

The schema types the lists as URLs, which is what reaches the provider; the client additionally accepts Blob, so a browser File goes straight through the way inputs already does.

One mechanism, three configurations

lib/referenceInputs.ts does what every provider shares — read the three parameters, coerce, check counts, inline binary as data URLs, emit under the provider's names. What differs is data:

const FAL_AI_REFERENCES: ReferenceInputsSpec = {
  fields: { images: "reference_image_urls", videos: "reference_video_urls", audio: "reference_audio_urls" },
  maxItems: { images: 9, videos: 3, audio: 3 },
  maxTotal: 12,
  rejectAudioAlone: true,
  encode: (blob, modality) => modality === "audio" ? buildFalAiAudioDataUrl(blob) : dataUrlFromBlob(blob, …),
};

One call, each provider's native payload:

fal-ai     → reference_image_urls: ["data:image/png;base64,AQ==", "https://…/style.png"]
             reference_audio_urls: ["data:audio/x-wav;base64,Aw=="]     ← fal's MIME remap
wavespeed  → reference_images: [...], reference_videos: [...], reference_audios: [...]

Adding replicate later is a ReferenceInputsSpec and a subclass — nothing here changes.

Errors name the task's field, not the provider's, because that is what the caller wrote.

Endpoint shape stays per-provider: both fal and wavespeed split the model across endpoints, and both refuse references an endpoint cannot honour rather than sending keys it will ignore — for wavespeed that also stops a Blob being serialised into the request body as {}.

⚠️ Still needs Hub mappings

Only fal's image-to-video and wavespeed's are registered; no reference-to-video endpoint is, and the router exact-matches the path against the registered providerId:

POST router.huggingface.co/fal-ai/minimax/h3/image-to-video      → 200 IN_QUEUE
POST router.huggingface.co/fal-ai/minimax/h3/reference-to-video  → 400 Model not supported by provider fal-ai

For fal and wavespeed, repointing trades away prompt-only and keyframing — verified live: reference-to-video rejects a call with no reference (At least one reference image, video, or audio must be provided) and has no end_image_url. Worth weighing against the follow-up: replicate's single minimax/h3 schema carries the first frame and the references together, so one mapping there would serve the whole surface with nothing given up.

Also fixed

• applyLoraWeights — the .../lora endpoints require a loras entry, but only the image tasks built one, so minimax/h3/image-to-video/lora was unreachable. Four copies folded into one helper, now also used by text-to-video.
• fal queue errors — the queue reports COMPLETED even for a rejected request and the result fetch never checked resultResponse.ok, so a 422 surfaced as ProviderOutputError: expected { video: { url: string } }. That is how the reference requirement above was found. Affects every fal queue task.
• Removed dropEndpointSegmentOnDirectCalls from image-text-to-video — it assumes the text-only variant sits at the parent path, but minimax/h3 is not an endpoint, so prompt-only direct calls 404'd.

Backward compatibility — checked, not assumed

Every pre-existing call shape replayed against the previous commit and diffed:

• fal, 23 shapes (each modality, both endpoints, every limit, scalar and ArrayBuffer refs, untyped blobs, lora on both paths) — byte-identical.
• wavespeed, 6 shapes (image+prompt, prompt-only placeholder image, caller-supplied images[], lora) — byte-identical.

replicate.ts and getProviderHelper.ts are untouched by this PR.

New fields are optional and both task types already carried an index signature.

Testing

45 tests green; tsc, eslint, oxfmt clean across packages/inference and packages/tasks. Live-tested against minimax/h3/reference-to-video with a fal key, which is how the reference requirement and the error-reporting bug surfaced.

Follow-ups

• Replicate, which needs a Hub mapping for MiniMax-H3 before a helper is reachable at all.
• Hub mappings for fal's and wavespeed's reference endpoints.
• The Hub widget — this PR supplies the task-level contract it needs.
• Snippets: getInferenceSnippets.ts routes image-text-to-video through prepareImageToImageInput, so the model page can only show an image+prompt snippet.
• Python huggingface_hub parity.

🤖 Generated with Claude Code

---

Note

Medium Risk
Changes provider request payloads and validation for image-text-to-video (and fal queue error handling). Wrong endpoint mapping or payload shape can reject previously valid calls or send unusable fields to providers.

Overview
Adds optional reference images, videos, and audio to image-text-to-video so callers can pass reference_images / reference_videos / reference_audio (URLs or Blobs) regardless of provider.

Shared buildReferenceInputs validates counts, inlines binaries, and remaps field names. fal and wavespeed only send those lists on …/reference-to-video endpoints; other video endpoints error instead of silently dropping extras. The primary inputs image is treated as “Image 1” on reference endpoints.

Also: LoRA weights now apply on fal text-to-video (and a shared helper), fal queue result fetches surface HTTP errors instead of “malformed response”, and image-text-to-video no longer rewrites fal URLs by dropping a path segment. Schema/types in @huggingface/tasks document the new optional fields.

^{Reviewed by Cursor Bugbot for commit 8d57e2d. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 3 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit fd79980. Configure here.}