Skip to content

Add run_sam3: text-prompted SAM 3 video segmentation CLI (on the uv setup)#602

Open
MasahiroOgawa wants to merge 17 commits into
facebookresearch:mainfrom
sensyn-robotics:mas/test_sam3video
Open

Add run_sam3: text-prompted SAM 3 video segmentation CLI (on the uv setup)#602
MasahiroOgawa wants to merge 17 commits into
facebookresearch:mainfrom
sensyn-robotics:mas/test_sam3video

Conversation

@MasahiroOgawa

@MasahiroOgawa MasahiroOgawa commented Jul 3, 2026

Copy link
Copy Markdown

Summary

Adds scripts/run_sam3, a CLI that runs SAM 3 text-prompted segmentation on an arbitrary video and writes a mask-overlay result video:

scripts/run_sam3 --prompt "powerline wire" --input-video path/to/clip.MOV
# -> result/clip_powerline_wire.mp4

Note on scope: this branch is built on top of the Conda→uv migration in #601 (feature/uv), so this PR also contains those uv changes (they are not yet on main). I kept it this way deliberately to confirm run_sam3 works end-to-end on the uv environment. For an isolated review of just the environment migration, see #601; the new material here is scripts/run_sam3 + the .gitignore rules. If #601 lands first, I'll rebase so this shows only the CLI.

The run_sam3 CLI

  • Single self-bootstrapping script: run directly, it re-execs itself via uv run with opencv-python (video I/O) and pycocotools (detector) as ephemeral deps — no separate launcher, no new permanent project deps.
  • Whole video, bounded memory: samples frames with ffmpeg (--sample-fps), processes them in independent --chunk-size sessions, and writes overlay frames incrementally — GPU/CPU memory stays bounded regardless of clip length.
  • Overlay: per-object colored masks + contours (helps thin structures like wires) with the prompt as an on-frame title; output under --result-dir (default result/, now git-ignored).
  • Sensible defaults for a 12 GB GPU: sam3 version, CPU offload of video+state, expandable_segments, use_fa3 off; robust to headless/CPU-only environments (clear errors instead of crashes).

Testing

End-to-end on a 2:50 / 1920x1080 clip with "powerline wire": produced a full 170 s / 850-frame overlay video across 15 chunks with no OOM; wires segmented and tracked on every frame.

Included uv migration (same as #601)

Conda→uv install flow + committed uv.lock, explicit PyTorch cu126 index via [tool.uv.sources], requires-python >=3.12,<3.14, einops/psutil added to core with lazy sam3.train.* imports so import sam3 works on a bare uv sync, and use_fa3=False defaults.

MasahiroOgawa and others added 17 commits July 3, 2026 11:50
@MasahiroOgawa
Migrate from Conda to uv
4abac50
@MasahiroOgawa @claude
Update README to use uv commands instead of pip
d58bfba 
Replace pip install commands with uv equivalents for running Jupyter
notebooks and setting up the development environment.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
@MasahiroOgawa @claude
Fix uv sync dependency resolution and make sam3 importable
76142c1 
- Route only torch/torchvision/torchaudio to the PyTorch cu126 index via
  an explicit index + [tool.uv.sources]; other packages (e.g. iopath)
  now resolve from PyPI, fixing the unsatisfiable iopath>=0.1.10 error
- Cap requires-python to <3.14 (torch 2.7.0 has no 3.14 wheels)
- Pin torchaudio==2.7.0 to match torch; the previously unpinned range
  resolved to 2.11.0 and crashed with "undefined symbol: torch_library_impl"
- Move einops from the notebooks extra to core dependencies, since
  sam3.sam.rope imports it at package import time

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@MasahiroOgawa @claude
Make training-data imports lazy so bare uv sync can import sam3
022f8be 
Core inference modules pulled in sam3.train.* at package-import time,
which transitively required extra-only deps (decord, pycocotools) and
broke `import sam3` without the notebooks/train extras.

- sam3_image.py, sam3_tracker_base.py: import BatchedDatapoint from
  sam3.model.data_misc (its real home) instead of sam3.train.data.collator,
  dropping the decord dependency
- sam3_video_base.py, agent/client_sam3.py: import rle_encode from
  sam3.train.masks_ops lazily at the call site (it needs pycocotools only
  when actually producing RLE output)
- pyproject.toml: declare psutil as a core dependency; it is imported by
  sam3_video_predictor.py but was only present transitively via extras

Verified `import sam3` and both model builders now import on a bare
`uv sync`, and the lazy import paths remain valid with extras installed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@MasahiroOgawa @claude
Add nbstripout as a dev dependency
d07a282 
The repo configures an nbstripout git clean filter (via .git/info/attributes
+ .git/config) that runs `.venv/bin/python -m nbstripout`, but nbstripout was
never a declared dependency, so a uv-managed venv lacked it and every git
operation on notebooks failed clone-wide. Declaring it in the dev dependency
group (installed by default on `uv sync`) restores the filter.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@MasahiroOgawa @claude
Require Python >=3.12 and default use_fa3=False for portable setup
dc779be 
- requires-python: >=3.9 -> >=3.12 (matches the README "Python 3.12 or higher"
  prerequisite); keep the <3.14 upper bound since torch 2.7.0 has no cp314
  wheels. Trim Python classifiers to 3.12/3.13 accordingly.
- build_sam3_predictor / build_sam3_multiplex_video_predictor: default
  use_fa3=False. Flash-Attention 3 is not a declared dependency, so the
  previous True default made the out-of-the-box call fail on machines without
  flash-attn. All internal modules already defaulted to False.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@MasahiroOgawa @claude
Address review: pin torchvision, fix sample_run and README nits
059d70f 
- pyproject: pin torchvision==0.22.0 to match torch/torchaudio==2.7.0
  (the >= range defeated the intent of pinning); re-locked uv.lock.
- examples/sample_run.py: print full traceback in the except handler so real
  errors are not masked; drop unused imports (torch, numpy) and unused `boxes`;
  remove leading blank line.
- README: fix install-step numbering (was 1,2,3,5 -> 1,2,3,4).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@MasahiroOgawa
fix sample image
08bed69
@MasahiroOgawa @claude
Lower confidence threshold in sample to fix empty mask output
0b9e7fa 
The default confidence_threshold of 0.5 was too high for the model's
output scores (~0.12-0.17), resulting in all detections being filtered.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
@MasahiroOgawa @claude
Fix BPE vocab path in image predictor notebook
bede421 
Remove explicit bpe_path argument since build_sam3_image_model()
already handles the default path correctly using pkg_resources.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
@MasahiroOgawa
add mas_test
b81f171
@MasahiroOgawa @claude
Add video predictor session setup to mas_test notebook; ignore genera…
721ec97 
…ted PNGs/artifacts

- mas_test.ipynb: add parameters cell (video_path), pass gpus_to_use to
  build_sam3_video_predictor, and start a streaming session via
  handle_stream_request
- .gitignore: ignore generated *.png and output/results/artifacts dirs,
  while preserving curated figures under assets/ via negation

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@MasahiroOgawa @claude
Add scripts/run_sam3: text-prompted SAM3 video segmentation CLI
29c2438 
Runs SAM 3 on an arbitrary video with a text prompt and writes a mask-overlay
result video, e.g.:

  scripts/run_sam3 --prompt "powerline wire" --input-video path/to/clip.MOV

- scripts/run_sam3: bash launcher that runs the tool via `uv run`, providing
  opencv-python (video I/O) and pycocotools (sam3 detector) ephemerally so they
  need not be permanent project dependencies
- scripts/run_sam3.py: samples frames (ffmpeg, --sample-fps/--max-frames),
  applies the text prompt on frame 0, propagates via handle_stream_request,
  and renders colored mask overlays + contours to result/<stem>_<prompt>.mp4
- Defaults tuned to fit a 12 GB GPU: version "sam3" (its init_state is
  compatible with the start_session API, unlike sam3.1 multiplex), CPU offload
  of video+state, expandable_segments, and a 60-frame cap
- .gitignore: ignore result/ (generated overlay videos)

Tested end-to-end on a 1920x1080 clip with prompt "powerline wire": SAM3
detected and tracked the wires across all 60 sampled frames and produced
result/powerline_sag_powerline_wire.mp4.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@MasahiroOgawa @claude
run_sam3: process whole video in memory-bounded chunks; merge launche…
61c1786 
…r into one file

- Process the entire input video by default (--max-frames now 0 = all) instead
  of stopping at 60 frames. Frames are handled in independent --chunk-size (60)
  sessions: each chunk re-applies the text prompt on its first frame and
  propagates through the chunk, and overlay frames are written to the output
  video incrementally. This bounds GPU/CPU memory regardless of clip length.
- Merge the bash launcher and Python implementation into a single
  self-bootstrapping script: when run directly it re-executes itself via
  `uv run` (with opencv-python + pycocotools as ephemeral deps), so
  `scripts/run_sam3 ...` still works with no separate wrapper. Removes the
  confusing two-file setup (scripts/run_sam3 + scripts/run_sam3.py).

Tested on the 2:50 / 1920x1080 clip with prompt "powerline wire": full 170s /
850-frame overlay video produced (15 chunks, no OOM), wires tracked on every
frame. Note: object colors may change at chunk boundaries since each chunk
re-detects independently.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@MasahiroOgawa @claude
Remove dead .gitattributes nbstripout filter mappings
4491506 
The file only mapped *.ipynb/*.zpln to the nbstripout git filter (and an
ipynb diff driver). That filter is not part of the repo — it was a local,
per-clone `nbstripout --install` integration — so these attributes referenced
a filter nothing defines, leaving notebooks perpetually flagged as modified
whenever the filter happened to be installed. Removing the mappings makes
notebooks plain files tracked verbatim.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@MasahiroOgawa @claude
Drop personal scratch/confirmation notebook edits
e8ef504 
Remove local experimentation that doesn't belong on the branch:
- Delete examples/mas_test.ipynb (scratch video-predictor test notebook).
- Revert examples/sam3_video_predictor_example.ipynb and
  sam3_image_predictor_example.ipynb to origin/main (session-setup and
  bpe_path tweaks were only for local confirmation).
- Reset examples/sample_run.py to the feature/uv baseline (drop the
  test-image / "kid wearing a red bib" / confidence_threshold=0.1 tweaks used
  for local checking).

The run_sam3 CLI, .gitignore rules, and the inherited uv-migration remain.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@MasahiroOgawa @claude
run_sam3: address review — portability and silent-failure fixes
595dd4d 
- Use getpass.getuser() instead of os.getlogin(), which raises OSError in
  headless contexts (cron/nohup/docker exec with no controlling terminal) and
  ran at import time, crashing the whole tool.
- Check cv2.VideoWriter.isOpened() and fail with a clear message; previously a
  missing mp4v codec / unwritable path silently produced an empty video after
  minutes of GPU work while still reporting success.
- Fail fast with a clear message when no CUDA GPU is available, instead of a
  confusing torch.cuda.synchronize() crash mid-run.
- Wrap the uv self-bootstrap exec so a missing `uv` gives an install hint
  rather than a bare FileNotFoundError traceback.
- Validate --chunk-size >= 1 and --sample-fps > 0 (avoid ZeroDivisionError /
  mis-slicing).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Meta Open Source bot. label Jul 3, 2026
@MasahiroOgawa MasahiroOgawa mentioned this pull request Jul 3, 2026
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

CLA Signed This label is managed by the Meta Open Source bot.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@MasahiroOgawa