Skip to content

[Engine] Vision: keep image ownership through request preparation - #4264

Open
Tsury wants to merge 3 commits into
Niko1221:mainfrom
Tsury:fix-2286-vision-ownership
Open

Tsury wants to merge 3 commits into
Niko1221:mainfrom
Tsury:fix-2286-vision-ownership

Conversation

@Tsury

@Tsury Tsury commented Oct 26, 2026

Copy link
Copy Markdown
Contributor

Problem and result

The image cache keeps its records during encode_all, but the service releases the FIFO before it prepares the combined embedding file. Another request can evict those files between encoding and the stat/copy step, so the first request fails or copies a missing file. A refused request with more than 64 cached images can also leave the cache above its bound.

After this change, a request owns its images from encoding through the combined-file copy. Eviction waits for the outermost batch, and the cache trims back to 64 afterward. Encoder pipes close on teardown, and failed startup cleans up. A native decoder refusal retries once through the existing Pillow normalization path; conversion failures still remove their temporary files.

This replaces #2286 (closed in the 0.2.0-beta tracker reset), rebased on current main.

Related discussion

#2286 (this change on the old base, with the review note about Vision.__new__ tests). #1072 and #1207 fixed retention during encoding; this covers the remaining interval through the combined-file copy.

Approach

A reentrant batch counter (cache_batches) defers eviction until the outermost batch exits, so nested encode_all/prepare calls keep ownership without a second lock. The FIFO stays held through validation and copying. The existing VisionCacheEviction tests that build Vision with __new__ now set cache_batches (the review note on #2286). No config, default or API change. The longer FIFO hold protects ownership; its throughput cost is unmeasured.

Validation

Passed:

  • python -m unittest serve.test_vision_cache serve.test_server.VisionDecode serve.test_server.ImageMarkers serve.test_server.VisionTempFiles serve.test_server.VisionShutdown serve.test_server.VisionCacheEviction serve.test_vision_elastic on this branch (main 4b9e05a2 + this commit), Linux, Python 3.14, CPU only: 30 tests OK. On stock 4b9e05a2 with the same tests: 16 failures and 2 errors (eviction before stat, eviction between stat and copy, out-of-order records for >64 images, untrimmed failure paths, the decoder retry).
  • python -m unittest discover -s serve -p 'test_*.py' on this branch: 1046 tests, 1 failure (below), 12 skipped.

Failed:

  • serve.test_keepawake.KeepAwakeTests.test_service_holds_for_the_whole_request_including_early_close fails intermittently on this branch and on stock 4b9e05a2 alike (2 of 3 runs on stock here). It is a timing test unrelated to vision.

Not run:

  • No GPU, model or device build. The race tests schedule two requests against the real cache and copy boundary without an encoder; they do not qualify a new image decoder.

Limits

The FIFO now holds longer per image request; I did not measure the throughput cost.

🤖 Generated with Claude Code

@Tsury
Tsury force-pushed the fix-2286-vision-ownership branch from 55ba1c2 to 2a82ae0 Compare October 28, 2026 10:07
@Tsury

Tsury commented Oct 28, 2026

Copy link
Copy Markdown
Contributor Author

Rebased onto current main (v0.2.7). The only overlap was the DeepSeek pictures change: the auto-pin line keeps its new and not ds_pics condition inside the image-ownership block. Locally, serve.test_vision_cache + serve.test_vision_elastic (18 tests) and serve.test_server (359 tests) pass.

@Tsury
Tsury force-pushed the fix-2286-vision-ownership branch from 30cadc8 to d62913f Compare November 3, 2026 00:36
@Tsury

Tsury commented Nov 3, 2026

Copy link
Copy Markdown
Contributor Author

Rebased onto v0.2.10. Two overlaps, both kept:

  • The all_cached check stays inside the image-ownership block, so a picture that is already encoded does not start the reader.
  • The ContextOverflow raises keep their prompt, context and max_new fields inside the re-indented block.

Locally, serve.test_vision_cache, serve.test_vision_elastic and serve.test_server pass: 387 tests, Linux, Python 3.12, mock engine. No GPU run on this rebase.

@Tsury
Tsury force-pushed the fix-2286-vision-ownership branch 5 times, most recently from a04e87d to 48d86c8 Compare November 20, 2026 09:38
@Tsury

Tsury commented Nov 20, 2026

Copy link
Copy Markdown
Contributor Author

CI note: the two failing jobs fail the same way on main, so they are not caused by this PR.

  • Linux / runtime requirements only: test_abort_sync fails with ModuleNotFoundError: No module named 'serve'. The same job fails on main and on v0.2.17.
  • Windows / Python terminal/tools/: test_strata_mcp.Install::test_a_start_that_came_in_before_a_cancel_does_not_launch_the_job also fails on main run 44699739037 (3142b66).

All other jobs pass.

@Niko1221

Copy link
Copy Markdown
Owner

Thank you for this pull request, and for the CI note. It keeps a request's pictures in the cache from encoding until the combined file is copied, so another request cannot free them in between. It is not in 0.2.17. It changes how every request holds the picture cache, and 0.2.17 already changed the same Vision code (the picture reader fix from #5900), so it needs a rebase on 0.2.17 and a test with more than 64 pictures before we take it. The PR stays open, and we will reply here once that has been done.

Tsury and others added 3 commits November 22, 2026 14:06
Keep the FIFO and a reentrant image batch lock through encoding,
validation and combined-file copying, defer eviction to the outermost
batch, and trim back to the 64-entry bound afterward. Close encoder
pipes on teardown; retry a native decoder refusal once through the
existing Pillow normalization.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s no batch()

A vision encoder without batch() (the HistoryReferences test stub) now prepares the request without the batch lock, as encode_all() already falls back to encode().
Tests that build a reader with Vision.__new__ (test_vision_workdir) have no
cache_batches, so encode() raised AttributeError. A class attribute keeps the
counter at 0 until batch() sets it on the instance.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Tsury
Tsury force-pushed the fix-2286-vision-ownership branch from 48d86c8 to c82afd0 Compare November 22, 2026 12:12
@Tsury

Tsury commented Nov 22, 2026

Copy link
Copy Markdown
Contributor Author

I rebased onto current main at c6f6427d (v0.2.19, which includes v0.2.17 and the picture-reader change). The PR head is c82afd00.

On Linux with Python 3.12, python -m unittest serve.test_vision_cache serve.test_vision_elastic serve.test_server passed all 479 tests. The cache test exercised 64, 65, 70, and 130 distinct pictures in one request. It checked the combined records in order and verified that the cache returned to 64 entries after preparation.

This run used the mock engine and picture reader. I did not run the case on a GPU. The GitHub checks for the rebased head are separate from these local results.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants