Open streaming responses do not keep cells active, and eviction does not terminate their response bodies

[bplotnick]

Description

celld considers a Durable Object request finished once the Worker returns its
HttpResponse. The request's activity guard does not remain attached to a
streaming response body, and subsequently forwarded chunks do not otherwise
update the cell's last_used_mono_ms.

As a result, an active SSE response that emits periodic heartbeat comments can
still become eligible for CELLD_IDLE_EVICT_S eviction. When the cell is
evicted, celld stops its residency/runtime but does not deterministically
complete or fail response bodies that were already handed to the HTTP server.
The client can therefore be left with an open body whose producer has stopped,
rather than receiving a clean EOF.

For a gzip-compressed response, a later proxy or transport timeout can surface
this as a truncated gzip body, for example:

java.io.EOFException: source exhausted prematurely
    at okio.InflaterSource.read(InflaterSource.kt:45)
    at okio.GzipSource.read(GzipSource.kt:67)

Reproduction

Regression/characterization tests based on upstream v0.3.0
(89e4ffc53a14ecb496d2ca5014ff9d19b0061ad9) are available on this branch:

https://github.com/bplotnick/celld/tree/repro/sse-idle-eviction

The reproduction commit is
d6b2b9a.

The patch contains two focused tests:

1. response_body_chunks_do_not_refresh_the_idle_lease models a resident cell
   with a one-second idle lease and SSE heartbeats at 250, 500, and 750 ms.
   Because there is no response-body activity event to feed into the decision
   core, the 1,000 ms resource sample selects the cell for eviction. This
   characterization test passes on the affected revision.
2. evicting_a_streaming_cell_orphans_the_response_body runs a real V8 Durable
   Object returning a ReadableStream, observes its SSE heartbeat, evicts the
   cell, and requires the response body to complete. It fails on the affected
   revision with eviction left the response body open.

Run them with:

cargo test -p celld-logic response_body_chunks_do_not_refresh_the_idle_lease -- --nocapture
cargo test -p celld evicting_a_streaming_cell_orphans_the_response_body -- --nocapture

Observed results:

test response_stream_repro_tests::response_body_chunks_do_not_refresh_the_idle_lease ... ok

thread 'runtime::tests::evicting_a_streaming_cell_orphans_the_response_body' panicked:
eviction left the response body open

We also reproduced this through a deployed HTTP endpoint with
CELLD_IDLE_EVICT_S=300:

• The SSE client received : ping every 15 seconds through T+315s.
• Heartbeats then stopped.
• No final SSE frame, clean EOF, or response error arrived.
• The HTTP request remained pending until the client test timed out at T+390s.
• Durable application state remained intact and reconnecting immediately
  produced : connected followed by new heartbeats.

The timing is consistent with the 300-second idle lease plus the next eviction
or isolate-maintenance pass.

Expected behavior

To match Cloudflare Durable Object semantics, celld should hold request
activity for the lifetime of the response body. An open streaming response
should therefore prevent ordinary idle eviction and hibernation until the body
closes, errors, or is canceled by the client. This should not require
application code to call waitUntil() or emit periodic chunks merely to retain
the runtime.

Forced lifecycle operations still need a bounded escape hatch. If celld must
stop the cell for shutdown, handoff, fencing, or another non-idle reason, every
response stream owned by that cell should be deterministically terminated and
its writer bookkeeping released. For an HTTP response whose headers have
already been sent, this should result in a prompt, well-defined client-visible
completion/cancellation rather than an orphaned body that waits for a
downstream timeout.

Cloudflare documents that Durable Objects remain active while handling response
streams and that waitUntil() is unnecessary for this behavior:

https://developers.cloudflare.com/workers/runtime-apis/context/#waituntil

Cloudflare's lifecycle documentation also excludes a request/event that is
still being processed from hibernation eligibility and describes bounded
handling during runtime shutdown:

https://developers.cloudflare.com/durable-objects/concepts/durable-object-lifecycle/

Suggested direction

• Associate every forwarded response stream with its owning cell and request.
• Transfer or extend the request's activity guard into the response body so it
  is released only when the body closes, errors, or is canceled.
• Exclude stream-active cells from ordinary idle eviction, just as celld
  already excludes an actively executing request.
• During forced runtime stop, close or cancel all response streams registered
  to the cell and remove their response-stream writer entries before discarding
  the runtime.
• Bound forced shutdown with an explicit grace period so a broken or malicious
  infinite stream cannot prevent node termination indefinitely.

This should apply to all streaming HTTP response bodies, not only SSE.