io-and-byte-streams.md

io-and-byte-streams — notes

Hand-written. ../harvested/io-and-byte-streams.md is what the documents say; this file is what the implementation found, and it wins. Each entry names the harvested entry it answers by that entry's stable key.

Superseded

• IO-1's read primitive and IO-16's host-native bridge cannot be the same Ruby method, and reading the two entries together as one #read corrupts every streamed upload. Supersedes io-and-byte-streams/ba53c43b (IO-1: "A source read MUST append bytes to the tail of a caller-provided destination buffer (never overwriting existing content) and return the number transferred … -1 at end-of-stream") where it is read as naming a method called read, and narrows io-and-byte-streams/f009344c (IO-16) in the same breath. Verified on 2026-09-08 against 3.2.11, 3.4.10 and 4.0.6. The fact: IO.copy_stream — which is literally what Net::HTTP#send_request_with_body_stream calls, so it is the code path every body_stream= upload takes — drives a duck-typed source by checking respond_to?(:to_path), then respond_to?(:readpartial), then calling readpartial(16384, buf) with one and the same buf object on every call, expecting it overwritten; with no #readpartial it falls back to read(16384, buf) with the identical expectation. Ruby's own IO#read(n, buf) and StringIO#read(n, buf) overwrite the destination on all three interpreters. IO-1 requires the exact opposite. A Dexpace::IO::BufferedSource that satisfied IO-1 under the name #read or #readpartial would therefore append the whole stream into copy_stream's single reused buffer and write ever-growing garbage to the socket — a corruption no unit test of #read alone would show, because it only appears when a foreign driver reuses the buffer. What the SDK does instead, per docs/work/mvp/phase3/phase3a/2026-09-08-phase3a-io-contracts-design.md: IO-1's primitive is #read_into(dest, count:) with the tail-append, the four return values and the -1 sentinel, while #read, #readpartial, #getbyte, #readbyte and #each keep Ruby's own semantics and are IO-16's bridge — which is what makes docs/sdk-design-ruby/03-seam-by-seam-idiomatic-mapping.md §3.1's "IO-16's native-stream bridge is satisfied by construction" true rather than merely asserted. Two consequences that travel with the split. IO-16's "-1 at end" is the reference host's InputStream convention and the requirement's own word is host-native, so the bridge signals end of stream with nil (#read, #getbyte) and by raising (#readpartial, #readbyte); the portable vocabulary keeps -1, because IO-17's pump reads it and a foreign source implementing IO-1 literally would return it. And the raised class must inherit ::EOFError: verified that IO.copy_stream terminates cleanly on an EOFError subclass raised by a duck-typed #readpartial on all three interpreters, so Dexpace::EndOfStreamError < ::EOFError is load-bearing rather than tidy — outside that family copy_stream propagates and every streaming upload phase 8 performs fails. _{review · docs/work/mvp/phase3/phase3a/2026-09-08-phase3a-io-contracts-design.md · high · sha:manual-phase3a-read-into-split}
• force_encoding is unusable on the ingress path, because the chunks arriving there are frozen; String#b is the retag idiom. Supersedes io-and-byte-streams/0a4773a6 ("String#b is the idiom for representing 'these bytes, untagged,' while force_encoding is a retag used only where the bytes are known to conform"), which is true and incomplete: force_encoding also requires a String you own. Verified on 2026-09-08 against 3.2.11, 3.4.10 and 4.0.6. The fact: force_encoding on a frozen String raises FrozenError even when the target encoding is already the string's own — "abc".b.freeze.force_encoding(Encoding::BINARY) raises on all three. And the chunks reaching Dexpace::IO::BufferedSource.over(body) are routinely frozen: design §10.2's canonical body representation is Rack's #each-yielding-chunks protocol, a Rack body is commonly a literal array, and every file in this repository carries # frozen_string_literal: true, so ["chunk", "more"].each yields two frozen strings. A retag written the obvious way therefore raises on the most ordinary body there is. String#b always returns a new, unfrozen copy (fz.b.equal?(fz) is false and fz.b.frozen? is false on all three), so it is safe where force_encoding is not. What the SDK does instead: a chunk arriving from an upstream or a wrapped body is stored as-is when it is both frozen and already Encoding::BINARY — frozen means it cannot become caller-mutable state, so no copy is owed — and as chunk.b otherwise; force_encoding appears nowhere on the ingress path. The companion trap, verified in the same run and the reason every encoding test in phase 3a uses non-ASCII bytes: appending a non-ASCII UTF-8 String to a BINARY one silently retags the result to UTF-8, while appending an ASCII-only one leaves it BINARY, and #concat behaves identically — so an ASCII-only fixture passes under exactly the bug the rule exists to prevent. io-and-byte-streams/ff0de47e and /d2b47c89 are unaffected and are adopted verbatim. _{review · docs/work/mvp/phase3/phase3a/2026-09-08-phase3a-io-contracts-design.md · high · sha:manual-phase3a-frozen-ingress-retag}
• The single decode boundary is two steps, not one, and the transcode must name its target encoding explicitly. Supersedes io-and-byte-streams/fbcb4d19 ("There is exactly one decode boundary, Response#body_string, which applies the media type's charset via String#encode(invalid: :replace, undef: :replace) and falls back to UTF-8 when the charset is absent or unknown, per HTTP-42"), whose rule is right — one boundary, the declared charset, UTF-8 as the fallback — and whose recipe destroys the payload. Verified on 2026-09-08 against 3.2.11, 3.4.10 and 4.0.6. The fact, in two halves. (a) The bytes reaching Response#body_string are Encoding::BINARY, because io-and-byte-streams/d2b47c89 requires the port to retag every response body to BINARY on ingress and never trust a transport's tagging. From BINARY, every byte ≥ 0x80 is an undefined character in the source encoding, so String#encode with undef: :replace replaces it: "café".b.encode(::Encoding::UTF_8, invalid: :replace, undef: :replace) returns "caf" followed by two U+FFFD replacement characters — one per byte of the two-byte UTF-8 é — on all three interpreters, and is not == to the correct answer. The bytes must be retagged to the declared charset first, which is what Dexpace::IO::TypedReads#read_string(encoding) already does, and only then transcoded. (b) The cited call takes no target argument, and String#encode(invalid: :replace, undef: :replace) with none converts to Encoding.default_internal — a process-global the host application sets. With Encoding.default_internal = ::Encoding::ISO_8859_1, the same call on the same string returns an ISO-8859-1 result with the accented character destroyed, while the explicit-target form is unaffected; verified both ways on all three. That is the same passes-where-you-look shape docs/sdk-design-ruby/03-seam-by-seam-idiomatic-mapping.md §3.5 pins URI::RFC3986_PARSER against and that IO-14 avoids IO#gets and $/ for: a library must not let a host global decide what its decode returns. What the SDK does instead, per docs/work/mvp/phase3/phase3b/2026-09-08-phase3b-body-lifecycle-design.md: resolve the charset from Dexpace::MediaType#charset, which is already nil for an absent or unknown-to-this-Ruby charset so Encoding.find cannot raise and HTTP-42's fallback needs no second validation; retag through 3a's #read_string; then #encode(enc, invalid: :replace, undef: :replace) with both encodings named. The result is valid_encoding? in every case tested — UTF-8 bytes declared UTF-8, ISO-8859-1 bytes declared ISO-8859-1, and ISO-8859-1 bytes mis-declared as UTF-8, which scrubs to a replacement character rather than raising. On phase 10's inbound list as §3.1's decode sentence, because the sentence in that frozen design chapter is what a later reader will copy. io-and-byte-streams/d2b47c89 is unaffected and is adopted verbatim. _{review · docs/work/mvp/phase3/phase3b/2026-09-08-phase3b-body-lifecycle-design.md · high · sha:manual-phase3b-decode-retag-then-transcode}