starknet_committer: buffer streamed compress/decompress for state commitment infos

[gkaempfer]

Summary

Automated performance follow-up, scoped to the CompressedStateCommitmentInfos codec introduced in #14884 (which reached main around the same time as the branch-sync #14883 that triggered this review).

StateCommitmentInfos::compress() / CompressedStateCommitmentInfos::decompress() previously buffered the full uncompressed bincode payload in an intermediate Vec<u8> before/after feeding it to zstd. This runs once per block on the committer↔batcher critical path, and the payload is exactly the data #14884 needed to compress in the first place because it was large enough to blow past the RPC response cap.

This PR streams bincode serialization directly into a zstd encoder (and zstd-decompressed bytes directly into bincode deserialization), wrapped in BufWriter/BufReader so bincode's small per-field writes/reads are coalesced before crossing into zstd's FFI layer.

Net effect, measured with a tracking allocator on realistic payloads (~18.8 MB uncompressed):

• Peak allocation: compress 23.1 MB → 7.1 MB, decompress 96.1 MB → 62.7 MB
• Speed: ~24% faster compress, ~5% faster decompress vs. the original buffered implementation
• Compressed output is byte-identical to the original implementation (verified both directions: old-format frames decode via the new path and vice versa), so on-disk data written by the original code stays readable.

Test plan

• cargo test -p starknet_committer --features os_input — 129 passed
• cargo test -p apollo_committer --features os_input — 14 passed (round-trips through compress()/decompress() via read_paths_and_commit_block_happy_flow, revert_removes_witnesses_and_digest, etc.)
• cargo test -p apollo_storage --features os_input state_commitment_infos — 2 passed
• cargo clippy -p starknet_committer --features os_input --all-targets -- -D warnings — clean
• scripts/rust_fmt.sh --check — clean
• Reviewed by an independent Opus pass, which verified correctness (byte-identical round-trip, no panics, matching bincode configs) and identified that the naive streaming version alone was actually a memory-for-CPU tradeoff — the BufWriter/BufReader wrapping in this PR was added specifically to address that and make it an unambiguous win.

Note

This is scoped to the codec itself. The review also surfaced a pre-existing, higher-severity issue outside this diff's scope: echonet/os_input_builder.py still does json.loads() on what is now a bincode-encoded payload (a consequence of #14884), which will break echonet OS runs — flagging separately rather than fixing here since it's unrelated to this optimization.

---

🤖 Generated with Claude Code

---

Generated by Claude Code

[reviewable-StarkWare]

This change is 

[cursor]

PR Summary

Low Risk
Localized codec refactor with the same bincode/zstd semantics and byte-identical output; no API or storage format changes beyond implementation.

Overview
StateCommitmentInfos::compress() and CompressedStateCommitmentInfos::decompress() no longer build a full uncompressed bincode Vec in memory. Compression now uses bincode::serialize_into into a BufWriter around a zstd::Encoder, and decompression uses bincode::deserialize_from on a BufReader over a zstd::Decoder, so bincode’s small per-field I/O is batched before hitting zstd.

Docs on both methods explain why buffering is required. Wire format and compression level are unchanged, so existing stored frames should still round-trip.

^{Reviewed by Cursor Bugbot for commit dedd21c. Bugbot is set up for automated code reviews on this repo. Configure here.}