[Bug] v0.9.3 堆外内存(ArrayBuffer)无界增长导致内核级 OOM，上游流错误后疑未销毁

[yangxiang92]

概述

v0.9.3 在生产环境反复出现 内核级 OOM（非 V8 heap OOM）：进程 anon-rss 涨到 13.514.2GB 被内核 SIGKILL，周期 47 分钟，单日 8 次并多次拖垮整机。

内存增长发生在 V8 堆外（ArrayBuffer backing store），因此 --max-old-space-size 无效，也不会产生 fatal report。

与 #1408 不同：#1408 是 V8 heap OOM（有 fatal report），其修复（#1414）已包含在 v0.9.3 中；本 issue 是堆外内存，失效模式和触发路径都不同。

环境

• 镜像：ghcr.io/ding113/claude-code-hub:v0.9.3
• Node：v26.7.0，V8 heap 上限 4.09GB
• 宿主：15GB 内存 / 8 核，Docker Compose（app 初始无 mem_limit）
• 配置：除 ENABLE_MULTI_PROVIDER_TYPES=true、STORE_SESSION_MESSAGES=false 外基本为默认值
• 负载：约 200~550 请求/分钟，客户端为 Claude Code 与 Codex，上游为第三方中转站

观测数据

用 5 秒分辨率采样器记录，起爆时自动抓取 smaps 与 socket 队列。

增长形态（一次典型起爆）：

09:22:22   813,868 kB
09:22:27  1,759,180 kB   +945 MB / 5s
09:22:33  2,676,196 kB   +917 MB / 5s
09:22:38  3,934,496 kB  +1258 MB / 5s
09:22:44  5,386,984 kB  +1452 MB / 5s

内存位置 —— 增长集中在单块匿名映射内，映射数量恒定：

爆炸期： 3,602,476 kB  @ 7f7567ff9000-7f7667ffa000   regions=1779
        （4GB 边界对齐，单块 > heap 上限 4.09GB → V8 cage 内的 ArrayBuffer）
正常期：    97,220 kB  @ 2bc40000-31c4c000            regions=1101

早期一次采样更明显：regions=1213 全程不变，而 total_anon 从 5.08GB 涨到 8.09GB —— 是既有映射原地长大，不是新建缓冲区。

socket 队列 —— 这是最有指向性的一组：

爆炸期：
  recvq=273,206  peer=<上游 CF>:443
  recvq=121,701  peer=<容器网关>:33186
  recvq=119,274  peer=<容器网关>:44196
  recvq=119,274  peer=<容器网关>:44170     ← 多条连接堆积完全相同的字节数
  recvq=119,274  peer=<容器网关>:33220
  recvq=119,274  peer=<容器网关>:33218
  total_recvq=1,337,790   total_sendq=0

正常期：
  total_recvq=878         total_sendq=694,177

爆炸时 recvq 大量积压而 sendq 归零，正常时恰好相反。多条连接堆积完全相同的 119,274 字节说明不是随机积压。

其他排除项：连接数在爆炸期恒定（48 / 61~72，不随内存增长）；线程恒定 11；data/reports/ 始终为空（V8 从未触及自身上限）；replay_payloads 仅 92MB；Redis 399MB。

暴涨期间应用完全静默：日志从 908 行/分钟衰减到 0，采样器记录 req5s=0，即无新请求进入时内存仍在以 370~500MB/s 增长。事件循环被饿死是结果而非原因。

可疑代码路径

起爆前的错误榜首是 undici body stream error (caught early)，一个容器生命周期内 141 次，同时伴随 Provider error occurred 122 次、All providers failed 46 次、熔断器触发 —— 上游中转站大面积出错时触发。

src/app/v1/_lib/proxy/forwarder.ts:8292 附近：

const rawBody = undiciRes.body as Readable;
rawBody.on("error", (err) => {
  const code = (err as NodeJS.ErrnoException).code;
  const isExpectedDisconnect = code === "ECONNRESET" || ...;
  const log = isExpectedDisconnect ? logger.debug : logger.warn;
  log("ProxyForwarder: undici body stream error (caught early)", {...});
});   // 仅记录日志，未调用 destroy() 或 resume()

同一函数内的 gzip 分支有对应的销毁处理（gunzip.destroy(err)，注释引用了 #1147 的段错误教训），但非 gzip 分支的 rawBody 自身错误路径没有。

需要说明：nodeStreamToWebStreamSafe 内部确实注册了 nodeStream.on("error", onError) 并会 controller.error(err) 向下游传播，所以错误并非完全未被处理。我们未能证明具体的对象保留链 —— 抓 heap snapshot 需要在泄漏进行中操作，而当时进程已达 13GB、宿主机在 thrashing，无法安全采集。

一个可能相关的组合是 v0.9.0 引入的需求驱动背压（node-stream-to-web.ts:71 默认 pause()，仅在 pull() 时 resume()）：上游流出错后如果下游不再 pull、且该流未被 destroy，socket 数据会持续留在内核 recvq，这与观测到的 recvq 积压 / sendq=0 吻合。这一点是推断，仅供参考。

版本线索

• rawBody.on("error") 只记日志的写法自 v0.3.20（2025-11-30，commit eecdb757）就存在
• nodeStream.pause() 需求驱动背压在 v0.9.0（2026-07-17，commit 3724315b / 3673105a）引入
• v0.8.10 的 node-stream-to-web.ts 中 pause() 出现 0 次

即单看那个 error handler 长期存在且未致泄漏，问题在 v0.9.x 才显现。

缓解措施与效果

给 app 容器加 mem_limit: 3g + memswap_limit: 3g（后来放宽到 6g）后：

	加之前	加之后
全机 OOM	8 次	0 次
load average 峰值	95.74	7 以下
最低可用内存	105MB	9GB+
故障影响面	整机（16 个容器）	仅 app 容器自愈重启

另外设 ENABLE_REQUEST_REPLAY=false 后观察 10.5 小时未复发，但负载回到白天水位（连接数 20 → 72）后立即再次爆发两次 —— replay 不是根因。

附带一个发现：本部署中 replay 命中率为 0（ReplaySpool 完成 5694 次，日志与源码中均无缓存取用事件）。原因是 replayId 由 bodyHash 派生，而 Claude Code 在流中断后会把已接收内容并入对话历史再发起新请求，body 变化导致 replayId 永不匹配。这与本 issue 无关，但或许值得单独关注。

建议

1. rawBody.on("error") 中补充 rawBody.destroy(err)（或至少 resume() 排空），与同函数 gzip 分支保持一致
2. 为堆外缓冲增加全局配额 —— 目前 REPLAY_MAX_PAYLOAD_BYTES（8MB）、STREAM_GATE_PREBUFFER_BYTE_CAP（10MB）都是单流上限，并发流总量无天花板
3. 建议在 docker-compose.yaml 中为 app 服务提供默认 mem_limit，避免单进程失控拖垮整机

复现条件

无法稳定构造，但触发条件较明确：上游供应商大面积返回错误或断流（表现为 undici body stream error 密集出现、All providers failed、熔断器触发），同时保持 200+ 请求/分钟的并发流式负载。我们环境下白天高峰期必现，夜间低负载时不出现。

如需进一步数据（完整采样日志、smaps 序列、socket 队列快照）可以提供。

[coderabbitai]

🔗 Related PRs

#1152 - fix: harden proxy stream lifecycle to avoid SIGSEGV under load (#1147) [merged]
#1351 - feat(observability): add request-level Discovery routing trace [merged]
#1370 - fix(proxy): prevent failed responses from entering replay [merged]
#1413 - fix(replay): bound detached stream memory (#1408) [open]
#1414 - fix(proxy): bound Replay disconnect memory retention (#1408) [merged]

---

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

• Create Plan

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

[github-actions]

Thank you for this exceptionally thorough report. The off-heap vs. V8-heap distinction, the smaps/socket-queue evidence, and the version bisection are all well-founded. I've traced the code and can corroborate the core hypothesis, and add one concrete gap the destroy() path exposes.

Distinction from #1408

Confirmed. #1408 was a V8-heap OOM (fatal report), fixed by #1414 (fix(proxy): bound Replay disconnect memory retention, merged 2026-08-11, in v0.9.3). This issue is off-heap ArrayBuffer backing-store growth, a different failure mode.

Verified findings

1. rawBody.on("error") is log-only — src/app/v1/_lib/proxy/forwarder.ts:8293-8310

Confirmed: the handler logs and returns. It calls neither destroy() nor resume(). The comment at :8296 explicitly notes the code set must stay in sync with the streamPipeline callback below.

2. Asymmetry with the gzip branch — forwarder.ts:8376-8390

Confirmed: gunzip.on("error") logs AND calls gunzip.destroy(err) (:8385). The non-gzip rawBody error path has no equivalent. The gzip path additionally uses stream.pipeline() (:8394), which guarantees mutual destruction of both ends — a guarantee the non-gzip path (which hands rawBody directly to nodeStreamToWebStreamSafe, :8427) does not have.

3. nodeStreamToWebStreamSafe.onError does not destroy the Node stream — src/app/v1/_lib/proxy/node-stream-to-web.ts:113-128

Confirmed. On a Node-stream 'error' it sets settled, detach()es all listeners, and calls controller.error(err) — but does NOT call nodeStream.destroy(). The only place destroy() is called is cancel() (:184-209). So destroy is reachable only when the Web-stream side is cancelled.

4. Demand-driven backpressure — node-stream-to-web.ts:71, 178-182

Confirmed: start() calls nodeStream.pause() (:71); resume() happens only inside pull() (:180); onData re-pauses when controller.desiredSize <= 0 (:138-139). Matches the v0.9.0 introduction you cite (commits 3724315b / 3673105a, 2026-07-17; pause() is absent from the pre-9.0 version).

Additional gap: the error path never reaches cancel()/destroy()

This strengthens the "never destroyed" hypothesis with a specific mechanism. The demand-driven pump (src/app/v1/_lib/proxy/demand-driven-response-pump.ts) settles errors as follows:

• Upstream body errors -> nodeStreamToWebStreamSafe.onError -> controller.error(err) -> the pump's reader.read() rejects -> finishWithError(error) (:189) -> settle(false, error) (:97-108).
• settle() only calls reader.cancel(sourceCancelReason) when a sourceCancelReason is passed (:81). finishWithError passes none.
• By contrast, cancelSource() (:143-147) explicitly forwards the reason, and the pending-chunk deadline (:149-159) calls both startDrain and cancelSource.

Net effect: on a pure upstream stream error (the exact trigger in your report — dense undici body stream error, no client abort), neither nodeStreamToWebStreamSafe.cancel() nor nodeStream.destroy() is ever reached. The undici body is left errored-but-not-destroyed. Per undici's socket lifecycle, an unconsumed/undestroyed body means the socket is not returned to the pool — which lines up with your observation of growing recvq with sendq=0 and a constant connection count.

The part I cannot independently verify from the code alone is the exact retention link between that stalled socket buffer and the single 4GB-aligned anonymous mapping (V8 ArrayBuffer backing store) your smaps captured. That the stall correlates tightly with the recvq explosion is clear; whether the backing-store growth is a downstream consequence (e.g. unconsumed enqueued chunks on tee'd branches) would need an in-flight heap snapshot to confirm — which, as you note, is impractical to capture once the host is thrashing at 13GB.

Config / mitigations

Confirmed the two caps are per-stream only, with no global aggregate ceiling:

• STREAM_GATE_PREBUFFER_BYTE_CAP default 10 * 1024 * 1024 — src/lib/config/env.schema.ts:201-206
• REPLAY_MAX_PAYLOAD_BYTES default 8 * 1024 * 1024 — src/lib/config/env.schema.ts:218-223

Your mem_limit mitigation is sound for blast-radius containment, and your finding that ENABLE_REQUEST_REPLAY=false did not prevent recurrence (replay hit-rate 0 in your deployment) is consistent with replay being orthogonal to this path.

Summary of actionable observations

1. The non-gzip rawBody error path lacks the destroy/resume that the gzip branch and the streamPipeline callback provide — forwarder.ts:8293-8310.
2. nodeStreamToWebStreamSafe never destroys the Node stream from its own onError; destroy is gated behind cancel() — node-stream-to-web.ts:113-128, 184-209.
3. The pump's upstream-error settle does not call reader.cancel(), so cancel()/destroy() is unreachable on a pure upstream error — demand-driven-response-pump.ts:97-108, 189.
4. No global bound on concurrent off-heap buffering exists; only per-stream caps.

These are observations from the current source, not an applied fix. Thank you again for the detailed instrumentation data — it significantly narrows the search.

---

Automated response from Claude AI

[yangxiang92]

补充：第二组独立样本（同日 10:22）

同一环境再次复现，形态与首例一致，且提供了上游错误率与泄漏量的同向变化证据。

增长曲线：

10:22:32  4,098,928 kB  +1347 MB / 5s
10:22:37  5,427,764 kB  +1329 MB / 5s
10:22:43  重启 → 237,756 kB

两轮爆炸对照：

	09:22	10:22
undici body stream error	141	303
Provider error occurred	122	349
All providers failed	46	156
total_recvq 峰值	1,337,790	2,123,839
RSS 峰值	5.62 GB	5.43 GB

上游错误数翻倍，recvq 积压也接近翻倍。

socket 队列（爆炸期 → 重启后）：

爆炸期： total_recvq=2,123,839   total_sendq=0
         recvq=119274  peer=<容器网关>:47038
         recvq=119274  peer=<容器网关>:47022
         recvq=119274  peer=<容器网关>:47006
重启后： total_recvq=0           total_sendq=191

值得注意：119,274 这个字节数在两轮爆炸中反复出现于多条不同连接上，指向某段固定长度内容被重复缓冲，而非随机积压。

此外本轮伴随 All providers rate limited 178 次，即上游限流也会走进同一条路径。