Repository navigation
fix(datasource): 语雀/Notion/IMA/飞书连接器 API 响应体补体积上限 - #3985
Merged
Merged
Conversation
#3887 给 GitLab 客户端补了 readCapped 上限;语雀、Notion、IMA、飞书四个 连接器的 API 读取仍是裸 io.ReadAll,破损或恶意端点可让同步进程缓冲无界 响应体。各连接器比照 GitLab 同步同一上限(16 MiB)与显式超限报错,limit 做成字段以便测试下调。飞书 429/5xx 错误路径的 best-effort 读取刻意不动。
测试里直接用字面量构造的 Client 不经过 NewClient,jsonLimit 为 0, readCapped 会拒绝任何响应,导致 feishu/core 11 个现有测试失败。 readCapped 遇到非正数 limit 时改用 maxJSONResponseBytes。
Collaborator
|
感谢 @yetuge!我推了一个小补丁:测试里直接用字面量构造的 |
- yuque/ima/feishu/notion 的 readCapped 改用哨兵错误 errResponseTooLarge, 文案不变;yuque doRequest、ima callAPIAt、feishu DoRequest 遇到超限直接返回, 不再按退避重试 3 次(同一请求必然再次超限,只会白耗限流额度)。 - 飞书 GetTenantAccessToken 改为 readCapped + json.Unmarshal,和其他 API 响应同一上限。 - 飞书 downloadRawBytes 的 429/5xx/非 200 错误预览只读 4 KiB, download failed 错误不再携带完整响应体。
Collaborator
|
感谢 @yetuge!合之前做了一轮交叉复审,在你的基础上又追加了一个提交:
你在描述里说过下载诊断读取刻意不改,这里还是一起补了,因为截断是在整段读入内存之后才做的,内存风险并不会因此变小。飞书 blocks 模式内嵌表格按行窗口读取的问题我另开 issue 跟进。 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
问题
#3887 给 GitLab 连接器补了
readCapped响应体上限,确立了「所有连接器 HTTP 响应体在缓冲前必须有界」的规范("a larger body means a broken or hostile server")。四个更早接入的连接器 API 路径仍是裸io.ReadAll(resp.Body),破损或恶意端点可以让同步进程把任意大的响应体读进内存:internal/datasource/connector/yuque/client.go—doRequestinternal/datasource/connector/notion/client.go—doRequestinternal/datasource/connector/ima/client.go—callAPIAtinternal/datasource/connector/feishu/core/client.go—DoRequest这些路径拉取的元数据/文档/块载荷按契约都很小;同一批连接器的下载路径本来就有上限(
maxDownloadSize/maxDownloadBytes/maxFeishuDownloadBytes),唯独 API 读取漏了。修法
逐连接器完全比照 #3887:
maxJSONResponseBytes = 16 << 20常量(与 GitLab 客户端同值);jsonLimit字段,测试可以下调而不必真造几百 MB;readCapped帮助函数替换裸io.ReadAll;response exceeds maximum size (N bytes)显式报错,走各客户端既有的错误/重试路径。飞书 429/5xx 错误路径上的
body, _ := io.ReadAll(...)(仅为日志截断预览服务的 best-effort 读取)刻意保持原样:纯诊断用途、形态是有意的,收窄它们超出本 PR 范围。验证
每个连接器各一个测试:下调
jsonLimit后由httptest服务端返回 1 MiB+1 的响应体,断言请求以显式超限错误失败而不是静默缓冲:TestClient_DoRequest_RejectsOversizedResponse(yuque)TestDoRequest_RejectsOversizedResponse(notion、feishu/core)TestCallAPI_RejectsOversizedResponse(ima)把 cap 调用还原成
io.ReadAll时四个新测试全部失败(无界缓冲行为),恢复修复后全部通过。与 main 基线对照,四个包的失败集完全一致(仅TestParseYuqueConfig、TestDownloadFile_RejectsLoopbackURL两个与本次改动无关的环境型失败),零回归。验证环境:golang 1.27 容器(对应 go.mod 的 1.26 指令),go test ./internal/datasource/connector/{yuque,notion,ima}/... ./internal/datasource/connector/feishu/core/...。