feat(channels): migrate doubao/douyin to thick fetch() with bounded retry - #66
feat(channels): migrate doubao/douyin to thick fetch() with bounded retry#661012839419a-alt wants to merge 1 commit into
Conversation
…etry Both session-affinity channels declared capabilities (default_rate, auth_kind) but never overrode fetch(), so run_channel treated them as unmigrated: no RateLimitedClient, no runner protocol, and CDP race failures (CDP connection is not open, Inspected target navigated or closed, cloneNode) had no retry semantics — the outer worker had to brute-force retry, and a captcha wall was the only classified error. - doubao_research_channel: classify transient CDP races as ConnectionError (retryable in error_taxonomy) alongside the existing captcha_challenge classification; add fetch() override with exponential-backoff bounded retry (max_retries default 3, retry_base_delay default 2s). Captcha is NEVER auto-retried — the pipeline's captcha branch (PR 2233admin#65) pauses the source for a human. - douyin_detail_channel: same transient classification on open/eval failures + same fetch() retry loop; permanent failures fail immediately. - collect() stays the single source of truth in both; ctx.http is not threaded through (subprocess transport, same trade-off as opencli_channel.fetch()). Tests: 19 doubao (7 new), 6 douyin (3 new) — retry-then-success, give-up after max_retries, captcha/permanent never retried, metadata passthrough, CDP transient classification. channel contract + runner suites green.
|
Warning Review limit reached
Next review available in: 53 minutes You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (4)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
📋 At a glance Files & modules (2)
✅ Health gate: passed 🔎 More signals (2)🗺️ Change map flowchart LR
subgraph PR ["Changed in this PR (2 with dependents)"]
f_backend_channels_doubao_research_channel_py["backend/channels/doubao_research_channel.py 🔥"]:::changed
f_backend_channels_douyin_detail_channel_py["backend/channels/douyin_detail_channel.py 🔥"]:::changed
end
f_backend_channels_registry_py["backend/channels/registry.py"]
f_backend_channels_doubao_research_channel_py --> f_backend_channels_registry_py
f_backend_channels_douyin_detail_channel_py --> f_backend_channels_registry_py
classDef changed fill:#dbeafe,stroke:#1d4ed8,color:#1e3a5f
classDef warn fill:#fef3c7,stroke:#b45309,color:#78350f
classDef guard fill:#dcfce7,stroke:#15803d,color:#14532d
Solid arrows: code that imports the changed files (1 direct dependent, from the last indexed snapshot). Dashed: history/tests. 🔥 Hotspots touched (3)
👀 Suggested reviewers @2233admin 📊 See the full report for this PR |
Summary
doubao_researchanddouyin_detaildeclared capabilities (default_rate,auth_kind) but never overrodefetch(), sorun_channeltreated them as unmigrated: no runner protocol, and CDP race failures ("CDP connection is not open", "Inspected target navigated or closed", "cloneNode") had no retry semantics — the outer worker had to brute-force retry, and a captcha wall was the only classified error.Changes
doubao_research_channel: classify transient CDP races asConnectionError(retryable inerror_taxonomy) alongside the existingcaptcha_challengeclassification; addfetch()override with exponential-backoff bounded retry (max_retriesdefault 3,retry_base_delaydefault 2s). Captcha is NEVER auto-retried — the pipeline's captcha branch (PR fix(control): close the captcha governance loop #65) pauses the source for a human.douyin_detail_channel: same transient classification on open/eval failures + samefetch()retry loop; permanent failures fail immediately.collect()stays the single source of truth in both;ctx.httpis not threaded through (subprocess transport, same trade-off asopencli_channel.fetch()).Test Plan
Note: kept independent of PR #65 (captcha governance) so each merges separately; uses inline
captcha_challengemarker instead ofis_captcha()for that reason.