Skip to content

feat(channels): migrate doubao/douyin to thick fetch() with bounded retry - #66

Open
1012839419a-alt wants to merge 1 commit into
2233admin:mainfrom
1012839419a-alt:feat/thick-fetch-retry
Open

feat(channels): migrate doubao/douyin to thick fetch() with bounded retry#66
1012839419a-alt wants to merge 1 commit into
2233admin:mainfrom
1012839419a-alt:feat/thick-fetch-retry

Conversation

@1012839419a-alt

Copy link
Copy Markdown
Contributor

Summary

doubao_research and douyin_detail declared capabilities (default_rate, auth_kind) but never overrode fetch(), so run_channel treated them as unmigrated: no runner protocol, and CDP race failures ("CDP connection is not open", "Inspected target navigated or closed", "cloneNode") had no retry semantics — the outer worker had to brute-force retry, and a captcha wall was the only classified error.

Changes

  • doubao_research_channel: classify transient CDP races as ConnectionError (retryable in error_taxonomy) alongside the existing captcha_challenge classification; add fetch() override with exponential-backoff bounded retry (max_retries default 3, retry_base_delay default 2s). Captcha is NEVER auto-retried — the pipeline's captcha branch (PR fix(control): close the captcha governance loop #65) pauses the source for a human.
  • douyin_detail_channel: same transient classification on open/eval failures + same fetch() retry loop; permanent failures fail immediately.
  • collect() stays the single source of truth in both; ctx.http is not threaded through (subprocess transport, same trade-off as opencli_channel.fetch()).

Test Plan

  • doubao: 19 tests (7 new) — retry-then-success, give-up after max_retries, captcha/permanent never retried, metadata passthrough, CDP transient classification
  • douyin: 6 tests (3 new) — same retry semantics
  • channel contract + runner suites: green (38 passed total)

Note: kept independent of PR #65 (captcha governance) so each merges separately; uses inline captcha_challenge marker instead of is_captcha() for that reason.

…etry

Both session-affinity channels declared capabilities (default_rate,
auth_kind) but never overrode fetch(), so run_channel treated them as
unmigrated: no RateLimitedClient, no runner protocol, and CDP race failures
(CDP connection is not open, Inspected target navigated or closed,
cloneNode) had no retry semantics — the outer worker had to brute-force
retry, and a captcha wall was the only classified error.

- doubao_research_channel: classify transient CDP races as ConnectionError
  (retryable in error_taxonomy) alongside the existing captcha_challenge
  classification; add fetch() override with exponential-backoff bounded
  retry (max_retries default 3, retry_base_delay default 2s). Captcha is
  NEVER auto-retried — the pipeline's captcha branch (PR 2233admin#65) pauses the
  source for a human.
- douyin_detail_channel: same transient classification on open/eval
  failures + same fetch() retry loop; permanent failures fail immediately.
- collect() stays the single source of truth in both; ctx.http is not
  threaded through (subprocess transport, same trade-off as
  opencli_channel.fetch()).

Tests: 19 doubao (7 new), 6 douyin (3 new) — retry-then-success, give-up
after max_retries, captcha/permanent never retried, metadata passthrough,
CDP transient classification. channel contract + runner suites green.
@coderabbitai

coderabbitai Bot commented Aug 7, 2026

Copy link
Copy Markdown

Warning

Review limit reached

@1012839419a-alt, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 53 minutes

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 0bd0134e-f686-4b1a-9608-a1e5a401ad04

📥 Commits

Reviewing files that changed from the base of the PR and between 94ab53d and a440d81.

📒 Files selected for processing (4)
  • backend/channels/doubao_research_channel.py
  • backend/channels/douyin_detail_channel.py
  • tests/unit/channels/test_doubao_research_channel.py
  • tests/unit/channels/test_douyin_detail_channel.py

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@repowise-bot

repowise-bot Bot commented Aug 7, 2026

Copy link
Copy Markdown

⚠️ Health of changed files: 8.5 → 8.4 (-0.1)
🚨 Change risk: high, riskier than 73% of this repo's commits.

📋 At a glance
3 files changed health · 3 hotspots touched · 4 new findings introduced.

Files & modules (2)
  • backend (2 files)
    • backend/channels/douyin_detail_channel.py
    • backend/channels/doubao_research_channel.py
  • tests (2 files)
    • .../channels/test_douyin_detail_channel.py
    • .../channels/test_doubao_research_channel.py

✅ Health gate: passed

🔎 More signals (2)

🗺️ Change map

flowchart LR
  subgraph PR ["Changed in this PR (2 with dependents)"]
    f_backend_channels_doubao_research_channel_py["backend/channels/doubao_research_channel.py 🔥"]:::changed
    f_backend_channels_douyin_detail_channel_py["backend/channels/douyin_detail_channel.py 🔥"]:::changed
  end
  f_backend_channels_registry_py["backend/channels/registry.py"]
  f_backend_channels_doubao_research_channel_py --> f_backend_channels_registry_py
  f_backend_channels_douyin_detail_channel_py --> f_backend_channels_registry_py
  classDef changed fill:#dbeafe,stroke:#1d4ed8,color:#1e3a5f
  classDef warn fill:#fef3c7,stroke:#b45309,color:#78350f
  classDef guard fill:#dcfce7,stroke:#15803d,color:#14532d
Loading

Solid arrows: code that imports the changed files (1 direct dependent, from the last indexed snapshot). Dashed: history/tests.

🔥 Hotspots touched (3)

  • backend/channels/doubao_research_channel.py: 2 commits/90d, 2 dependents · primary owner: 2233admin (70%)
  • .../channels/test_doubao_research_channel.py: 2 commits/90d, 0 dependents
  • backend/channels/douyin_detail_channel.py: 1 commits/90d, 2 dependents · primary owner: 2233admin (100%)

👀 Suggested reviewers @2233admin


📊 See the full report for this PR
Your repo map with this PR's blast radius lit up, every caller of the contracts it changes, and health before and after. No sign-in. · ⭐ Star Repowise · 📥 Install bot · Silence on a single PR with [skip repowise] in the title · Per-repo toggle on repowise.dev/settings?tab=bot · Updated 2026-08-07 11:58 UTC

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant