Repository navigation
Conversation
PlaywrightCrawler copied the browser cookie jar into the session only after the request handler returned. ImpitHttpClient, HttpxHttpClient, and CurlImpersonateHttpClient send cookies from that jar, so send_request inside the handler still used the pre-navigation cookies. Copy the jar before the handler as well. The copy after the handler stays, so cookies created while the handler runs still persist. PlaywrightHttpClient is unchanged and keeps reading the live browser cookies. Fixes apify#2483
send_request in PlaywrightCrawler
| # Session-backed HTTP clients read this jar from inside the handler. Copy again after the | ||
| # handler returns so cookies created while it runs stay on the session. |
There was a problem hiding this comment.
Nit: "this jar" doesn't point to anything on the next line, and CONTRIBUTING.md asks to wrap comments to the full 120-character width. Suggestion:
| # Session-backed HTTP clients read this jar from inside the handler. Copy again after the | |
| # handler returns so cookies created while it runs stay on the session. | |
| # Session-backed HTTP clients read the session cookie jar inside the handler. Copy again after the handler returns | |
| # so cookies created while it runs stay on the session. |
✍️ Drafted by Claude Code
There was a problem hiding this comment.
Done in d3ec31a — used your wording and wrapped the comment to 120 characters.
|
|
||
| await self._persist_browser_cookies(context) | ||
|
|
||
| async def _persist_browser_cookies(self, context: TPostNavContext) -> None: |
There was a problem hiding this comment.
Nit: In crawlee, "persist" usually means saving to a KVS. This helper only copies into the in-memory session, as its docstring says. Maybe rename it to _copy_browser_cookies_to_session?
✍️ Drafted by Claude Code
There was a problem hiding this comment.
Agreed — renamed to _copy_browser_cookies_to_session in d3ec31a so it is not confused with persisting to a KVS.
| async def _create_crawling_context(self, context: TPostNavContext) -> AsyncGenerator[TCrawlingContext, None]: | ||
| # Session-backed HTTP clients read this jar from inside the handler. Copy again after the | ||
| # handler returns so cookies created while it runs stay on the session. | ||
| await self._persist_browser_cookies(context) |
There was a problem hiding this comment.
Note: This copy runs after the post-navigation hooks, so send_request in a post-navigation hook still sends the cookies from before navigation. crawlee JS behaves the same and #2483 is only about the handler, so it's fine to leave as is. Moving the copy to the end of _navigate would cover the hooks too.
✍️ Drafted by Claude Code
There was a problem hiding this comment.
Agreed, I will leave the copy where it is. #2483 is about the handler, and this matches the JS crawler's timing. Happy to move it to the end of _navigate in a follow-up if you want the hooks covered too.
|
|
||
| yield self._build_crawling_context(context) | ||
|
|
||
| await self._persist_browser_cookies(context) |
There was a problem hiding this comment.
Note: Older than this PR, not a blocker. The copy goes one way only, so if send_request stores token=B in the session during the handler and the browser still has token=A, this copy writes token=A back over it. Could be a follow-up issue.
✍️ Drafted by Claude Code
There was a problem hiding this comment.
Agreed — that one-way overwrite is older than this PR and not needed to fix #2483. I can open a follow-up issue for a merge (or a session-wins rule) if you would like one.
apify-factory
left a comment
There was a problem hiding this comment.
Looks good, nothing blocking.
🏭 Factory review — ℹ️ not auto-approved
This change was reviewed but cannot be auto-approved: a gate failed, so the review cannot end in an approval.
| check | result |
|---|---|
| Static gates | ❌ 14/16 — author-allowed, same-repo |
| Review findings | ✅ none |
| Reviewer escalation | ❌ Changes when browser cookies are synced into the session jar, which touches session handling in a public crawler. |
Juror 1 — claude-sonnet-5-5, adversarial |
⏭️ skipped — a gate failed |
Why it was not auto-approved
Failed gates:
author-allowed: rome-xi: not a member of apify/product-engineeringsame-repo: head repo is rome-xi/crawlee-python
The comments above are the review itself; this section only explains the approval decision. Push a change and request @apify-factory again to re-run it, or ask a human to review.
Reviewed 854e10cc5 · Review details on the dashboard · workflow run
Address review nits on apify#2485: wrap the session-cookie comment to 120 characters, and rename _persist_browser_cookies to _copy_browser_cookies_to_session so it is not confused with KVS persist.
|
Thanks @vdusek — addressed the nits in d3ec31a:
I left the copy after the post-navigation hooks (same as the JS crawler, and enough for #2483) and did not change the pre-existing one-way overwrite. Happy to follow up on either of those if you want. |
Problem
PlaywrightCrawleronly copied browser cookies intocontext.session.cookiesafter the request handler returned. Custom HTTP clients (ImpitHttpClient,HttpxHttpClient, and similar) read the session jar, socontext.send_request(...)inside the handler sent the pre-navigation cookies. The defaultPlaywrightHttpClientwas unaffected because it reads live browser cookies.Fixes #2483.
Change
Copy browser cookies into the session after navigation / before the handler, and keep the existing post-handler copy so cookies set during the handler still persist.
PlaywrightHttpClientis unchanged.Tests
RED on current
master(d656fd81):test_send_request_includes_cookies_set_during_navigationfailed becausesend_requestthroughImpitHttpClientechoed{}after navigation setbasic=1.GREEN: that test plus the existing cookie /
send_requestcoverage — 7 passed. The production change is one helper and two call sites in_playwright_crawler.py.