Skip to content

fix(crawlers): Send cookies set during navigation with send_request in PlaywrightCrawler - #2485

Open
rome-xi wants to merge 2 commits into
apify:masterfrom
rome-xi:fix/2483-playwright-session-cookies-before-handler
Open

rome-xi wants to merge 2 commits into
apify:masterfrom
rome-xi:fix/2483-playwright-session-cookies-before-handler

Conversation

@rome-xi

@rome-xi rome-xi commented Nov 19, 2026

Copy link
Copy Markdown

Problem

PlaywrightCrawler only copied browser cookies into context.session.cookies after the request handler returned. Custom HTTP clients (ImpitHttpClient, HttpxHttpClient, and similar) read the session jar, so context.send_request(...) inside the handler sent the pre-navigation cookies. The default PlaywrightHttpClient was unaffected because it reads live browser cookies.

Fixes #2483.

Change

Copy browser cookies into the session after navigation / before the handler, and keep the existing post-handler copy so cookies set during the handler still persist. PlaywrightHttpClient is unchanged.

Tests

RED on current master (d656fd81): test_send_request_includes_cookies_set_during_navigation failed because send_request through ImpitHttpClient echoed {} after navigation set basic=1.

GREEN: that test plus the existing cookie / send_request coverage — 7 passed. The production change is one helper and two call sites in _playwright_crawler.py.

PlaywrightCrawler copied the browser cookie jar into the session only
after the request handler returned. ImpitHttpClient, HttpxHttpClient,
and CurlImpersonateHttpClient send cookies from that jar, so
send_request inside the handler still used the pre-navigation cookies.

Copy the jar before the handler as well. The copy after the handler
stays, so cookies created while the handler runs still persist.
PlaywrightHttpClient is unchanged and keeps reading the live browser
cookies.

Fixes apify#2483
@github-actions github-actions Bot added the t-tooling Issues with this label are in the ownership of the tooling team. label Nov 19, 2026
@vdusek vdusek changed the title fix(playwright): sync browser cookies to the session before the handler fix(crawlers): Send cookies set during navigation with send_request in PlaywrightCrawler Nov 20, 2026
Comment on lines +566 to +567
# Session-backed HTTP clients read this jar from inside the handler. Copy again after the
# handler returns so cookies created while it runs stay on the session.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: "this jar" doesn't point to anything on the next line, and CONTRIBUTING.md asks to wrap comments to the full 120-character width. Suggestion:

Suggested change
# Session-backed HTTP clients read this jar from inside the handler. Copy again after the
# handler returns so cookies created while it runs stay on the session.
# Session-backed HTTP clients read the session cookie jar inside the handler. Copy again after the handler returns
# so cookies created while it runs stay on the session.

✍️ Drafted by Claude Code

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in d3ec31a — used your wording and wrapped the comment to 120 characters.


await self._persist_browser_cookies(context)

async def _persist_browser_cookies(self, context: TPostNavContext) -> None:

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: In crawlee, "persist" usually means saving to a KVS. This helper only copies into the in-memory session, as its docstring says. Maybe rename it to _copy_browser_cookies_to_session?

✍️ Drafted by Claude Code

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed — renamed to _copy_browser_cookies_to_session in d3ec31a so it is not confused with persisting to a KVS.

async def _create_crawling_context(self, context: TPostNavContext) -> AsyncGenerator[TCrawlingContext, None]:
# Session-backed HTTP clients read this jar from inside the handler. Copy again after the
# handler returns so cookies created while it runs stay on the session.
await self._persist_browser_cookies(context)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note: This copy runs after the post-navigation hooks, so send_request in a post-navigation hook still sends the cookies from before navigation. crawlee JS behaves the same and #2483 is only about the handler, so it's fine to leave as is. Moving the copy to the end of _navigate would cover the hooks too.

✍️ Drafted by Claude Code

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, I will leave the copy where it is. #2483 is about the handler, and this matches the JS crawler's timing. Happy to move it to the end of _navigate in a follow-up if you want the hooks covered too.


yield self._build_crawling_context(context)

await self._persist_browser_cookies(context)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note: Older than this PR, not a blocker. The copy goes one way only, so if send_request stores token=B in the session during the handler and the browser still has token=A, this copy writes token=A back over it. Could be a follow-up issue.

✍️ Drafted by Claude Code

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed — that one-way overwrite is older than this PR and not needed to fix #2483. I can open a follow-up issue for a merge (or a session-wins rule) if you would like one.

@vdusek
vdusek requested a review from apify-factory November 20, 2026 08:56

@apify-factory apify-factory left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, nothing blocking.

🏭 Factory review — ℹ️ not auto-approved

This change was reviewed but cannot be auto-approved: a gate failed, so the review cannot end in an approval.

check result
Static gates ❌ 14/16 — author-allowed, same-repo
Review findings ✅ none
Reviewer escalation ❌ Changes when browser cookies are synced into the session jar, which touches session handling in a public crawler.
Juror 1 — claude-sonnet-5-5, adversarial ⏭️ skipped — a gate failed
Why it was not auto-approved

Failed gates:

  • author-allowed: rome-xi: not a member of apify/product-engineering
  • same-repo: head repo is rome-xi/crawlee-python

The comments above are the review itself; this section only explains the approval decision. Push a change and request @apify-factory again to re-run it, or ask a human to review.

Reviewed 854e10cc5 · Review details on the dashboard · workflow run

Address review nits on apify#2485: wrap the session-cookie comment to 120
characters, and rename _persist_browser_cookies to
_copy_browser_cookies_to_session so it is not confused with KVS persist.
@rome-xi

rome-xi commented Nov 20, 2026

Copy link
Copy Markdown
Author

Thanks @vdusek — addressed the nits in d3ec31a:

  • Wrapped the session-cookie comment to 120 characters and pointed at the session cookie jar explicitly.
  • Renamed _persist_browser_cookies to _copy_browser_cookies_to_session.

I left the copy after the post-navigation hooks (same as the JS crawler, and enough for #2483) and did not change the pre-existing one-way overwrite. Happy to follow up on either of those if you want.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

t-tooling Issues with this label are in the ownership of the tooling team.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

PlaywrightCrawler: context.send_request with a custom http_client sends pre-navigation cookies

3 participants