Not urgent. Filing this so the research does not get lost.
Today
The residential proxy exit country is set explicitly or not at all:
ScrapeRequest.country / CrawlRequest.country (ISO 3166-1 alpha-2)Config::effective_proxy_credentials() (crw-core/src/config.rs:1114) resolves
request country -> renderer.proxy_default_country -> no suffix (global pool),
and composes the {user}__cr.{cc} credential.- The resolved value travels as the
REQUEST_COUNTRY task-local, set in exactly
two places (crw-crawl/src/single.rs:71, crw-crawl/src/crawl.rs:102) and read
by the CDP tier, the cloak path, and the config helper.
- If the requested country has no working exit, the CONNECT tunnel fails and
crw-renderer/src/cdp.rs retries once with the tier default. That safety net
already works.
So a caller who does not pass country gets a global-pool exit regardless of what
the target is. A request for a Swedish site can egress from anywhere.
The gap
Two separate failure modes:
- Geo-restricted targets. A site that only serves one country returns 403 /
451 / an empty shell from the wrong exit. We currently have no way to recover
unless the caller already knew to pass country.
- Locale inconsistency.
Accept-Language is hardcoded to en-US,en;q=0.9
(crw-renderer/src/http_only.rs:441) and there is no timezone emulation
anywhere in the workspace. When a country is selected, we send a German exit
IP with a US locale and a UTC clock. Detection vendors treat an IP/locale
mismatch as a stronger bot signal than a flagged datacenter IP, so today's
explicit country path is actively self-harming.
Research summary
Is there a ccTLD -> country dataset? Effectively yes and it needs no download.
ccTLDs are derived from ISO 3166-1 alpha-2 under RFC 1591, so the mapping is the
identity function on the last label. Hand-written exceptions are small: .uk maps
to GB, .eu is regional and not a country, .ac and .su are historical,
.tp / .an / .yu are retired.
But the inference is weak in both directions.
- False positives: the generic ccTLDs. Google's published set is
.ad .ai .as .bz .cc .cd .co .dj .fm .io .la .me .ms .nu .sc .sr .su .tv .tk .ws. Without an
exclusion list we would pin github.io to the British Indian Ocean Territory,
which has no residential exits at all.
- False negatives: most geo-locked commerce sits on a
.com with a /de/ path.
The TLD says nothing there.
So the TLD is a prior, not proof.
What the market does. ScraperAPI, ScrapingBee, Scrapfly, Scrape.do,
ScrapingDog and Bright Data all require a manual country parameter; none infers it
from the target. Zyte API is the exception and does not parse TLDs either: it
detects bans and learns a working exit location per target. That is the shape
worth copying.
How we would actually detect a geo-block
Signature matching only covers the loud cases:
- HTTP 451, unambiguous on its own.
- Cloudflare Error 1009 ("Access denied: country or region banned"), served as
403 with the literal error number in the body. Cloudflare's numeric codes are
stable, so this is the cleanest signal we can key on. Distinct from 1010
(browser signature).
- CloudFront 403 carrying "configured to block access from your country".
americastire.com is a live example already noted in our own docs.
Free-text phrases in arbitrary languages are not worth matching.
Everything else is silent: an empty page, a redirect to a country picker, or a
bare 403 indistinguishable from a bot block. For those the answer is not to
classify but to run the experiment: on failure, if we hold a country guess, retry
once from it. Success proves the site was geo-locked and proves which country
works. Persist that verdict per domain so the second request goes straight there.
Country guesses, cheapest first:
- ccTLD of the target host.
- The language of the response we did get. Even a block page usually carries
<html lang="fr">, Content-Language: fr-FR, or hreflang. A region-qualified
tag names the country outright. This covers a good part of the .com gap.
- The redirect target, when a
.com bounces to example.fr or /fr/.
If none of the three yields a guess, do nothing. Blind country sweeps burn latency
and bandwidth for an unknown return.
Proposed phases
Phase 0 - locale alignment. Derive Accept-Language from the selected country
and add Emulation.setTimezoneOverride on the CDP path. Independent of everything
below and fixes a defect that already exists.
Phase 1 - TLD inference, behind a config flag, default off. Applies only when
the request omits country. Last-label lookup, .uk -> gb, generic-ccTLD
exclusion list. Single insertion point at the two REQUEST_COUNTRY.scope sites.
Phase 2 - reactive retry. Add a geo-block signal to
crw-extract/src/antibot.rs classify() (currently classify(status, html), so
header-derived signals would need the signature widened). On a geo-ish failure,
retry once with the best available guess, capped at one attempt per domain.
Phase 3 - per-domain memory. Cache the country that worked.
crw-renderer/src/preference.rs already implements the moka + sliding-window
pattern this needs. Deliberately deferred until Phase 1 and 2 are measured.
Before writing any code
Two things gate this.
Measure the ceiling. Every scrape is already persisted server-side as JSONL.
Filter the 403/451 responses, then break them down by target ccTLD and by
1009 / CloudFront signature. If a low single-digit share of blocks carries a
non-US ccTLD or a geo signature, the whole feature is not worth building. That
number decides the scope.
Guard the benchmark. Pinning an exit country shrinks the available pool. A
.de request that succeeds today from the global pool could start failing from a
small, dirtier German pool. Scrape success must clear the existing gate before
this merges, since a merge to main reaches production within the hour.
Non-goals
- Blind multi-country retry sweeps.
- Any change to the explicit
country parameter's behaviour or precedence.
- IP geolocation of the target host. CDN edges make it meaningless.
Not urgent. Filing this so the research does not get lost.
Today
The residential proxy exit country is set explicitly or not at all:
ScrapeRequest.country/CrawlRequest.country(ISO 3166-1 alpha-2)Config::effective_proxy_credentials()(crw-core/src/config.rs:1114) resolvesrequest country ->
renderer.proxy_default_country-> no suffix (global pool),and composes the
{user}__cr.{cc}credential.REQUEST_COUNTRYtask-local, set in exactlytwo places (
crw-crawl/src/single.rs:71,crw-crawl/src/crawl.rs:102) and readby the CDP tier, the cloak path, and the config helper.
crw-renderer/src/cdp.rsretries once with the tier default. That safety netalready works.
So a caller who does not pass
countrygets a global-pool exit regardless of whatthe target is. A request for a Swedish site can egress from anywhere.
The gap
Two separate failure modes:
451 / an empty shell from the wrong exit. We currently have no way to recover
unless the caller already knew to pass
country.Accept-Languageis hardcoded toen-US,en;q=0.9(
crw-renderer/src/http_only.rs:441) and there is no timezone emulationanywhere in the workspace. When a country is selected, we send a German exit
IP with a US locale and a UTC clock. Detection vendors treat an IP/locale
mismatch as a stronger bot signal than a flagged datacenter IP, so today's
explicit
countrypath is actively self-harming.Research summary
Is there a ccTLD -> country dataset? Effectively yes and it needs no download.
ccTLDs are derived from ISO 3166-1 alpha-2 under RFC 1591, so the mapping is the
identity function on the last label. Hand-written exceptions are small:
.ukmapsto
GB,.euis regional and not a country,.acand.suare historical,.tp/.an/.yuare retired.But the inference is weak in both directions.
.ad .ai .as .bz .cc .cd .co .dj .fm .io .la .me .ms .nu .sc .sr .su .tv .tk .ws. Without anexclusion list we would pin
github.ioto the British Indian Ocean Territory,which has no residential exits at all.
.comwith a/de/path.The TLD says nothing there.
So the TLD is a prior, not proof.
What the market does. ScraperAPI, ScrapingBee, Scrapfly, Scrape.do,
ScrapingDog and Bright Data all require a manual country parameter; none infers it
from the target. Zyte API is the exception and does not parse TLDs either: it
detects bans and learns a working exit location per target. That is the shape
worth copying.
How we would actually detect a geo-block
Signature matching only covers the loud cases:
403 with the literal error number in the body. Cloudflare's numeric codes are
stable, so this is the cleanest signal we can key on. Distinct from 1010
(browser signature).
americastire.comis a live example already noted in our own docs.Free-text phrases in arbitrary languages are not worth matching.
Everything else is silent: an empty page, a redirect to a country picker, or a
bare 403 indistinguishable from a bot block. For those the answer is not to
classify but to run the experiment: on failure, if we hold a country guess, retry
once from it. Success proves the site was geo-locked and proves which country
works. Persist that verdict per domain so the second request goes straight there.
Country guesses, cheapest first:
<html lang="fr">,Content-Language: fr-FR, or hreflang. A region-qualifiedtag names the country outright. This covers a good part of the
.comgap..combounces toexample.fror/fr/.If none of the three yields a guess, do nothing. Blind country sweeps burn latency
and bandwidth for an unknown return.
Proposed phases
Phase 0 - locale alignment. Derive
Accept-Languagefrom the selected countryand add
Emulation.setTimezoneOverrideon the CDP path. Independent of everythingbelow and fixes a defect that already exists.
Phase 1 - TLD inference, behind a config flag, default off. Applies only when
the request omits
country. Last-label lookup,.uk->gb, generic-ccTLDexclusion list. Single insertion point at the two
REQUEST_COUNTRY.scopesites.Phase 2 - reactive retry. Add a geo-block signal to
crw-extract/src/antibot.rsclassify()(currentlyclassify(status, html), soheader-derived signals would need the signature widened). On a geo-ish failure,
retry once with the best available guess, capped at one attempt per domain.
Phase 3 - per-domain memory. Cache the country that worked.
crw-renderer/src/preference.rsalready implements the moka + sliding-windowpattern this needs. Deliberately deferred until Phase 1 and 2 are measured.
Before writing any code
Two things gate this.
Measure the ceiling. Every scrape is already persisted server-side as JSONL.
Filter the 403/451 responses, then break them down by target ccTLD and by
1009 / CloudFront signature. If a low single-digit share of blocks carries a
non-US ccTLD or a geo signature, the whole feature is not worth building. That
number decides the scope.
Guard the benchmark. Pinning an exit country shrinks the available pool. A
.derequest that succeeds today from the global pool could start failing from asmall, dirtier German pool. Scrape success must clear the existing gate before
this merges, since a merge to
mainreaches production within the hour.Non-goals
countryparameter's behaviour or precedence.