How to Scrape Sites Protected by Cloudflare

Which defense layers are scoring your request

How to implement this with residential proxies

1. 1

   Probe which Cloudflare layer is blocking you

   Compare responses from curl, curl_cffi with impersonate='chrome', and Playwright through the same proxy. If plain curl 403s but curl_cffi returns 200, TLS is your bottleneck. If both HTTP clients fail but Playwright passes, you need JS challenge clearance. Log cf-ray, server: cloudflare, and Set-Cookie headers for cf_clearance.

   Field note: Save the HTML of a failed response. A body containing 'Checking your browser' or 'cf-challenge' confirms JS layer; instant 403 with no challenge HTML points to TLS or IP reputation.

2. 2

   Impersonate Chrome at the TLS and HTTP/2 layers (curl_cffi)

   For sites where Managed Challenge is off or lenient, curl_cffi's impersonate parameter aligns JA3 and HTTP/2 with Chrome — often enough to collect JSON or static HTML without a browser. Route through Elite residential for reputation.

   Python (curl_cffi)Copy

   from curl_cffi import requests as cffi
   
   PROXY = "http://aethyn-XXXXX-country-us:PASSWORD@proxy.aethyn.io:5499"
   session = cffi.Session(impersonate="chrome124")
   session.proxies = {"http": PROXY, "https": PROXY}
   
   r = session.get("https://protected-shop.example/api/products", timeout=30)
   print(r.status_code, r.headers.get("cf-ray"))

   Field note: Pin impersonate to a recent Chrome version and update when curl_cffi adds newer profiles — stale impersonation profiles drift from live Chrome JA3 over time.

3. 3

   Clear JS challenges with Playwright and sticky sessions

   When cf_clearance is required, launch Playwright through a sticky Elite session so the cookie remains valid on the same IP. Wait for networkidle or a known post-challenge selector before reading cookies. Reuse the cookie jar for subsequent requests in that session.

   Python (Playwright)Copy

   from playwright.sync_api import sync_playwright
   
   with sync_playwright() as p:
       browser = p.chromium.launch(proxy={
           "server": "http://proxy.aethyn.io:5499",
           "username": "aethyn-XXXXX-country-us-session-cf1-lifetime-15",
           "password": "PASSWORD",
       })
       ctx = browser.new_context(locale="en-US")
       page = ctx.new_page()
       page.goto("https://protected-shop.example/", wait_until="networkidle")
       cookies = ctx.cookies()
       clearance = [c for c in cookies if c["name"] == "cf_clearance"]
       print("cleared:", bool(clearance))
       browser.close()

   Field note: Do not rotate IP between challenge solve and data fetch — cf_clearance is bound to the exit that earned it. Use the same -session- token for both steps.

4. 4

   Pace navigation to satisfy behavioral scoring

   After clearance, behave like a browsing session: load CSS, scroll, click a internal link or two, and add random delays between actions. Single-shot page.goto followed by immediate exit re-triggers challenge on the next visit.

   Field note: Record a HAR from a real Chrome session and compare resource load count and timing to your automation. Large gaps in subresource requests are a common behavioral tell.

5. 5

   Separate clearance workers from bulk extractors

   Architecturally, run a small pool of browser workers that earn cf_clearance and export cookies to a short-lived cache (Redis, 10–15 min TTL). Stateless extractors consume those cookies on the same sticky IP via curl_cffi. This keeps expensive browser minutes low while preserving layer alignment.

What goes wrong when this scraper fails

You pointed your scraper at a competitor's product catalog and every request returns a 403, a 503 'cf-ray' page, or a 200 with a five-second challenge spinner that never resolves in requests. Swapping datacenter proxies does nothing because Cloudflare never trusted those ASNs to begin with. The frustrating part is intermittent success: one URL works, the next does not, because clearance cookies, IP reputation, and fingerprint must stay aligned for the whole session.

Why this failure mode happens

Cloudflare Bot Management aggregates signals across the five layers into a bot score. Datacenter IPs start with a penalty. Non-browser TLS fingerprints add another. Missing or stale cf_clearance after a JS challenge means subsequent API calls fail even though the first page looked fine. Behavioral gaps — instant navigation, no resource loads, perfect timing — push borderline sessions back into challenge. The system is designed so naive automation spends budget on challenges instead of data.

How Aethyn residential proxies help here

Cloudflare-heavy targets need high-trust exits and stable sessions when a cf_clearance cookie is earned. Elite tier plus sticky sessions covers both without separate infrastructure per layer.

• Elite residential pool (port 5499) with reputation suited to Bot Management–protected sites
• Sticky sessions (-session-…-lifetime-) so clearance cookies survive follow-up API calls on the same IP
• Country targeting to match the locale Cloudflare expects for geo-sensitive rules
• Per-request rotation for uncached public pages that do not need clearance continuity
• Single endpoint integrates with both curl_cffi and Playwright proxy config

Common questions about this scraper problem

Best practices that keep scrapers reliable

• Diagnose layer-by-layer before buying more proxies
• Use curl_cffi when TLS is the only gap; reserve Playwright for JS challenges
• Keep IP sticky while cf_clearance is valid
• Pace navigation with subresource loads and human delays
• Cache clearance cookies with TTL aligned to Cloudflare expiry
• Stay within robots.txt and the site's Terms of Service

Common mistakes that burn proxy budget

• Rotating IP immediately after earning cf_clearance
• Using datacenter proxies on Bot Management sites
• Spoofing User-Agent without TLS impersonation
• Single page.goto with no subresources — instant behavioral flag
• Retrying challenge pages faster instead of slowing down
• Assuming one solver integration works forever without monitoring