Identify Which Anti-Bot System Is Blocking Your Scraper

Signal observed → what it means for your scraper

Match the signal you captured, then decide whether a residential proxy can change that layer before you rotate.

How to diagnose and fix this scraper failure

1. 1

   Capture status, Server, Set-Cookie, and the first 500 bytes of body

   Before rotating anything, log the full response for one failing URL: status code, Server header, every Set-Cookie name, custom x-* headers, and a body prefix. That tuple is what the matrix below matches. Do not discard 200s — soft blocks live there.

   Field note: Save raw headers to a file. Memory summaries drop Mime-Version, x-datadome, and access-control-expose-headers — the exact tells you need.

2. 2

   Run the three-way diagnostic recipe

   Plain curl, curl with a current Chrome User-Agent, and curl through a residential proxy must all hit the same URL. Two-way tests misattribute UA-triggered blocks to IP reputation. Record all three outcomes before changing your scraper framework.

   bash (three-way)Copy

   # 1) plain
   curl -sS -D - -o /tmp/body1.html "https://TARGET/" -w "\nHTTP %{http_code}\n" | head -n 40
   
   # 2) Chrome UA only
   curl -sS -D - -o /tmp/body2.html "https://TARGET/" \
     -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36" \
     -w "\nHTTP %{http_code}\n" | head -n 40
   
   # 3) residential proxy (Premium HTTP 2099)
   curl -sS -D - -o /tmp/body3.html "https://TARGET/" \
     -x "http://aethyn-XXXXX-country-us:PASSWORD@proxy.aethyn.io:2099" \
     -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36" \
     -w "\nHTTP %{http_code}\n" | head -n 40

   Field note: If (1) and (2) fail identically but (3) succeeds, IP reputation is implicated. If all three fail the same way, look at TLS/JS — not the proxy pool size.

3. 3

   Match signals to a vendor, then follow the deep page

   Use the playbook table on this page: Cloudflare (cf-ray), Akamai (AkamaiGHost + Reference #), DataDome (x-datadome), Kasada (x-kpsdk-*), Imperva (X-Iinfo / visid_incap_), PerimeterX (_px3 / px-captcha meta), Shape (response silence + endpoint asymmetry). Then open the vendor guide — do not invent a bypass from a generic 403 article.

4. 4

   Assert content, not only status, on every accepted response

   Treat HTTP 200 with an Access Denied title, challenge shell, or missing price node as a failure. lowes.com has been observed returning 200 with an Access Denied title — status-code-only monitors green-light poison.

   Python (content assert)Copy

   def accept(resp, min_len=2000, must_have=("price",)):
       if resp.status_code != 200:
           return False
       text = resp.text or ""
       title = text[text.find("<title>"): text.find("</title>") + 8].lower()
       if "access denied" in title or "just a moment" in title or "px-captcha" in text.lower():
           return False
       if len(text) < min_len:
           return False
       return all(tok in text.lower() for tok in must_have)

5. 5

   Only then rotate residential IPs — and only for IP-shaped rows

   When the matrix says the failure is ASN reputation, velocity, or geo, switch to Premium or Elite residential with country targeting. When it says TLS, JS challenge, or sensor token, keep the same IP and fix the client (curl_cffi / Playwright). Publishing what a proxy cannot fix is part of a correct diagnosis.

What goes wrong when this scraper fails

Your scraper failed and the first instinct is to rotate proxies. That is wrong when the signal is a TLS mismatch, a JavaScript challenge, or a vendor that returns HTTP 200 with a block body. The useful first question is which anti-bot (or CDN WAF) produced the response — because each one exposes different artifacts and each one treats IP reputation as a different weight. This page is the cross-vendor entry point: match what you observed, then follow the deep guide for that vendor.

When residential proxies fix this — and when they cannot

Residential proxies change the IP-shaped layers: datacenter ASN penalties, shared-IP reputation, per-IP velocity, and geo gates. They do not execute JavaScript, forge TLS/JA3, or mint vendor sensor tokens. Use them after you know which layer failed — otherwise you will rotate forever on a client fingerprint the edge never tied to the IP.

How Aethyn residential proxies help here

Once the signal matrix points to an IP-shaped failure, Aethyn gives you country/city/ISP targeting and sticky or rotating residential exits on published ports — so the three-way diagnostic and the fix use the same credential format.

• Premium HTTP 2099 / SOCKS5 1099 and Elite HTTP 5499 / SOCKS5 3499 — dedicated ports, not one shared endpoint
• Username suffixes for country, city, ISP, and sticky sessions (1–1440 min) documented at /docs/targeting
• Self-serve credentials so you can A/B plain curl vs residential without a sales call
• Elite tier when Bot Management–class targets penalize commodity residential reputation
• No claim that residential alone defeats a named vendor — pair with curl_cffi or Playwright when the matrix says fingerprint/JS

Best practices that keep scrapers reliable

• Identify before you rotate — one logged response beats ten blind retries
• Always run the three-way curl matrix on a new target
• Treat cookie presence as deployment evidence, not clearance
• Disambiguate CDN-in-front from the bot vendor behind it
• Keep public-data collection and ToS limits in scope — no CAPTCHA-solving recipes here
• Prefer Elite residential when Bot Management–class edges penalize commodity exits

Common mistakes that burn proxy budget

• Blaming the proxy for Akamai/DataDome fingerprint failures
• Calling every 429 a rate limit (Kasada challenges often use 429)
• Treating AKA_A2 as a bot cookie
• Assuming Reference #9.x is an access denial (malformed URL)
• Identifying Shape by TS* or reese84 cookies (wrong products)
• Accepting HTTP 200 without body assertions

Common questions about this scraper problem