Handle DataDome Blocks in Python (Same 403, Different Outcomes)

Signal observed → what it means for your scraper

Match the signal you captured, then decide whether a residential proxy can change that layer before you rotate.

How to diagnose and fix this scraper failure

1. 1

   Detect DataDome from headers and cookies, not vibes

   Look for x-datadome: protected (literal value protected), Set-Cookie: datadome=…, and script references to js.datadome.co/tags.js. Optional headers such as x-datadome-cid may appear on challenges. Confirm before you assume Cloudflare or Akamai.

   Python (detect)Copy

   def is_datadome(resp) -> bool:
       h = {k.lower(): v for k, v in resp.headers.items()}
       if h.get("x-datadome", "").lower() == "protected":
           return True
       sc = h.get("set-cookie", "")
       if "datadome=" in sc.lower():
           return True
       body = resp.text or ""
       return "js.datadome.co/tags.js" in body or "captcha-delivery.com" in body

2. 2

   Classify HTML vs JSON 403 bodies safely

   Never treat a 403 JSON object with a single url key as business data. Extract the challenge URL and log query parameters for diagnosis. Values such as t=bv appear on some captcha URLs and correlate with hard IP pressure on that tenant — use them as hints, not as a guarantee that another IP will pass.

   Python (classify body)Copy

   import json
   from urllib.parse import urlparse, parse_qs
   
   def classify_datadome_body(resp):
       text = (resp.text or "").strip()
       if text.startswith("{"):
           try:
               data = json.loads(text)
           except json.JSONDecodeError:
               data = None
           if isinstance(data, dict) and set(data) == {"url"}:
               q = parse_qs(urlparse(data["url"]).query)
               return {"kind": "json_challenge", "t": (q.get("t") or [None])[0], "url": data["url"]}
       if "captcha-delivery.com" in text or "var dd=" in text or "Please enable JS" in text:
           return {"kind": "html_challenge_or_block"}
       return {"kind": "other", "status": resp.status_code}

   Field note: Interstitial vs captcha field sets differ — do not assume every HTML 403 embeds a full dd object with a t field.

3. 3

   Three-way curl before you rewrite the scraper

   Plain curl, Chrome-UA curl, and residential-proxied curl separate UA triggers from IP reputation. Record x-datadome and body class for each.

   bashCopy

   PROXY="http://aethyn-XXXXX-country-fr:PASSWORD@proxy.aethyn.io:5499"
   URL="https://TARGET/"
   
   for label in plain ua res; do
     case $label in
       plain) extra=();;
       ua) extra=(-A "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36");;
       res) extra=(-x "$PROXY" -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36");;
     esac
     echo "=== $label ==="
     curl -sS -D - -o "/tmp/dd-$label.body" "$URL" "${extra[@]}" -w "\nHTTP %{http_code}\n" | grep -iE "HTTP/|x-datadome|set-cookie: datadome|server:"
   done

4. 4

   Branch: rotate residential only for IP-shaped pressure

   When fresh residential exits change the outcome, keep rotation or sticky sessions in Python via the proxy URL. When they do not, switch to curl_cffi or Playwright through the same proxy instead of expanding concurrency.

   Python (requests + proxy)Copy

   import requests
   
   PROXY = "http://aethyn-XXXXX-country-fr:PASSWORD@proxy.aethyn.io:5499"
   proxies = {"http": PROXY, "https": PROXY}
   headers = {
       "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36",
       "Accept": "text/html,application/xhtml+xml",
   }
   r = requests.get("https://TARGET/", headers=headers, proxies=proxies, timeout=45)
   print(r.status_code, r.headers.get("x-datadome"), is_datadome(r))

5. 5

   Assert success on content — DataDome can fail open

   Intermittent 200 responses can mean protection degraded or a custom allow path, not that your scraper is correct. Require expected selectors and reject challenge titles before writing rows.

What goes wrong when this scraper fails

Python scrapers talking to DataDome-protected classifieds, travel, and luxury retail properties often see a wall of HTTP 403s with almost no vendor logo. Because device check, CAPTCHA, and ban share that envelope, teams either rotate proxies forever or parse the JSON challenge as if it were API data. Correct handling starts by identifying DataDome explicitly, then reading the body.

When residential proxies fix this — and when they cannot

Residential proxies help when the failure is reputation or velocity shaped — including cases where challenge URLs show t=bv on a given tenant. They also spread request rate so you trip fewer rate/timebox policies. They do not run the DataDome JS tag or fix a Python TLS fingerprint that the edge dislikes. If every clean exit gets the same challenge shell, stop rotating and fix the client.

How Aethyn residential proxies help here

Use Aethyn to control the IP variable deliberately while your Python code classifies DataDome responses. Sticky Elite sessions support browser-backed flows; rotating Premium covers volume on softer paths.

• Elite HTTP 5499 for higher-trust exits on hard tenants
• Per-request rotation on Premium 2099 for velocity-sensitive catalogs
• Country targeting for geo-differentiated properties
• Documented username grammar for reproducible experiments
• Honest positioning: proxy ≠ DataDome bypass

Best practices that keep scrapers reliable

• Detect x-datadome before generic 403 handlers
• Reject single-key captcha-delivery JSON as data
• Run three-way diagnostics on each new tenant
• Prefer sticky Elite exits for browser-backed sessions
• Stay in public-data / ToS-safe scope — no CAPTCHA solve recipes

Common mistakes that burn proxy budget

• Ingesting challenge JSON as API payloads
• Forging or replaying datadome cookies across IPs
• Assuming tags.datadome.co is the script host
• Celebrating flaky 200s without content asserts
• Rotating forever on a pure client fingerprint failure

Common questions about this scraper problem