How to Fix 403 and 429 Errors When Scraping

Signal observed → what it means for your scraper

Match the signal you captured, then decide whether a residential proxy can change that layer before you rotate.

How to diagnose and fix this scraper failure

1. 1

   Classify the error: proxy layer vs origin layer

   Before rotating anything, determine whether the response came from proxy.aethyn.io or the target host. A 407 or 502 with a proxy error body is infrastructure; a 403/429 with the target's HTML (or WAF challenge page) is the site. Log response.url, status, and the first 200 bytes of body on every failure — that tuple is your diagnosis key.

   Python (classify)Copy

   def classify_error(resp, proxy_host="proxy.aethyn.io"):
       if resp.status_code == 407:
           return "proxy_auth"
       if resp.status_code == 502 and proxy_host in (resp.url or ""):
           return "proxy_upstream"
       if resp.status_code == 429:
           return "rate_limit"
       if resp.status_code == 403:
           return "forbidden"
     return "other"

   Field note: If your HTTP client swallows the proxy URL in logs, hit httpbin.org/ip through the proxy first. A 407 there confirms credentials before you burn time debugging the target.

2. 2

   Fix 407 and 502 before touching rotation strategy

   407 means the gateway rejected auth — re-copy credentials from the dashboard and verify USER:PASS encoding. 502 means the chosen exit failed upstream — retry with a new IP (omit sticky session) and a sane timeout. Do not lower concurrency on 407; it will never help.

   cURLCopy

   curl -x "http://aethyn-XXXXX-country-us:PASSWORD@proxy.aethyn.io:2099" \
     -w "\nHTTP %{http_code}\n" https://httpbin.org/ip

   Field note: Special characters in passwords must be URL-encoded in the proxy URL. A 407 that appears only for certain country suffixes often means an illegal character in the username string.

3. 3

   Respond to 429 with backoff plus fresh IPs

   On 429, sleep with exponential backoff and jitter, then retry through a rotated residential IP. Lower global concurrency if 429s appear across many fresh exits simultaneously — that pattern means your aggregate velocity is too high, not that individual IPs are burned.

   Python (429 handler)Copy

   import time, random, requests
   
   PROXY = "http://aethyn-XXXXX:PASSWORD@proxy.aethyn.io:2099"
   proxies = {"http": PROXY, "https": PROXY}
   
   def fetch(url, attempt=0):
       r = requests.get(url, proxies=proxies, timeout=30)
       if r.status_code == 429:
           time.sleep(min(60, 2 ** attempt) + random.random())
           return fetch(url, attempt + 1)
       return r

   Field note: Respect Retry-After when the header is present — some APIs encode the exact cooldown. Ignoring it and retrying at T+1s is how polite throttles become hard blocks.

4. 4

   Address 403 with fingerprint and geo alignment

   When 403 persists on fresh residential IPs, the block is likely fingerprint or geography, not reputation alone. Align Accept-Language with -country-XX, send a full browser header set, and for Cloudflare-class targets use curl_cffi or Playwright so JA3 matches Chrome.

   Python (curl_cffi)Copy

   from curl_cffi import requests as cffi
   
   PROXY = "http://aethyn-XXXXX-country-de:PASSWORD@proxy.aethyn.io:2099"
   r = cffi.get(
       "https://example.com",
       impersonate="chrome",
       proxies={"http": PROXY, "https": PROXY},
       timeout=30,
   )
   print(r.status_code)

   Field note: A 403 on the homepage but 200 on /robots.txt often means geo-fencing, not a full IP ban. Test a neutral path before declaring the exit dead.

5. 5

   Detect soft blocks and alert on error-rate SLOs

   Not every block returns 4xx. Treat 200 responses missing expected selectors or shorter than baseline as soft blocks — log them alongside hard errors. Set an error-rate SLO (e.g. <2% 403/429 per hour) and page when it breaches so you catch drift before a week of bad data lands in production.

What goes wrong when this scraper fails

Your scraper was humming along at 200 OK and then, without warning, every worker starts returning 403 Forbidden or 429 Too Many Requests. Teams often respond by doubling concurrency or swapping one datacenter IP for another — which makes things worse. The status code is the site's way of telling you which detection layer fired, and each code wants a different response. Treating them all as 'get a new proxy' wastes time when the real issue is credentials (407), upstream routing (502), or a fingerprint that no IP can fix.

When residential proxies fix this — and when they cannot

Residential proxies address the two causes behind most 403/429 pairs: low-trust IPs and concentrated velocity. Per-request rotation spreads load so each exit stays near human-scale request rates, which is what 429 is asking for. When a 403 is reputation-driven, a fresh residential exit often clears it immediately — unlike rotating within the same datacenter /24. For 502s, rotation simply selects a healthier exit path without you touching application code.

How Aethyn residential proxies help here

Most scraping error playbooks come down to credentials, rotation, and tier. Aethyn controls all three through the proxy username on one endpoint — no separate gateway per country.

• Premium residential pool (port 2099) sized for high-volume crawls where 429 is the main risk
• Per-request rotation by default so 429 back-off actually lowers per-IP velocity
• Explicit country and session tokens in the username for geo-correct retries after 403
• Elite tier upgrade path when 403 persists on Premium due to high-security targets
• Standard http://USER:PASS@proxy.aethyn.io:PORT format — easy to validate when debugging 407

Best practices that keep scrapers reliable

• Log status, proxy country, URL, and body fingerprint together on every failure
• Fix 407/502 on the proxy path before tuning anti-bot strategy
• Backoff with jitter on 429; never immediate retry on the same IP
• Align geo headers with -country-XX when debugging 403
• Detect soft blocks (200 with missing content) as first-class errors
• Track error-rate SLOs and alert before datasets corrupt
• Deduplicate URLs upstream and cache unchanged pages — a 429 often means you are re-hitting the same endpoint
• Segment workers by target difficulty; don't send a hardened marketplace and an open blog through the same pool tier
• Inspect the Server and X-Cache response headers to tell a proxy-edge failure from a target CDN block (e.g. Cloudflare) before you start rotating

Common mistakes that burn proxy budget

• Retrying 429 instantly and escalating to permanent 403
• Rotating IPs when the password is wrong (407)
• Treating all 403s as 'bad IP' when fingerprint is the culprit
• Ignoring Retry-After headers on rate-limited APIs
• Monitoring status codes only and missing soft-blocked 200s
• Scaling concurrency up when 429 appears — the opposite of what the site asked for

Common questions about this scraper problem