HTTP 200 but the Body Is a Block Page

Signal observed → what it means for your scraper

Match the signal you captured, then decide whether a residential proxy can change that layer before you rotate.

How to diagnose and fix this scraper failure

1. 1

   Log status, Server, title, and byte length together

   For one failing URL, capture status, Server, the <title>/<TITLE> text, Content-Length or len(body), and a 300-byte prefix. Soft blocks announce themselves in that tuple long before your parser notices missing JSON keys.

2. 2

   Reproduce the known soft-block shapes (dated observations)

   On 2026-08-09 probes: www.lowes.com returned HTTP 200 with an Access Denied TITLE and entity-encoded Reference material; www.autotrader.com returned 200 from Server: AkamaiNetStorage with an unavailable title and /akamai-block/ assets. Re-run before you cite either in a ticket — live edges change.

   Field note: If lowes returns real HTML today, keep the assert logic anyway — the failure class is what matters.

3. 3

   Three-way probe: plain, Chrome UA, residential

   Compare the title and length across three transports. If only residential flips a deny title to a real catalog, IP was implicated. If all three share Access Denied or page unavailable, fix the client or accept the soft template.

   bashCopy

   UA='Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36'
   URL='https://TARGET/'
   PX='http://aethyn-XXXXX-country-us:PASSWORD@proxy.aethyn.io:2099'
   
   summarize() {
     local f="$1"
     python - "$f" <<'PY'
   import sys,re,html
   from pathlib import Path
   b=Path(sys.argv[1]).read_text(errors="ignore")
   t=re.search(r"<title[^>]*>(.*?)</title>", b, re.I|re.S)
   title=html.unescape(t.group(1)).strip() if t else "(no title)"
   print(f"len={len(b)} title={title[:80]!r}")
   print("access_denied", "access denied" in title.lower())
   print("akamai_block_path", "/akamai-block/" in b.lower())
   PY
   }
   
   curl -sS -D /tmp/h1 -o /tmp/b1 "$URL" -w "plain %{http_code}\n"
   summarize /tmp/b1
   curl -sS -D /tmp/h2 -o /tmp/b2 "$URL" -A "$UA" -w "ua %{http_code}\n"
   summarize /tmp/b2
   curl -sS -D /tmp/h3 -o /tmp/b3 "$URL" -x "$PX" -A "$UA" -w "res %{http_code}\n"
   summarize /tmp/b3
   grep -iE '^(HTTP/|server:)' /tmp/h1 /tmp/h2 /tmp/h3

4. 4

   Install a content gate before storage

   Reject Access Denied titles, px-captcha meta, Just a Moment, page unavailable, and below-baseline lengths. Unescape HTML before Reference matching.

   PythonCopy

   import html, re
   from bs4 import BeautifulSoup  # optional; regex enough for titles
   
   DENY_TITLE = re.compile(
       r"access denied|page unavailable|just a moment|attention required",
       re.I,
   )
   
   def content_ok(status: int, body: str, min_len: int = 2500) -> bool:
       if status != 200:
           return False
       text = html.unescape(body or "")
       if len(text) < min_len:
           return False
       m = re.search(r"<title[^>]*>(.*?)</title>", text, re.I | re.S)
       title = (m.group(1) if m else "").strip()
       if DENY_TITLE.search(title):
           return False
       if "/akamai-block/" in text.lower() or "px-captcha" in text.lower():
           return False
       return True

5. 5

   Only then decide if residential helps

   When the soft shell disappears on Elite/Premium residential but not on direct egress, keep country-targeted exits. When it does not, move to curl_cffi or Playwright and keep asserting content on every navigation response.

What goes wrong when this scraper fails

Collectors and uptime checks that key only on HTTP status will green-light poison. Retail and classified edges increasingly return deny or unavailable documents with 200 so naive clients keep writing rows. The failure mode is quiet: dashboards fill with Access Denied titles, empty product grids, or AkamaiNetStorage “unavailable” pages that never raise.

When residential proxies fix this — and when they cannot

Residential exits help when the soft shell is reputation- or geo-shaped and flips under a clean ASN. They do nothing when the same Access Denied title appears on plain curl, Chrome-UA curl, and residential curl alike — that pattern is fingerprint or always-on substitution. Use proxies to isolate the IP variable, not as a content validator.

How Aethyn residential proxies help here

Run the same URL through Premium or Elite residential while your assert layer rejects deny titles. Sticky sessions keep a passing client on one exit once content checks are green.

• Premium HTTP 2099 / SOCKS5 1099 for volume diagnostics
• Elite HTTP 5499 / SOCKS5 3499 when Bot Management–class soft shells punish commodity ASNs
• Country and ISP suffixes so geo matches the catalog you expect
• Documented username form for reproducible three-way tests
• No implication that a proxy turns a deny shell into HTML by itself

Best practices that keep scrapers reliable

• Pair every status check with title + length + selector asserts
• Unescape HTML before vendor string greps
• Date-stamp live example captures
• Keep three-way diagnostics in the runbook
• Stay in public-data / Terms-safe collection — no CAPTCHA solving

Common mistakes that burn proxy budget

• Trusting 200 alone
• Grepping Reference # without html.unescape
• Burning exits on an identical soft shell
• Skipping Server: AkamaiNetStorage as a tell
• Celebrating flaky real HTML without asserts

Common questions about this scraper problem