HTTP 200 Access Denied: Soft Blocks That Poison Scrapers

Your monitor is green. The status line says 200. The body says Access Denied — or “page unavailable” under Server: AkamaiNetStorage. Soft blocks are the quiet failure mode of modern scraping: they never trip raise_for_status(), they fill warehouses with junk, and they make “success rate” dashboards look healthy while the business data is gone.

Playbook with validators: HTTP 200 but the body is a block page.

Why edges serve soft denies

A readable deny document with a healthy status keeps browsers calm and dumb uptime checks quiet. Akamai-shaped properties may serve Access Denied HTML or NetStorage block assets without flipping the status line. Your framework reports success because transport succeeded, not because the catalog document arrived.

Related silence problem: Playwright did not tell you it was blocked.

What breaks in production pipelines

Soft blocks fail differently depending on where you look:

The last row is especially cruel: a colleague opens Chrome on a residential ISP and sees the real page, while your Python job stores Access Denied shells from a library TLS fingerprint. Without content asserts, the debate becomes “is the proxy bad?” when the real question is “did we define success?”

Verified shapes (re-check live)

Dated 2026-08-09 probes (edges change — re-run before tickets):

Same class as hard Reference #18 denials — different status cosmetics.

Grep trap: 'Reference #' in resp.text can be False until html.unescape().

Minimum content gate

Code Snippet
import html, re

DENY_TITLE = re.compile(
    r"access denied|page unavailable|just a moment|attention required",
    re.I,
)

def content_ok(status: int, body: str, min_len: int = 2500) -> bool:
    if status != 200:
        return False
    text = html.unescape(body or "")
    if len(text) < min_len:
        return False
    m = re.search(r"<title[^>]*>(.*?)</title>", text, re.I | re.S)
    title = (m.group(1) if m else "").strip()
    if DENY_TITLE.search(title):
        return False
    if "/akamai-block/" in text.lower() or "px-captcha" in text.lower():
        return False
    return True

Calibrate min_len per URL class. A 200 that fails the gate is a soft block, full stop — count it as failure in cost-per-success math.

Also log a structured tuple on every accepted write: status, Server, title, body length, and a 300-byte prefix. Soft blocks announce themselves in that tuple long before your JSON schema notices missing keys.

Fingerprint soft blocks vs reputation soft blocks

The lowes.com-shaped pattern (same machine, same IP: library client gets Access Denied HTML on 200; browser-grade TLS gets real HTML) is a fingerprint lesson. Rotating residential exits would not have changed the library outcome. Conversely, if only residential curl flips a deny title to a real catalog while plain and UA curls stay denied, the IP layer is implicated — keep Premium or Elite in the path and re-run asserts.

Avoid relying on unverifiable “success rates” from marketing pages. Measure content-verified accepts on your own basket.

Three-way probe: is IP the lever?

1. Plain curl — log status, Server, title, length
2. Chrome User-Agent curl
3. Same through residential

Aethyn Premium HTTP proxy.aethyn.io:2099 (SOCKS5 1099) or Elite HTTP 5499 / SOCKS5 3499:

Code Snippet
PX='http://aethyn-XXXXX-country-us:PASSWORD@proxy.aethyn.io:2099'
UA='Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36'
curl -sS -o /tmp/b3 -w "res %{http_code}\n" -x "$PX" -A "$UA" "https://TARGET/"

What not to do

• Alert only on non-2xx
• Store every 200 HTML blob
• Rotate Elite exits against an identical soft shell
• Skip Server: AkamaiNetStorage as a tell
• Grep for Reference # without html.unescape
• Treat a colleague’s browser screenshot as proof your library client is fine

After asserts fail, classify the vendor with Identify which anti-bot blocked you, then remediate once. Soft shells often sit next to hard Reference #18 denials on the same property — different status cosmetics, same edge family.

Full assert recipes and dated examples: HTTP 200 but the body is a block page.