Fix Akamai Access Denied When Scraping (Reference # Explained)

Signal observed → what it means for your scraper

Match the signal you captured, then decide whether a residential proxy can change that layer before you rotate.

How to diagnose and fix this scraper failure

1. 1

   Read the Reference # before touching proxies

   Akamai Access Denied pages commonly include Reference #18.<hex>.<unix-epoch>.<hex> (entity-encoded &#35; in HTML) or Reference 0.<hex>…. Segment 18 indicates access denied; the epoch segment is the decision second. Reference #9.x is a malformed request — fix the URL, do not rotate. Site operators can further translate Reference strings via Akamai Edge Diagnostics.

   Python (parse Reference #)Copy

   import re
   from datetime import datetime, timezone
   
   REF = re.compile(
       r"Reference\s*(?:#|&#35;)?(?P<code>\d+)\.(?P<a>[^.]+)\.(?P<epoch>\d+)\.(?P<b>\S+)",
       re.I,
   )
   
   def explain_reference(html: str) -> str:
       m = REF.search(html)
       if not m:
           return "no Reference # found"
       code = m.group("code")
       if code == "9":
           return "Reference #9 — malformed URL/request, not a bot ban"
       if code == "18":
           ts = datetime.fromtimestamp(int(m.group("epoch")), tz=timezone.utc)
           return f"Reference #18 access denied at {ts.isoformat()}"
       return f"Reference #{code} — inspect full string with site operator"

   Field note: Also accept the Reference 0.<hex> template — parsers that require a literal # will miss real denials.

2. 2

   Confirm AkamaiGHost + cookie evidence

   On many deny responses, Server: AkamaiGHost appears with Mime-Version: 1.0 and text/html. Bot Manager cookies listed on Akamai’s own preferences material include _abck, ak_bmsc, and bm_sv (Strictly Necessary) and bm_sz (Performance). Co-occurrence of _abck and ak_bmsc is a strong Bot Manager indicator. AKA_A2 alone is Adaptive Acceleration — not clearance.

3. 3

   Run the three-way curl recipe through residential

   Compare plain curl, Chrome-UA curl, and Chrome-UA curl via Elite or Premium residential. Fingerprint-driven denials often persist across clean IPs; reputation-driven denials flip when the exit ASN changes.

   bashCopy

   PROXY="http://aethyn-XXXXX-country-us-session-akamai1-lifetime-30:PASSWORD@proxy.aethyn.io:5499"
   
   curl -sS -D - -o /tmp/ak.html "https://TARGET/" -w "\nHTTP %{http_code}\n" | head -n 30
   curl -sS -D - -o /tmp/ak-ua.html "https://TARGET/" -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36" -w "\nHTTP %{http_code}\n" | head -n 30
   curl -sS -D - -o /tmp/ak-res.html "https://TARGET/" -x "$PROXY" -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36" -w "\nHTTP %{http_code}\n" | head -n 30
   grep -iE "Reference|AkamaiGHost|Mime-Version|set-cookie: _abck|set-cookie: ak_bmsc|AKA_A2" /tmp/ak*.html -n || true

4. 4

   If IP is not the lever, upgrade the client — not the pool myth

   Use curl_cffi or Playwright so TLS/HTTP behaviour matches a real browser. Keep sticky residential sessions when a browser-backed flow must reuse the same exit. Avoid relying on unverifiable success-rate claims — measure your own accept/assert rate on a fixed URL basket.

   Python (curl_cffi)Copy

   from curl_cffi import requests as cffi
   
   PROXY = "http://aethyn-XXXXX-country-us:PASSWORD@proxy.aethyn.io:5499"
   r = cffi.get(
       "https://TARGET/",
       impersonate="chrome",
       proxies={"http": PROXY, "https": PROXY},
       timeout=45,
   )
   print(r.status_code, r.headers.get("server"), (r.text or "")[:200])

   Field note: Community reports show the same IP succeeding in requests and failing in aiohttp against Akamai — treat client stack as a first-class variable.

5. 5

   Watch for tarpit and soft substitution

   If connects hang until your client times out, consider tarpit behaviour and raise timeouts while lowering concurrency. If you receive 200 with plausible but wrong catalog data, assert invariants (price nodes, result counts) before writing to storage.

What goes wrong when this scraper fails

Scrapers hitting Akamai-protected retail and media properties often see Access Denied with a Reference # string, or intermittent 200s that collapse on the first authenticated or search endpoint. Teams burn residential bandwidth rotating IPs while the edge is scoring HTTP/2 and TLS fingerprints — the layer Akamai pioneered and still operates at massive scale. The fix path starts by reading the receipt Akamai already printed on the deny page.

When residential proxies fix this — and when they cannot

Residential proxies address Akamai’s IP-shaped gate: hosting ASN classification, prior abuse reputation, per-IP velocity, and geo constraints. If a crawl works for a while then #18s until you change exit, IP was the lever. If plain curl and Chrome-UA curl fail identically through a clean residential exit, the decision is above IP — impersonate a real browser stack instead of buying more bandwidth.

How Aethyn residential proxies help here

Akamai-heavy targets need trustworthy exits and sticky sessions when you finally clear a browser-backed flow. Elite residential plus documented targeting keeps the IP layer honest while you fix the client.

• Elite HTTP 5499 / SOCKS5 3499 for higher-trust residential exits
• Sticky sessions via -session-…-lifetime-N for multi-step flows on one exit
• Country and city suffixes so Accept-Language and geo match the exit
• Premium 2099 for volume paths where Bot Manager is absent and velocity is the issue
• Same username grammar across tests — three-way diagnostics stay reproducible

Best practices that keep scrapers reliable

• Parse Reference # codes before rotating
• Log Server, Mime-Version, and Bot Manager cookie names together
• Keep three-way diagnostics in CI for Akamai-heavy targets
• Use sticky Elite exits only after the client stack is browser-grade
• Respect public-data scope and site Terms — diagnosis ≠ unauthorized access

Common mistakes that burn proxy budget

• Rotating proxies on Reference #9
• Treating _abck as proof of clearance
• Misreading AKA_A2 as Bot Manager
• Tight retry loops into a tarpit
• Copying cookies between different User-Agents or IPs

Common questions about this scraper problem