Monitoring Best Buy and Target Prices, Stock, and Promotions at Scale

What goes wrong when this scraper fails

Retail price intelligence teams need current public prices, promotional flags, and stock status for thousands of SKUs across Best Buy and Target, broken out by store and region because both retailers vary price and availability geographically. Best Buy fronts its site and internal availability calls with Akamai Bot Manager, which issues sensor-based challenges and sets short-lived _abck and bm_sz cookies that must be solved by real browser JavaScript. Target routes its product and pricing data through RedSky-style JSON APIs (redsky.target.com/redsky_aggregations/...) guarded by an Imperva-class WAF that fingerprints clients and gates responses on a visitor cookie and rotating api_key. The core difficulty is doing this from enough distinct residential vantage points to capture ZIP-level truth, without your traffic clustering onto a handful of IPs that get flagged in minutes.

Why this failure mode happens

Akamai and Imperva do not simply count requests - they score sessions on IP reputation (datacenter ASNs are effectively pre-flagged), TLS fingerprint (JA3/JA4), HTTP/2 frame ordering, header completeness and order, and behavioral velocity. A datacenter IP sending a header set that does not match a real Chrome build gets a challenge on request one, long before any rate limit matters. On top of that, both retailers deliberately localize pricing and stock to the store serving a given ZIP, so identical requests from different geographies legitimately return different data, and any anti-bot system reads a single IP fanning across hundreds of ZIPs as a strong automation signal.

Challenges that make this hard to automate

• Akamai Bot Manager on Best Buy sets _abck / bm_sz cookies via a JS sensor payload; a cold HTTP client with no browser context gets a 'pardon our interruption' page instead of product JSON.
• Target's RedSky endpoints demand a currently-valid api_key plus store_id and pricing_store_id, and reject requests missing the visitor_id / GuestID cookie minted by the anti-bot layer.
• Pricing and availability differ by store, so you must resolve the correct store_id or fulfillment ZIP for every location you monitor, not just hit a national URL.
• Soft blocks return HTTP 200 with a decoy body (empty items array, access-denied HTML), silently corrupting datasets if you only check status codes.
• Promotions surface inconsistently - as a strikethrough price, a 'Save $X' badge, a cart-only price, or a limited-time flag - so a single parser field rarely captures the true promotional state.

Approaches that usually fail

• Hitting the public product pages directly from a server's own datacenter IP - works for a handful of requests, then Akamai/Imperva challenge everything.
• A small pool of static datacenter proxies rotated round-robin - IPs share a bad ASN reputation and get burned together within minutes.
• Headless browser farms (Puppeteer/Playwright) without proxy geo-targeting - solves the JS challenge but returns one region's pricing for the whole country.
• Buying a third-party 'retail pricing API' - convenient but often stale, missing ZIP granularity, and opaque about where its numbers actually come from.

When residential proxies fix this — and when they cannot

Residential proxies place your requests on real consumer IPs with clean reputation, which is the single biggest factor in passing Akamai and Imperva reputation scoring - the same request that gets challenged from a datacenter ASN often sails through from a residential exit. Geo-targeting by country and city lets you deliberately source each request from the region whose store and ZIP pricing you actually want to capture, so the localized data you get back is correct rather than accidental. Per-request rotation spreads a large SKU crawl across thousands of distinct exits so no single IP accumulates the velocity that triggers a block, while short sticky sessions keep a warmed cookie jar on one exit when a multi-step flow (set ZIP, then read availability) must stay coherent.

How Aethyn residential proxies help here

Aethyn gives you a large residential pool with per-request rotation by default and precise country/city geo-targeting, which is exactly the shape of a retail price-intelligence workload where each request must originate from the region it is measuring. The Premium tier (port 2099) handles the bulk of Best Buy and Target reads cleanly, and when Akamai or Imperva start issuing challenges you can escalate the same code to the Elite pool (port 5499) for higher-trust exits.

• Per-request rotation on port 2099 by default, so a 5,000-SKU crawl lands on thousands of distinct residential IPs instead of clustering.
• Country and city geo-targeting (-country-us-city-chicago) to source each request from the exact market whose store pricing you need.
• Elite pool on port 5499 for the harder targets - higher-trust exits that get through Akamai/Imperva challenges the Premium pool starts to see.
• Sticky sessions up to 30 minutes (-session-TOKEN-lifetime-10) to hold one exit while you set a ZIP cookie and then read availability on the same IP.
• A single endpoint (proxy.aethyn.io) with a username-encoded config, so switching region, city, or session is a string change, not new infrastructure.
• Clean IP reputation across the pool, which is the dominant signal for passing the reputation scoring both Akamai and Imperva apply before any rate limit.

How to implement this with residential proxies

1. 1

   Route every request through a geo-targeted Premium residential exit

   Start with per-request rotation on the Premium pool (port 2099). Encode the target market directly in the proxy username - country plus city - so each request originates from the region whose store pricing you want. Do not reuse one IP across many ZIPs; that pattern is exactly what the anti-bot systems read as automation. Confirm your exit geography before you trust a single price.

   pythonCopy

   import requests
   
   # Premium tier -> port 2099. Per-request rotation is the default (no session token).
   PROXY = "http://aethyn-XXXXX-country-us-city-chicago:PASSWORD@proxy.aethyn.io:2099"
   proxies = {"http": PROXY, "https": PROXY}
   
   # Sanity-check the exit region before scraping localized prices.
   who = requests.get("https://ipinfo.io/json", proxies=proxies, timeout=15).json()
   print(who.get("city"), who.get("region"), who.get("country"))
   # Expect a US exit near Chicago; if it lands elsewhere your ZIP pricing will be wrong.

   Field note: Verify the exit's city/region on the first request of every session, not just once at startup. Residential exits drift, and a request that silently lands in the wrong metro will hand you correct-looking prices for the wrong store.

2. 2

   Send a complete, ordered browser header set - never a bare User-Agent

   Both Akamai and Imperva fingerprint your header set and its order against real Chrome builds. A lone User-Agent with no Accept-Language, sec-ch-ua, or sec-fetch-* headers is an instant tell. Mirror a real recent Chrome exactly, including the client-hint headers, and keep Accept-Language consistent with the country you are geo-targeting from.

   pythonCopy

   HEADERS = {
       "User-Agent": (
           "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
           "(KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36"
       ),
       "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8",
       "Accept-Language": "en-US,en;q=0.9",
       "sec-ch-ua": '"Chromium";v="126", "Google Chrome";v="126", "Not-A.Brand";v="24"',
       "sec-ch-ua-mobile": "?0",
       "sec-ch-ua-platform": '"Windows"',
       "sec-fetch-dest": "document",
       "sec-fetch-mode": "navigate",
       "sec-fetch-site": "none",
       "upgrade-insecure-requests": "1",
   }
   
   r = requests.get("https://www.bestbuy.com/", headers=HEADERS, proxies=proxies, timeout=20)
   print(r.status_code, len(r.text))

   Field note: Keep Accept-Language aligned with your proxy country. A US-geo exit sending Accept-Language: ru-RU is an internally inconsistent fingerprint that scoring engines flag even when every other signal looks clean.

3. 3

   Pull Target availability from the RedSky JSON API with the right store context

   Target's product and fulfillment data comes back as clean JSON from RedSky-style aggregation endpoints, but you must supply a valid api_key, the tcin (Target's SKU id), and store context (store_id, pricing_store_id, and a fulfillment ZIP). Resolve the correct store for each ZIP first, then request pricing and availability against that store. Hold one exit for the store-lookup-then-availability pair with a short sticky session so the cookie jar stays coherent.

   pythonCopy

   import requests
   
   # Short sticky session so the store lookup and availability read share one exit + cookie jar.
   STICKY = "http://aethyn-XXXXX-country-us-city-dallas-session-tgt42-lifetime-10:PASSWORD@proxy.aethyn.io:2099"
   proxies = {"http": STICKY, "https": STICKY}
   
   API = "https://redsky.target.com/redsky_aggregations/v1/web/pdp_fulfillment_v1"
   params = {
       "key": "REDSKY_API_KEY",          # visitor api_key observed from the site
       "tcin": "54191097",
       "store_id": "1234",
       "pricing_store_id": "1234",
       "zip": "75201",
       "state": "TX",
       "latitude": "32.7767",
       "longitude": "-96.7970",
       "has_store_positions_store_id": "true",
   }
   headers = {**HEADERS, "Accept": "application/json", "Origin": "https://www.target.com",
              "Referer": "https://www.target.com/"}
   
   resp = requests.get(API, params=params, headers=headers, proxies=proxies, timeout=20)
   data = resp.json()
   fulfillment = data["data"]["product"]["fulfillment"]
   print(fulfillment["store_options"][0]["location_available_to_promise_quantity"])

   Field note: Treat the api_key as perishable, not constant. Target rotates it and scopes it to a visitor context; scrape the current key from the page's bootstrap JSON at session start rather than hardcoding one, or you will get 401s the moment it rotates.

4. 4

   Handle Best Buy's Akamai challenge with a warmed browser context

   Best Buy's Akamai layer sets _abck and bm_sz cookies from a JS sensor payload that a cold requests.get cannot produce, so you typically need a real browser to mint them. Drive Playwright through the same Premium residential exit, let the page execute the sensor, then read product and availability. Route Playwright's traffic through the proxy so the browser and the challenge solve happen on the same clean IP.

   pythonCopy

   from playwright.sync_api import sync_playwright
   
   PROXY_SERVER = "http://proxy.aethyn.io:2099"
   PROXY_USER = "aethyn-XXXXX-country-us-city-seattle-session-bby7-lifetime-10"
   PROXY_PASS = "PASSWORD"
   
   with sync_playwright() as p:
       browser = p.chromium.launch(
           headless=True,
           proxy={"server": PROXY_SERVER, "username": PROXY_USER, "password": PROXY_PASS},
       )
       ctx = browser.new_context(locale="en-US", timezone_id="America/Los_Angeles")
       page = ctx.new_page()
       # Landing on a normal page first lets Akamai's sensor set _abck / bm_sz.
       page.goto("https://www.bestbuy.com/", wait_until="domcontentloaded", timeout=45000)
       page.wait_for_timeout(2500)
       page.goto("https://www.bestbuy.com/site/searchpage.jsp?st=laptop",
                 wait_until="domcontentloaded", timeout=45000)
       html = page.content()
       print("challenge" if "Pardon Our Interruption" in html else "ok", len(html))
       browser.close()

   Field note: Warm the cookie jar by loading the homepage before the product/search page. Akamai's _abck cookie only becomes valid after the sensor runs; jumping straight to a deep URL on a cold context reliably returns the interruption page.

5. 5

   Parse defensively and set the fulfillment ZIP before reading stock

   Best Buy resolves store availability from a selected store or ZIP, so set the location context first, then read the availability button state. Never assume a field exists - promos appear as strikethrough prices, 'Save $X' badges, or cart-only prices, and stock shows as 'Add to Cart', 'Sold Out', or 'Check Stores'. Extract each into an explicit, nullable field so a missing element becomes a null, not a crash.

   pythonCopy

   from bs4 import BeautifulSoup
   
   def parse_bestbuy_item(html: str) -> dict:
       soup = BeautifulSoup(html, "html.parser")
   
       def text_or_none(sel):
           el = soup.select_one(sel)
           return el.get_text(strip=True) if el else None
   
       current = text_or_none('[data-testid="customer-price"]')
       was = text_or_none(".pricing-price__regular-price")   # strikethrough when on promo
       save_badge = text_or_none(".pricing-price__savings")
       cta = text_or_none(".add-to-cart-button") or text_or_none(".fulfillment-add-to-cart-button")
   
       return {
           "price": current,
           "was_price": was,
           "promo": save_badge,                      # None when not discounted
           "on_promo": bool(was and save_badge),
           "stock_state": cta,                        # 'Add to Cart' / 'Sold Out' / 'Check Stores'
       }
   
   record = parse_bestbuy_item(html)
   assert record["price"] is not None, "soft block or layout change - do not store"
   print(record)

   Field note: Assert a non-empty price before writing to your database. A 200 OK with a null price on Best Buy is almost never a genuinely priceless product - it is a soft block or a layout change, and storing it quietly poisons your promo-detection logic for weeks.

6. 6

   Detect blocks and escalate from Premium to the Elite pool

   Distinguish real responses from soft blocks by inspecting the body, not just the status code - watch for 'Pardon Our Interruption', 'Access Denied', empty items arrays, or a suspiciously tiny payload. When challenge rate climbs on the Premium pool, retry the same request on the Elite pool (port 5499) for higher-trust exits before you back off. Back off with jitter so you are not hammering a target that is already flagging you.

   pythonCopy

   import random, time, requests
   
   BLOCK_MARKERS = ("Pardon Our Interruption", "Access Denied", "unusual traffic", "px-captcha")
   
   def looks_blocked(resp) -> bool:
       if resp.status_code in (403, 429):
           return True
       body = resp.text
       if any(m in body for m in BLOCK_MARKERS):
           return True
       return len(body) < 800   # decoy pages are tiny
   
   def fetch(url, headers, user_base):
       # Try Premium (2099); on block, escalate the SAME request to Elite (5499).
       for port in (2099, 5499):
           px = "http://" + user_base + ":PASSWORD@proxy.aethyn.io:" + str(port)
           try:
               r = requests.get(url, headers=headers, proxies={"http": px, "https": px}, timeout=20)
           except requests.RequestException:
               time.sleep(1.5 + random.random())
               continue
           if not looks_blocked(r):
               return r
           time.sleep(2 + random.random() * 3)   # jittered backoff before escalating
       return None
   
   resp = fetch("https://www.target.com/p/-/A-54191097",
                {**HEADERS, "Accept": "application/json"},
                "aethyn-XXXXX-country-us-city-dallas")
   print("blocked" if resp is None else resp.status_code)

   Field note: Escalate to Elite (5499) before you slow your whole crawl down. A rising challenge rate on Premium usually means the specific target is scrutinizing that pool right now - the higher-trust Elite exits often clear it immediately, so you only pay the latency cost on the requests that actually need it.

Best practices that keep scrapers reliable

• Collect only public, non-personal price and availability data - no account creation, no login, no checkout automation - and keep your request volume reasonable and within each retailer's Terms of Service.
• Geo-target every request to the market it is measuring; a price captured from the wrong region is worse than no data because it looks valid.
• Rotate per request by default (port 2099) and reserve sticky sessions for genuine multi-step flows like set-ZIP-then-read-stock.
• Assert on real fields (a non-null price, a non-empty items array) before persisting, so soft blocks never enter your dataset.
• Scrape perishable tokens - Target's api_key, Akamai's cookies - fresh at session start instead of hardcoding them.
• Add jittered backoff and Premium-to-Elite escalation so a flagged pool degrades gracefully instead of hammering a target that is already challenging you.
• Cache store_id / pricing_store_id lookups per ZIP so you resolve each location once rather than re-querying it on every SKU.

Common mistakes that burn proxy budget

• Checking only the HTTP status code and treating every 200 as success, so tiny decoy 'access denied' bodies get stored as real data.
• Scraping a national product URL from a single IP and assuming the price and stock apply everywhere, when both are store-specific.
• Sending a bare User-Agent with no Accept-Language or sec-ch-ua client hints, which Akamai and Imperva flag on the first request.
• Hardcoding Target's RedSky api_key and getting a wall of 401s the moment it rotates.
• Hitting Best Buy's deep product URLs with a cold HTTP client, skipping the homepage warm-up that lets Akamai's sensor set valid _abck / bm_sz cookies.
• Reading only a single price field and missing promotions that show up as strikethrough was-prices, 'Save $X' badges, or cart-only pricing.

Common questions about this scraper problem