Scrape Best Buy & Target Prices and Stock with Proxies

What goes wrong when this scraper fails

Retail price intelligence teams need current public prices, promotional flags, and stock status for thousands of SKUs across Best Buy and Target, broken out by store and region because both retailers vary price and availability geographically. Best Buy fronts its site and internal availability calls with Akamai Bot Manager, which issues sensor-based challenges and sets short-lived _abck and bm_sz cookies that must be solved by real browser JavaScript. Target routes its product and pricing data through RedSky-style JSON APIs (redsky.target.com/redsky_aggregations/...) guarded by an Imperva-class WAF that fingerprints clients and gates responses on a visitor cookie and rotating api_key. The core difficulty is doing this from enough distinct residential vantage points to capture ZIP-level truth, without your traffic clustering onto a handful of IPs that get flagged in minutes.

Why this failure mode happens

Akamai and Imperva do not simply count requests - they score sessions on IP reputation (datacenter ASNs are effectively pre-flagged), TLS fingerprint (JA3/JA4), HTTP/2 frame ordering, header completeness and order, and behavioral velocity. A datacenter IP sending a header set that does not match a real Chrome build gets a challenge on request one, long before any rate limit matters. On top of that, both retailers deliberately localize pricing and stock to the store serving a given ZIP, so identical requests from different geographies legitimately return different data, and any anti-bot system reads a single IP fanning across hundreds of ZIPs as a strong automation signal.

Challenges that make this hard to automate

Akamai Bot Manager on Best Buy sets _abck / bm_sz cookies via a JS sensor payload; a cold HTTP client with no browser context gets a 'pardon our interruption' page instead of product JSON.
Target's RedSky endpoints demand a currently-valid api_key plus store_id and pricing_store_id, and reject requests missing the visitor_id / GuestID cookie minted by the anti-bot layer.
Pricing and availability differ by store, so you must resolve the correct store_id or fulfillment ZIP for every location you monitor, not just hit a national URL.
Soft blocks return HTTP 200 with a decoy body (empty items array, access-denied HTML), silently corrupting datasets if you only check status codes.
Promotions surface inconsistently - as a strikethrough price, a 'Save $X' badge, a cart-only price, or a limited-time flag - so a single parser field rarely captures the true promotional state.

Approaches that usually fail

Hitting the public product pages directly from a server's own datacenter IP - works for a handful of requests, then Akamai/Imperva challenge everything.
A small pool of static datacenter proxies rotated round-robin - IPs share a bad ASN reputation and get burned together within minutes.
Headless browser farms (Puppeteer/Playwright) without proxy geo-targeting - solves the JS challenge but returns one region's pricing for the whole country.
Buying a third-party 'retail pricing API' - convenient but often stale, missing ZIP granularity, and opaque about where its numbers actually come from.

When residential proxies fix this — and when they cannot

Residential proxies place your requests on real consumer IPs with clean reputation, which is the single biggest factor in passing Akamai and Imperva reputation scoring - the same request that gets challenged from a datacenter ASN often sails through from a residential exit. Geo-targeting by country and city lets you deliberately source each request from the region whose store and ZIP pricing you actually want to capture, so the localized data you get back is correct rather than accidental. Per-request rotation spreads a large SKU crawl across thousands of distinct exits so no single IP accumulates the velocity that triggers a block, while short sticky sessions keep a warmed cookie jar on one exit when a multi-step flow (set ZIP, then read availability) must stay coherent.

How Aethyn residential proxies help here

Aethyn gives you a large residential pool with per-request rotation by default and precise country/city geo-targeting, which is exactly the shape of a retail price-intelligence workload where each request must originate from the region it is measuring. The Premium tier (port 2099) handles the bulk of Best Buy and Target reads cleanly, and when Akamai or Imperva start issuing challenges you can escalate the same code to the Elite pool (port 5499) for higher-trust exits.

Per-request rotation on port 2099 by default, so a 5,000-SKU crawl lands on thousands of distinct residential IPs instead of clustering.
Country and city geo-targeting (-country-us-city-chicago) to source each request from the exact market whose store pricing you need.
Elite pool on port 5499 for the harder targets - higher-trust exits that get through Akamai/Imperva challenges the Premium pool starts to see.
Sticky sessions up to 30 minutes (-session-TOKEN-lifetime-10) to hold one exit while you set a ZIP cookie and then read availability on the same IP.
A single endpoint (proxy.aethyn.io) with a username-encoded config, so switching region, city, or session is a string change, not new infrastructure.
Clean IP reputation across the pool, which is the dominant signal for passing the reputation scoring both Akamai and Imperva apply before any rate limit.

How to implement this with residential proxies

1
Route every request through a geo-targeted Premium residential exit
Start with per-request rotation on the Premium pool (port 2099). Encode the target market directly in the proxy username - country plus city - so each request originates from the region whose store pricing you want. Do not reuse one IP across many ZIPs; that pattern is exactly what the anti-bot systems read as automation. Confirm your exit geography before you trust a single price.
python
```
import requests

# Premium tier -> port 2099. Per-request rotation is the default (no session token).
PROXY = "http://aethyn-XXXXX-country-us-city-chicago:PASSWORD@proxy.aethyn.io:2099"
proxies = {"http": PROXY, "https": PROXY}

# Sanity-check the exit region before scraping localized prices.
who = requests.get("https://ipinfo.io/json", proxies=proxies, timeout=15).json()
print(who.get("city"), who.get("region"), who.get("country"))
# Expect a US exit near Chicago; if it lands elsewhere your ZIP pricing will be wrong.
```
Field note: Verify the exit's city/region on the first request of every session, not just once at startup. Residential exits drift, and a request that silently lands in the wrong metro will hand you correct-looking prices for the wrong store.

Send a complete, ordered browser header set - never a bare User-Agent

Both Akamai and Imperva fingerprint your header set and its order against real Chrome builds. A lone User-Agent with no Accept-Language, sec-ch-ua, or sec-fetch-* headers is an instant tell. Mirror a real recent Chrome exactly, including the client-hint headers, and keep Accept-Language consistent with the country you are geo-targeting from.

python

HEADERS = {
    "User-Agent": (
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
        "(KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36"
    ),
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8",
    "Accept-Language": "en-US,en;q=0.9",
    "sec-ch-ua": '"Chromium";v="126", "Google Chrome";v="126", "Not-A.Brand";v="24"',
    "sec-ch-ua-mobile": "?0",
    "sec-ch-ua-platform": '"Windows"',
    "sec-fetch-dest": "document",
    "sec-fetch-mode": "navigate",
    "sec-fetch-site": "none",
    "upgrade-insecure-requests": "1",
}

r = requests.get("https://www.bestbuy.com/", headers=HEADERS, proxies=proxies, timeout=20)
print(r.status_code, len(r.text))

Field note: Keep Accept-Language aligned with your proxy country. A US-geo exit sending Accept-Language: ru-RU is an internally inconsistent fingerprint that scoring engines flag even when every other signal looks clean.

Pull Target availability from the RedSky JSON API with the right store context

Target's product and fulfillment data comes back as clean JSON from RedSky-style aggregation endpoints, but you must supply a valid api_key, the tcin (Target's SKU id), and store context (store_id, pricing_store_id, and a fulfillment ZIP). Resolve the correct store for each ZIP first, then request pricing and availability against that store. Hold one exit for the store-lookup-then-availability pair with a short sticky session so the cookie jar stays coherent.

python

import requests

# Short sticky session so the store lookup and availability read share one exit + cookie jar.
STICKY = "http://aethyn-XXXXX-country-us-city-dallas-session-tgt42-lifetime-10:PASSWORD@proxy.aethyn.io:2099"
proxies = {"http": STICKY, "https": STICKY}

API = "https://redsky.target.com/redsky_aggregations/v1/web/pdp_fulfillment_v1"
params = {
    "key": "REDSKY_API_KEY",          # visitor api_key observed from the site
    "tcin": "54191097",
    "store_id": "1234",
    "pricing_store_id": "1234",
    "zip": "75201",
    "state": "TX",
    "latitude": "32.7767",
    "longitude": "-96.7970",
    "has_store_positions_store_id": "true",
}
headers = {**HEADERS, "Accept": "application/json", "Origin": "https://www.target.com",
           "Referer": "https://www.target.com/"}

resp = requests.get(API, params=params, headers=headers, proxies=proxies, timeout=20)
data = resp.json()
fulfillment = data["data"]["product"]["fulfillment"]
print(fulfillment["store_options"][0]["location_available_to_promise_quantity"])

Field note: Treat the api_key as perishable, not constant. Target rotates it and scopes it to a visitor context; scrape the current key from the page's bootstrap JSON at session start rather than hardcoding one, or you will get 401s the moment it rotates.

Handle Best Buy's Akamai challenge with a warmed browser context

Best Buy's Akamai layer sets _abck and bm_sz cookies from a JS sensor payload that a cold requests.get cannot produce, so you typically need a real browser to mint them. Drive Playwright through the same Premium residential exit, let the page execute the sensor, then read product and availability. Route Playwright's traffic through the proxy so the browser and the challenge solve happen on the same clean IP.

python

from playwright.sync_api import sync_playwright

PROXY_SERVER = "http://proxy.aethyn.io:2099"
PROXY_USER = "aethyn-XXXXX-country-us-city-seattle-session-bby7-lifetime-10"
PROXY_PASS = "PASSWORD"

with sync_playwright() as p:
    browser = p.chromium.launch(
        headless=True,
        proxy={"server": PROXY_SERVER, "username": PROXY_USER, "password": PROXY_PASS},
    )
    ctx = browser.new_context(locale="en-US", timezone_id="America/Los_Angeles")
    page = ctx.new_page()
    # Landing on a normal page first lets Akamai's sensor set _abck / bm_sz.
    page.goto("https://www.bestbuy.com/", wait_until="domcontentloaded", timeout=45000)
    page.wait_for_timeout(2500)
    page.goto("https://www.bestbuy.com/site/searchpage.jsp?st=laptop",
              wait_until="domcontentloaded", timeout=45000)
    html = page.content()
    print("challenge" if "Pardon Our Interruption" in html else "ok", len(html))
    browser.close()

Field note: Warm the cookie jar by loading the homepage before the product/search page. Akamai's _abck cookie only becomes valid after the sensor runs; jumping straight to a deep URL on a cold context reliably returns the interruption page.

Parse defensively and set the fulfillment ZIP before reading stock

Best Buy resolves store availability from a selected store or ZIP, so set the location context first, then read the availability button state. Never assume a field exists - promos appear as strikethrough prices, 'Save $X' badges, or cart-only prices, and stock shows as 'Add to Cart', 'Sold Out', or 'Check Stores'. Extract each into an explicit, nullable field so a missing element becomes a null, not a crash.

python

from bs4 import BeautifulSoup

def parse_bestbuy_item(html: str) -> dict:
    soup = BeautifulSoup(html, "html.parser")

    def text_or_none(sel):
        el = soup.select_one(sel)
        return el.get_text(strip=True) if el else None

    current = text_or_none('[data-testid="customer-price"]')
    was = text_or_none(".pricing-price__regular-price")   # strikethrough when on promo
    save_badge = text_or_none(".pricing-price__savings")
    cta = text_or_none(".add-to-cart-button") or text_or_none(".fulfillment-add-to-cart-button")

    return {
        "price": current,
        "was_price": was,
        "promo": save_badge,                      # None when not discounted
        "on_promo": bool(was and save_badge),
        "stock_state": cta,                        # 'Add to Cart' / 'Sold Out' / 'Check Stores'
    }

record = parse_bestbuy_item(html)
assert record["price"] is not None, "soft block or layout change - do not store"
print(record)

Field note: Assert a non-empty price before writing to your database. A 200 OK with a null price on Best Buy is almost never a genuinely priceless product - it is a soft block or a layout change, and storing it quietly poisons your promo-detection logic for weeks.

Detect blocks and escalate from Premium to the Elite pool

Distinguish real responses from soft blocks by inspecting the body, not just the status code - watch for 'Pardon Our Interruption', 'Access Denied', empty items arrays, or a suspiciously tiny payload. When challenge rate climbs on the Premium pool, retry the same request on the Elite pool (port 5499) for higher-trust exits before you back off. Back off with jitter so you are not hammering a target that is already flagging you.

python

import random, time, requests

BLOCK_MARKERS = ("Pardon Our Interruption", "Access Denied", "unusual traffic", "px-captcha")

def looks_blocked(resp) -> bool:
    if resp.status_code in (403, 429):
        return True
    body = resp.text
    if any(m in body for m in BLOCK_MARKERS):
        return True
    return len(body) < 800   # decoy pages are tiny

def fetch(url, headers, user_base):
    # Try Premium (2099); on block, escalate the SAME request to Elite (5499).
    for port in (2099, 5499):
        px = "http://" + user_base + ":PASSWORD@proxy.aethyn.io:" + str(port)
        try:
            r = requests.get(url, headers=headers, proxies={"http": px, "https": px}, timeout=20)
        except requests.RequestException:
            time.sleep(1.5 + random.random())
            continue
        if not looks_blocked(r):
            return r
        time.sleep(2 + random.random() * 3)   # jittered backoff before escalating
    return None

resp = fetch("https://www.target.com/p/-/A-54191097",
             {**HEADERS, "Accept": "application/json"},
             "aethyn-XXXXX-country-us-city-dallas")
print("blocked" if resp is None else resp.status_code)

Field note: Escalate to Elite (5499) before you slow your whole crawl down. A rising challenge rate on Premium usually means the specific target is scrutinizing that pool right now - the higher-trust Elite exits often clear it immediately, so you only pay the latency cost on the requests that actually need it.

Best practices that keep scrapers reliable

Collect only public, non-personal price and availability data - no account creation, no login, no checkout automation - and keep your request volume reasonable and within each retailer's Terms of Service.
Geo-target every request to the market it is measuring; a price captured from the wrong region is worse than no data because it looks valid.
Rotate per request by default (port 2099) and reserve sticky sessions for genuine multi-step flows like set-ZIP-then-read-stock.
Assert on real fields (a non-null price, a non-empty items array) before persisting, so soft blocks never enter your dataset.
Scrape perishable tokens - Target's api_key, Akamai's cookies - fresh at session start instead of hardcoding them.
Add jittered backoff and Premium-to-Elite escalation so a flagged pool degrades gracefully instead of hammering a target that is already challenging you.
Cache store_id / pricing_store_id lookups per ZIP so you resolve each location once rather than re-querying it on every SKU.

Common mistakes that burn proxy budget

Checking only the HTTP status code and treating every 200 as success, so tiny decoy 'access denied' bodies get stored as real data.
Scraping a national product URL from a single IP and assuming the price and stock apply everywhere, when both are store-specific.
Sending a bare User-Agent with no Accept-Language or sec-ch-ua client hints, which Akamai and Imperva flag on the first request.
Hardcoding Target's RedSky api_key and getting a wall of 401s the moment it rotates.
Hitting Best Buy's deep product URLs with a cold HTTP client, skipping the homepage warm-up that lets Akamai's sensor set valid _abck / bm_sz cookies.
Reading only a single price field and missing promotions that show up as strikethrough was-prices, 'Save $X' badges, or cart-only pricing.

Common questions about this scraper problem

Is scraping public prices from Best Buy and Target legal?

Collecting publicly displayed, non-personal data like prices and stock status is generally lower-risk than accessing anything behind a login, but it is still governed by each site's Terms of Service. Do not create accounts, log in, or automate checkout; keep your request volume reasonable; and consult counsel for your jurisdiction and use case before running at scale. The framing that keeps you defensible is: public, non-personal, reasonable-volume data only.

Why do I get different prices from different proxy locations?

That is not a bug - it is the whole point. Both Best Buy and Target localize price and availability to the store serving a given ZIP, so a residential exit in Dallas legitimately returns different numbers than one in Seattle. This is exactly why ZIP/city-targeted residential proxies are necessary for accurate price intelligence.

Do I need a real browser for Best Buy, or can I use plain requests?

Best Buy's Akamai Bot Manager sets its _abck and bm_sz cookies from a JavaScript sensor payload that a plain HTTP client cannot generate, so cold requests.get calls usually get the 'Pardon Our Interruption' page. Use Playwright (or a warmed cookie jar captured from a browser session) routed through the same proxy exit, and load the homepage first to let the sensor run before hitting deep URLs.

Can I use requests for Target's RedSky API instead of a browser?

Often yes - RedSky-style endpoints return clean JSON and can be reached with a plain HTTP client, provided you supply a currently-valid api_key, the tcin, and store context (store_id, pricing_store_id, ZIP). The catch is the Imperva-class layer gates responses on a visitor cookie and the api_key rotates, so scrape both fresh from the page at session start rather than hardcoding them.

Premium (2099) or Elite (5499) - which tier should I start on?

Start on Premium (port 2099); it handles the bulk of Best Buy and Target reads cleanly and is the recommended tier for this workload. Escalate the same request to the Elite pool (port 5499) when you see challenge rates climb on a specific target - Elite's higher-trust exits often clear Akamai/Imperva challenges that Premium starts to catch.

How do I tell a soft block from a genuinely empty result?

Inspect the body, not just the status code. A soft block typically returns a 200 with a tiny payload, an 'Access Denied' / 'Pardon Our Interruption' string, or an empty items array where a real product would have fields. Assert on a concrete field - a non-null price, a populated fulfillment object - before storing anything, and treat its absence as a block rather than as data.

When should I use a sticky session instead of per-request rotation?

Default to per-request rotation so a large SKU crawl spreads across thousands of exits and no single IP builds up velocity. Use a short sticky session (for example -session-tgt42-lifetime-10) only when a flow must stay on one IP and cookie jar - like setting a fulfillment ZIP and then reading availability on Target, or warming Best Buy's Akamai cookies before a product read.

How do I find the right store_id for a ZIP on Target?

Target exposes a nearby-stores lookup that maps a ZIP or lat/long to store objects with their store_id. Query it once per ZIP through an exit in that region, cache the resulting store_id and pricing_store_id, and reuse them across all SKUs for that location so you are not re-resolving the store on every request.

What request rate is safe?

There is no published number, and both stacks score on velocity relative to a single IP's reputation, so the right lever is distribution rather than a fixed rate. With per-request rotation across a large residential pool, each individual exit stays well under any threshold; add jittered delays and Premium-to-Elite escalation, and keep aggregate volume reasonable and respectful of the sites' ToS.

How do I reliably detect promotions?

Promos surface inconsistently - a strikethrough regular price, a 'Save $X' badge, a limited-time flag, or a cart-only price - so capture each as its own explicit field and derive an on_promo boolean from their combination rather than trusting one element. On Target, compare the current_retail against reg_retail in the RedSky price object; on Best Buy, watch for the strikethrough regular-price and savings badge together.

What goes wrong when this scraper fails

Why this failure mode happens

Challenges that make this hard to automate

Akamai Bot Manager on Best Buy sets _abck / bm_sz cookies via a JS sensor payload; a cold HTTP client with no browser context gets a 'pardon our interruption' page instead of product JSON.
Target's RedSky endpoints demand a currently-valid api_key plus store_id and pricing_store_id, and reject requests missing the visitor_id / GuestID cookie minted by the anti-bot layer.
Pricing and availability differ by store, so you must resolve the correct store_id or fulfillment ZIP for every location you monitor, not just hit a national URL.
Soft blocks return HTTP 200 with a decoy body (empty items array, access-denied HTML), silently corrupting datasets if you only check status codes.
Promotions surface inconsistently - as a strikethrough price, a 'Save $X' badge, a cart-only price, or a limited-time flag - so a single parser field rarely captures the true promotional state.

Approaches that usually fail

Hitting the public product pages directly from a server's own datacenter IP - works for a handful of requests, then Akamai/Imperva challenge everything.
A small pool of static datacenter proxies rotated round-robin - IPs share a bad ASN reputation and get burned together within minutes.
Headless browser farms (Puppeteer/Playwright) without proxy geo-targeting - solves the JS challenge but returns one region's pricing for the whole country.
Buying a third-party 'retail pricing API' - convenient but often stale, missing ZIP granularity, and opaque about where its numbers actually come from.

When residential proxies fix this — and when they cannot

How Aethyn residential proxies help here

Per-request rotation on port 2099 by default, so a 5,000-SKU crawl lands on thousands of distinct residential IPs instead of clustering.
Country and city geo-targeting (-country-us-city-chicago) to source each request from the exact market whose store pricing you need.
Elite pool on port 5499 for the harder targets - higher-trust exits that get through Akamai/Imperva challenges the Premium pool starts to see.
Sticky sessions up to 30 minutes (-session-TOKEN-lifetime-10) to hold one exit while you set a ZIP cookie and then read availability on the same IP.
A single endpoint (proxy.aethyn.io) with a username-encoded config, so switching region, city, or session is a string change, not new infrastructure.
Clean IP reputation across the pool, which is the dominant signal for passing the reputation scoring both Akamai and Imperva apply before any rate limit.

How to implement this with residential proxies

1
Route every request through a geo-targeted Premium residential exit
Start with per-request rotation on the Premium pool (port 2099). Encode the target market directly in the proxy username - country plus city - so each request originates from the region whose store pricing you want. Do not reuse one IP across many ZIPs; that pattern is exactly what the anti-bot systems read as automation. Confirm your exit geography before you trust a single price.
python
```
import requests

# Premium tier -> port 2099. Per-request rotation is the default (no session token).
PROXY = "http://aethyn-XXXXX-country-us-city-chicago:PASSWORD@proxy.aethyn.io:2099"
proxies = {"http": PROXY, "https": PROXY}

# Sanity-check the exit region before scraping localized prices.
who = requests.get("https://ipinfo.io/json", proxies=proxies, timeout=15).json()
print(who.get("city"), who.get("region"), who.get("country"))
# Expect a US exit near Chicago; if it lands elsewhere your ZIP pricing will be wrong.
```
Field note: Verify the exit's city/region on the first request of every session, not just once at startup. Residential exits drift, and a request that silently lands in the wrong metro will hand you correct-looking prices for the wrong store.

Send a complete, ordered browser header set - never a bare User-Agent

python

HEADERS = {
    "User-Agent": (
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
        "(KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36"
    ),
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8",
    "Accept-Language": "en-US,en;q=0.9",
    "sec-ch-ua": '"Chromium";v="126", "Google Chrome";v="126", "Not-A.Brand";v="24"',
    "sec-ch-ua-mobile": "?0",
    "sec-ch-ua-platform": '"Windows"',
    "sec-fetch-dest": "document",
    "sec-fetch-mode": "navigate",
    "sec-fetch-site": "none",
    "upgrade-insecure-requests": "1",
}

r = requests.get("https://www.bestbuy.com/", headers=HEADERS, proxies=proxies, timeout=20)
print(r.status_code, len(r.text))

Pull Target availability from the RedSky JSON API with the right store context

python

import requests

# Short sticky session so the store lookup and availability read share one exit + cookie jar.
STICKY = "http://aethyn-XXXXX-country-us-city-dallas-session-tgt42-lifetime-10:PASSWORD@proxy.aethyn.io:2099"
proxies = {"http": STICKY, "https": STICKY}

API = "https://redsky.target.com/redsky_aggregations/v1/web/pdp_fulfillment_v1"
params = {
    "key": "REDSKY_API_KEY",          # visitor api_key observed from the site
    "tcin": "54191097",
    "store_id": "1234",
    "pricing_store_id": "1234",
    "zip": "75201",
    "state": "TX",
    "latitude": "32.7767",
    "longitude": "-96.7970",
    "has_store_positions_store_id": "true",
}
headers = {**HEADERS, "Accept": "application/json", "Origin": "https://www.target.com",
           "Referer": "https://www.target.com/"}

resp = requests.get(API, params=params, headers=headers, proxies=proxies, timeout=20)
data = resp.json()
fulfillment = data["data"]["product"]["fulfillment"]
print(fulfillment["store_options"][0]["location_available_to_promise_quantity"])

Handle Best Buy's Akamai challenge with a warmed browser context

python

from playwright.sync_api import sync_playwright

PROXY_SERVER = "http://proxy.aethyn.io:2099"
PROXY_USER = "aethyn-XXXXX-country-us-city-seattle-session-bby7-lifetime-10"
PROXY_PASS = "PASSWORD"

with sync_playwright() as p:
    browser = p.chromium.launch(
        headless=True,
        proxy={"server": PROXY_SERVER, "username": PROXY_USER, "password": PROXY_PASS},
    )
    ctx = browser.new_context(locale="en-US", timezone_id="America/Los_Angeles")
    page = ctx.new_page()
    # Landing on a normal page first lets Akamai's sensor set _abck / bm_sz.
    page.goto("https://www.bestbuy.com/", wait_until="domcontentloaded", timeout=45000)
    page.wait_for_timeout(2500)
    page.goto("https://www.bestbuy.com/site/searchpage.jsp?st=laptop",
              wait_until="domcontentloaded", timeout=45000)
    html = page.content()
    print("challenge" if "Pardon Our Interruption" in html else "ok", len(html))
    browser.close()

Parse defensively and set the fulfillment ZIP before reading stock

python

from bs4 import BeautifulSoup

def parse_bestbuy_item(html: str) -> dict:
    soup = BeautifulSoup(html, "html.parser")

    def text_or_none(sel):
        el = soup.select_one(sel)
        return el.get_text(strip=True) if el else None

    current = text_or_none('[data-testid="customer-price"]')
    was = text_or_none(".pricing-price__regular-price")   # strikethrough when on promo
    save_badge = text_or_none(".pricing-price__savings")
    cta = text_or_none(".add-to-cart-button") or text_or_none(".fulfillment-add-to-cart-button")

    return {
        "price": current,
        "was_price": was,
        "promo": save_badge,                      # None when not discounted
        "on_promo": bool(was and save_badge),
        "stock_state": cta,                        # 'Add to Cart' / 'Sold Out' / 'Check Stores'
    }

record = parse_bestbuy_item(html)
assert record["price"] is not None, "soft block or layout change - do not store"
print(record)

Detect blocks and escalate from Premium to the Elite pool

python

import random, time, requests

BLOCK_MARKERS = ("Pardon Our Interruption", "Access Denied", "unusual traffic", "px-captcha")

def looks_blocked(resp) -> bool:
    if resp.status_code in (403, 429):
        return True
    body = resp.text
    if any(m in body for m in BLOCK_MARKERS):
        return True
    return len(body) < 800   # decoy pages are tiny

def fetch(url, headers, user_base):
    # Try Premium (2099); on block, escalate the SAME request to Elite (5499).
    for port in (2099, 5499):
        px = "http://" + user_base + ":PASSWORD@proxy.aethyn.io:" + str(port)
        try:
            r = requests.get(url, headers=headers, proxies={"http": px, "https": px}, timeout=20)
        except requests.RequestException:
            time.sleep(1.5 + random.random())
            continue
        if not looks_blocked(r):
            return r
        time.sleep(2 + random.random() * 3)   # jittered backoff before escalating
    return None

resp = fetch("https://www.target.com/p/-/A-54191097",
             {**HEADERS, "Accept": "application/json"},
             "aethyn-XXXXX-country-us-city-dallas")
print("blocked" if resp is None else resp.status_code)

Best practices that keep scrapers reliable

Collect only public, non-personal price and availability data - no account creation, no login, no checkout automation - and keep your request volume reasonable and within each retailer's Terms of Service.
Geo-target every request to the market it is measuring; a price captured from the wrong region is worse than no data because it looks valid.
Rotate per request by default (port 2099) and reserve sticky sessions for genuine multi-step flows like set-ZIP-then-read-stock.
Assert on real fields (a non-null price, a non-empty items array) before persisting, so soft blocks never enter your dataset.
Scrape perishable tokens - Target's api_key, Akamai's cookies - fresh at session start instead of hardcoding them.
Add jittered backoff and Premium-to-Elite escalation so a flagged pool degrades gracefully instead of hammering a target that is already challenging you.
Cache store_id / pricing_store_id lookups per ZIP so you resolve each location once rather than re-querying it on every SKU.

Common mistakes that burn proxy budget

Checking only the HTTP status code and treating every 200 as success, so tiny decoy 'access denied' bodies get stored as real data.
Scraping a national product URL from a single IP and assuming the price and stock apply everywhere, when both are store-specific.
Sending a bare User-Agent with no Accept-Language or sec-ch-ua client hints, which Akamai and Imperva flag on the first request.
Hardcoding Target's RedSky api_key and getting a wall of 401s the moment it rotates.
Hitting Best Buy's deep product URLs with a cold HTTP client, skipping the homepage warm-up that lets Akamai's sensor set valid _abck / bm_sz cookies.
Reading only a single price field and missing promotions that show up as strikethrough was-prices, 'Save $X' badges, or cart-only pricing.

Common questions about this scraper problem

Is scraping public prices from Best Buy and Target legal?

Why do I get different prices from different proxy locations?

Do I need a real browser for Best Buy, or can I use plain requests?

Can I use requests for Target's RedSky API instead of a browser?

Premium (2099) or Elite (5499) - which tier should I start on?

How do I tell a soft block from a genuinely empty result?

When should I use a sticky session instead of per-request rotation?

How do I find the right store_id for a ZIP on Target?

What request rate is safe?

How do I reliably detect promotions?

Monitoring Best Buy and Target Prices, Stock, and Promotions at Scale

Key takeaways

On this page

What goes wrong when this scraper fails

Challenges that make this hard to automate

How Aethyn residential proxies help here

How to implement this with residential proxies

Route every request through a geo-targeted Premium residential exit

Send a complete, ordered browser header set - never a bare User-Agent

Pull Target availability from the RedSky JSON API with the right store context

Handle Best Buy's Akamai challenge with a warmed browser context

Parse defensively and set the fulfillment ZIP before reading stock

Detect blocks and escalate from Premium to the Elite pool

Best practices that keep scrapers reliable

Common questions about this scraper problem

Solve this with Aethyn residential proxies

Monitoring Best Buy and Target Prices, Stock, and Promotions at Scale

Key takeaways

On this page

What goes wrong when this scraper fails

Challenges that make this hard to automate

How Aethyn residential proxies help here

How to implement this with residential proxies

Route every request through a geo-targeted Premium residential exit

Send a complete, ordered browser header set - never a bare User-Agent

Pull Target availability from the RedSky JSON API with the right store context

Handle Best Buy's Akamai challenge with a warmed browser context

Parse defensively and set the fulfillment ZIP before reading stock

Detect blocks and escalate from Premium to the Elite pool

Best practices that keep scrapers reliable

Common questions about this scraper problem

Solve this with Aethyn residential proxies

Key takeaways

On this page

What goes wrong when this scraper fails

Challenges that make this hard to automate

How Aethyn residential proxies help here

How to implement this with residential proxies

Route every request through a geo-targeted Premium residential exit

Send a complete, ordered browser header set - never a bare User-Agent

Pull Target availability from the RedSky JSON API with the right store context

Handle Best Buy's Akamai challenge with a warmed browser context

Parse defensively and set the fulfillment ZIP before reading stock

Detect blocks and escalate from Premium to the Elite pool

Best practices that keep scrapers reliable

Common questions about this scraper problem

Related guides & resources

Solve this with Aethyn residential proxies

Key takeaways

On this page

What goes wrong when this scraper fails

Challenges that make this hard to automate

How Aethyn residential proxies help here

How to implement this with residential proxies

Route every request through a geo-targeted Premium residential exit

Send a complete, ordered browser header set - never a bare User-Agent

Pull Target availability from the RedSky JSON API with the right store context

Handle Best Buy's Akamai challenge with a warmed browser context

Parse defensively and set the fulfillment ZIP before reading stock

Detect blocks and escalate from Premium to the Elite pool

Best practices that keep scrapers reliable

Common questions about this scraper problem

Related guides & resources

Solve this with Aethyn residential proxies