How to Scrape Google Shopping Results Without Getting Blocked

Google Shopping offer pipeline

1. Pin the market

   Do: encode country on the username, match gl, hl, and Playwright locale, add Elite city when pickup or shipping is the metric, then verify the exit with an IP lookup. Why: Shopping localizes catalog by country and local offers by approximate location. Wrong: a country-de exit with gl=us, or a requested city the tunnel did not land in. Store: requested_country, requested_city, resolved_city, and the IP lookup timestamp.

2. Launch a browser context

   Do: launch Playwright Chromium with a per-context residential proxy and credentials in username and password fields. Why: Shopping hydrates after JavaScript; stock HTTP clients see a SERP shell or fail TLS checks. Wrong: user:pass in the proxy URL (407), one context reused across SKUs (cookie leak), or requests instead of a browser. Store: user-agent/locale on the context, not a password.

3. Render the offer grid

   Do: open the public Shopping surface, wait for the grid to hydrate, and read visible merchant, product, price, currency, and shipping or pickup text. Why: generated class names are not a parser contract; entry chrome (tbm=shop / udm=28) drifts. Wrong: parsing #rso as offers, or treating a still-loading page as empty. Store: raw visible fields only after classification says populated.

4. Classify the document

   Do: label the HTML as populated, empty_grid, consent, challenge, or wrong_geo before any insert. Why: HTTP 200 is not successful Shopping collection — those five documents can share a 200. Wrong: writing merchants=[] for a consent wall or a /sorry/ page. Store: status, and persist prices only when status is populated.

5. Key the offers

   Do: write query, geo keys, merchant, product, price, currency, shipping, pickup, status, and collected_at. Why: country-catalog FX and city-local inventory are different series; mixing them poisons both. Wrong: one sticky IP for a huge SKU list, or merging Shopping rows into a SERP table. Store: country rows and city rows in separate series.

How to scrape Google Shopping results with Playwright

1. 1

   Pin the market and make locale agree

   Start by naming the object. Country targeting (-country-us) owns catalog, currency, and merchant eligibility. City targeting (-city-chicago, Elite only) owns local inventory, pickup, shipping ETA, and some sponsored slots. ISP targeting (-isp-, Elite only) is for carrier-specific research and is usually unnecessary for standard Shopping collection. Then make every geographic signal agree: exit country, gl, hl, and Playwright locale. A country-de exit with gl=us, or an en-US locale on a DE session, is a contradictory request and an unreliable dataset. Add Elite city when pickup or shipping is the metric, then verify the city you landed in with an IP lookup before the first Shopping request. Asking for a city is not landing there — city and ISP suffixes resolve to the closest available match and can fall back to country. Store requested_city next to resolved_city. Stay on Premium (HTTP 2099 / SOCKS5 1099) when the job is national catalog and FX only; Premium refuses a city suffix rather than ignore it. Do not mix those rows into a city series.

   textCopy

   Requirement                    Premium (-country-)     Elite city (-city-)      Elite ISP (-isp-)
   ------------------------------  ----------------------  -----------------------  -------------------------
   National catalog / FX           Yes                     Yes                      Yes (usually overkill)
   City-specific Shopping results  No                      Yes                      Yes
   Local inventory                 No                      Yes                      Yes
   Store pickup                    No                      Yes                      Yes
   Shipping / location testing     No                      Yes                      Yes
   ISP / carrier-specific research No                      No                       Yes
   Standard Shopping collection    Yes                     Yes                      Usually unnecessary
   Sticky pagination / session     Available               Available                Available

   cURLCopy

   curl -x "http://aethyn-XXXXX-country-us-city-chicago:PASSWORD@proxy.aethyn.io:5499" \
     "https://ipinfo.io/json"

   Field note: cURL here is only to prove the exit. Do not parse Shopping offers from this response — the offer grid will not be in it. Record resolved_city from the lookup, not from the username you sent.

2. 2

   Run a small end-to-end pass: query, proxy, verify, classify, extract

   Put credentials in username and password fields. Chromium strips user:pass from a proxy URL and answers 407. Launch with Playwright’s per-context placeholder, then set the real proxy on each newContext. One context per independent SKU query so cookies never leak onto the next exit. The pass is: encode the query, open a geo-matched residential context, verify the exit, navigate to the public Shopping page, classify the document, extract visible offer fields only if the grid is populated, then write a structured record. Entry chrome drifts between tbm=shop and udm=28 — keep a small adapter around navigation. Generated class names are not a parser contract; prefer visible merchant, product, price, currency, and shipping or pickup text. This is public-page collection at a human-like rate, not a login, CAPTCHA, or checkout bypass.

   Python (Playwright)Copy

   from datetime import datetime, timezone
   import json
   from playwright.sync_api import sync_playwright
   
   QUERY = "wireless earbuds"
   REQUESTED_CITY = "chicago"
   PROXY = {
       "server": "http://proxy.aethyn.io:5499",
       "username": "aethyn-XXXXX-country-us-city-chicago",
       "password": "PASSWORD",
   }
   CHALLENGE = ("unusual traffic", "sorry/index", "verify you are human")
   CONSENT = ("before you continue", "consent.google.com")
   
   def classify(html: str, n_offers: int, geo_ok: bool) -> str:
       if not geo_ok:
           return "wrong_geo"
       low = html.lower()
       if any(m in low for m in CHALLENGE):
           return "challenge"
       if any(m in low for m in CONSENT):
           return "consent"
       if n_offers:
           return "populated"
       return "empty_grid"
   
   def extract_offers(page):
       # Generated class names drift. Prefer visible price-like text, then cap the sample.
       rows = []
       for loc in page.get_by_text("$").all()[:12]:
           text = " ".join((loc.inner_text() or "").split())
           if text:
               rows.append({"raw": text[:240]})
       return rows
   
   with sync_playwright() as p:
       browser = p.chromium.launch(proxy={"server": "per-context"}, headless=True)
       context = browser.new_context(proxy=PROXY, locale="en-US")
       page = context.new_page()
       page.goto("https://ipinfo.io/json", wait_until="domcontentloaded")
       geo = json.loads(page.inner_text("body"))
       resolved_city = (geo.get("city") or "").strip()
       geo_ok = REQUESTED_CITY.lower() in resolved_city.lower()
       page.goto(
           "https://www.google.com/search?q=wireless+earbuds&tbm=shop&gl=us&hl=en",
           wait_until="domcontentloaded",
       )
       page.wait_for_timeout(2500)  # grid hydrate; tbm=shop / udm=28 chrome drifts
       html = page.content()
       offers = extract_offers(page) if geo_ok else []
       status = classify(html, len(offers), geo_ok)
       record = {
           "query": QUERY,
           "requested_city": REQUESTED_CITY,
           "resolved_city": resolved_city,
           "status": status,
           "offers": offers if status == "populated" else [],
           "collected_at": datetime.now(timezone.utc).isoformat(),
       }
       print(json.dumps(record, indent=2))
       context.close()
       browser.close()

   Field note: If per-context proxies are ignored, you launched Chromium without the placeholder. Launch proxy: { server: 'per-context' }, then set the real gateway on newContext(). Adapt the extractor when Google’s chrome changes; keep classification and geo keys stable.

3. 3

   Parse merchant, price, currency, shipping — not organic rank

   Wait for offer cards to settle before you extract. Stable enough signals are a product title, a merchant name, a visible price with a currency, and a shipping or pickup line. Do not store organic #rso links as offers. If the page is a classic SERP with a shopping module, that is a different object — collect it with the Google Search how-to and do not merge the rows. Fail closed: if you cannot name merchant and price on a populated grid, do not invent them from nearby SERP chrome.

   Python (record fields)Copy

   offer = {
       "query": "wireless earbuds",
       "merchant": "Loop Audio",
       "product": "Loop Audio buds",
       "price": "79.00",
       "currency": "USD",
       "shipping": "Free delivery",
       "pickup": None,  # or a store-pickup chip when the card shows one
   }

   Field note: Entry URLs for Shopping drift (tbm=shop, udm=28, consent). Keep a small adapter around navigation; keep classification and offer keying stable.

4. 4

   Classify empty grids before you store them

   A 200 with zero offer cards is not automatically “this SKU has no merchants.” Distinguish five documents: a populated offer grid; a genuine empty grid (the UI loaded and attached no cards for this query and IP); a consent wall; a challenge or /sorry/ interstitial; and an invalid or wrong geographic exit. Those five outcomes must not share one empty array. Fail closed: if you cannot name the document, do not persist a price. HTTP 200 is not successful Shopping collection.

   Python (record shape)Copy

   record = {
       "query": "wireless earbuds",
       "requested_country": "us",
       "requested_city": "chicago",
       "resolved_city": "Chicago",  # from IP lookup at session start
       "collected_at": "2026-09-18T12:00:00Z",
       "status": "populated",  # populated | empty_grid | consent | challenge | wrong_geo
       "merchant": "Loop Audio",
       "price": "79.00",
       "currency": "USD",
       "shipping": "pickup",
   }
   
   # Never write empty_grid, consent, challenge, or wrong_geo as merchants=[].
   # Write status=... and omit prices.

   Field note: Assert resolved_city (and country) before the first Shopping request. A mis-typed username is a silent wrong market, not a block. IP geolocation is a sanity check, not GPS.

5. 5

   Rotate per SKU; sticky only for pagination and store cookies

   Rotate: close the Playwright context when the independent SKU query is done so the next query gets a new household exit. Use a sticky session only while you page the same grid, while a store-selection cookie must stay on the IP that earned it, or while a consent cookie must survive the next render — typically well inside the default 30-minute window. Terminate the sticky session when that flow ends: drop the session id, close the context, and rotate again. Do not pin one sticky IP for a 5,000-SKU morning run. Sticky holds a rotating-pool residential IP for 1–1440 minutes (default 30). It is not a static, dedicated, or ISP product. Elite ISP suffixes are targeting, not a different SKU.

   Python (sticky username)Copy

   # Same rotating-pool IP for ~30 minutes while a store cookie lives
   user = "aethyn-XXXXX-country-us-city-chicago-session-shop42-lifetime-30"
   proxy = {
       "server": "http://proxy.aethyn.io:5499",
       "username": user,
       "password": "PASSWORD",
   }
   # When pagination / store selection ends: drop -session-…-lifetime-N and close the context.

   Field note: Sticky holds a rotating-pool residential IP. It is not a static, dedicated, or ISP product. Elite ISP suffixes are targeting, not a different SKU.

What are you actually scraping?

A Google Search SERP is a ranked list of organic results. Collectors usually parse the #rso container for title, URL, and snippet. Google Shopping is a different product. The unit of a Shopping page is an offer card: a merchant, a product title, a price, a currency, and often a shipping line or a store-pickup chip. Those fields describe commerce, not rank. Mixing them with organic SERP rows is how price dashboards start reporting a blue link as a seller. Store Shopping offers in their own table. Keep country-catalog rows — currency, eligible merchants, national price — separate from city-local rows — pickup, shipping ETA, local inventory. The same HTTP 200 can carry a populated grid, a genuine empty carousel, a consent wall, a /sorry/ challenge, or a session that resolved in the wrong city. Classify the document before you persist a price. HTTP 200 is not successful Shopping collection.

What breaks Google Shopping collection

• Shopping is a rendered offer grid — there is no stable public JSON feed to poll
• Entry URLs drift between tbm=shop and udm=28; generated class names are not a parser contract
• Country and city are different variables; mixing them in one series poisons FX and local inventory alike
• gl, hl, browser locale, requested city, and resolved city can disagree; contradictory geo signals produce an unreliable dataset
• Soft 200s (empty carousel, challenge, consent, wrong geo) look like “no merchants” if you do not classify them
• Consent walls and store cookies bind to the IP and browser that earned them

How Aethyn Elite fits Google Shopping collection

Use Premium when the job is national catalog, currency, and merchant eligibility. Use Elite when pickup, shipping, local inventory, or a sponsored mix that changes by metro is the object — city targeting lives on Elite, and Premium refuses a city suffix rather than silently ignore it. ISP targeting (-isp-) is Elite-only and usually unnecessary for Shopping unless the carrier is the research variable. Sticky sessions help only while pagination or a store cookie must survive; rotate again when that flow ends. HTTP CONNECT is the usual Playwright path; SOCKS5 if the client requires it. Same residential pool either way. A proxy does not skip Google’s terms, robots.txt, or rate limits.

• Premium HTTP 2099 / SOCKS5 1099 for country catalog and FX — encode -country-XX; do not put a city on this username
• Elite HTTP 5499 / SOCKS5 3499 with -country-XX-city- when pickup, shipping, local inventory, or metro-varying offers are the metric
• Elite -isp- only when the carrier or network is the variable; skip it for standard Shopping collection
• Per-request rotation by omitting a session id so each independent SKU query gets a fresh household exit
• Sticky 1–1440 minutes (default 30) as -session-…-lifetime-N only while pagination or a store cookie lives, then drop the session id

Google Shopping scraping FAQ

Best practices for Shopping offer pipelines

• Collect public Shopping offer cards only. Do not log in, do not solve CAPTCHAs, and do not treat a proxy as a license to ignore Google’s terms, robots.txt, or rate limits.
• Match exit country, gl, hl, and Playwright locale on every context. Contradictory locale is both a bad dataset and a bot tell.
• Verify the exit with an IP lookup at the start of the context and store resolved_city next to requested_city and the price.
• Keep country-catalog series and city-local series in different tables. Mixing them is how national FX pretends to be store inventory.
• Classify populated vs empty_grid vs consent vs challenge vs wrong_geo before insert. Empty-as-absent is how price dashboards lie.
• One Playwright context per independent SKU query. Reusing a context across queries leaks cookies onto a new identity.
• Reserve sticky (1–1440 minutes, default 30) for pagination and the identity that holds a store or consent cookie; terminate the session when that flow ends.
• Keep classic SERP collection on its own pipeline. Merging #rso HTML into a Shopping table poisons both products.

Common mistakes when scraping Google Shopping

• Using requests or httpx and parsing a SERP as if it were Shopping offers.
• Reporting a US datacenter crawl as a DE (or any other) market’s catalog.
• Sending gl/hl/locale that contradict the exit country, then treating the mix as a valid national catalog.
• Storing a 200 with zero cards as merchants=[] instead of status=empty_grid, consent, challenge, or wrong_geo.
• Embedding user:pass in the Playwright proxy server URL and debugging the resulting 407 as a Google block.
• Rotating the IP while a store cookie is still required, then calling the next empty page “out of stock.”
• Pinning one sticky IP for the entire SKU list and recreating a per-IP velocity problem.
• Mixing city-local pickup rows into a national FX series.