How to Scrape Google Search Results Without Getting Blocked

What goes wrong when this scraper fails

A rank-tracking product might need to check 50,000 keywords across 12 countries every morning, and a competitor-research dashboard might fan out hundreds of related queries on demand. Both hit the same wall: Google returns the '/sorry/index' interstitial ('Our systems have detected unusual traffic from your computer network'), a reCAPTCHA, or — more insidiously — a 200 OK with a stripped-down or consent-walled page that your parser silently records as 'zero results.' The data quietly rots before anyone notices the dashboards are wrong.

Why this failure mode happens

Google does not score requests on volume alone. It combines IP reputation (datacenter ASNs are pre-discounted and shared across thousands of scrapers), request velocity per IP and subnet, and client fingerprint — the TLS/JA3 handshake and HTTP/2 frame ordering of libraries like requests or Go's net/http look nothing like Chrome's. Add an inconsistent header set (a desktop User-Agent with no Accept-Language, no sec-ch-ua hints) and you have signed your request as a bot. Once an IP trips the threshold, Google throttles the whole exit, which is why one greedy worker can poison an entire datacenter range for everyone behind it.

Challenges that make this hard to automate

• The 'unusual traffic' interstitial and reCAPTCHA appear after surprisingly few queries from one IP
• Soft blocks return HTTP 200 with consent walls or thin pages, corrupting data without raising an error
• Results are localized by IP geography and personalized by gl/hl — a US IP gives a US SERP regardless of intent
• Result containers use churned, generated class names, so selectors break every few weeks
• &num=100 is no longer honored, so deep result sets require start= pagination and more requests

Approaches that usually fail

• A small datacenter proxy pool — cheap, but the ASNs are flagged and the whole subnet throttles together
• Fixed time.sleep() delays — cut throughput without changing the IP reputation that actually triggers blocks
• Paid SERP APIs (SerpApi, DataForSEO) — fast to start, but pricey at six-figure query volumes and opaque about freshness/locale
• Manual incognito checks — accurate for spot checks but biased by your own IP and impossible to scale or schedule

When residential proxies fix this — and when they cannot

Residential IPs carry the trust of real consumer connections, so an individual query reads as an ordinary search rather than one of a thousand from a server farm. The leverage is not any single 'magic' IP — it is the pool size. Spreading 50,000 queries across tens of thousands of rotating residential exits keeps each IP's velocity near what a human would generate, which is the signal Google actually penalizes. Country targeting then guarantees the SERP you parse is the one a searcher in that market sees, not a US default.

How Aethyn residential proxies help here

Google rewards low velocity and correct geography, and punishes predictable fingerprints. Aethyn maps onto exactly those needs — everything below is controlled through the proxy username on a single endpoint, so a multi-country tracker is one credential, not twelve gateways.

• A large, diverse residential pool so a 50k-keyword run spreads thin — the velocity-per-IP that keeps SERPs flowing
• Country targeting (-country-XX) that aligns the exit IP with your gl/hl so localized rankings are accurate
• Elite city/ISP targeting (-city-) for local-intent SERPs where rankings shift block by block
• Per-request rotation by default so consecutive queries never stack on one IP
• Sticky sessions (-session-…-lifetime-, up to 30 min) for the rare case where you page a single keyword from one exit

How to implement this with residential proxies

1. 1

   Pick the right tier and pin the locale

   Google is a high-security target, so start on the Elite residential pool (port 5499). Pin the country with -country-XX in the username and set the matching gl (country) and hl (language) query parameters. These two must agree: a -country-de exit with gl=us produces a contradictory SERP that reflects no real searcher and is easy for Google to flag as inconsistent.

   cURLCopy

   curl -x "http://aethyn-XXXXX-country-de:PASSWORD@proxy.aethyn.io:5499" \
     "https://www.google.com/search?q=residential+proxies&hl=de&gl=de"

   Field note: For local-intent keywords ("plumber near me", "coffee"), country alone is too coarse — rankings change by city. Use the Elite tier with -city- targeting so the SERP reflects the neighborhood you are actually reporting on.

2. 2

   Send queries through rotating residential IPs (Python)

   Rotate on every request so consecutive queries never stack on one IP. Send a realistic, complete header set — a bare User-Agent with no Accept-Language is a classic bot tell. requests reads the same proxy URL for HTTP and HTTPS. Note the start parameter: since &num=100 stopped being honored, you collect deep results by stepping start in increments of 10.

   Python (requests)Copy

   import requests
   
   PROXY = "http://aethyn-XXXXX-country-de:PASSWORD@proxy.aethyn.io:5499"
   proxies = {"http": PROXY, "https": PROXY}
   
   HEADERS = {
       "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
                     "(KHTML, like Gecko) Chrome/124.0 Safari/537.36",
       "Accept-Language": "de-DE,de;q=0.9,en;q=0.6",
       "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
   }
   
   def fetch_page(query, start=0, gl="de", hl="de"):
       params = {"q": query, "gl": gl, "hl": hl, "start": start}
       return requests.get("https://www.google.com/search",
                           params=params, headers=HEADERS, proxies=proxies, timeout=30)
   
   resp = fetch_page("residential proxies")
   print(resp.status_code, len(resp.text))

   Field note: Plain HTTP clients also leak a non-browser TLS/JA3 fingerprint. If you still see challenges after fixing headers and velocity, switch to a client that mimics a browser handshake (e.g. curl_cffi with impersonate="chrome") or move to the Playwright approach below — the residential IP fixes reputation, the browser fixes the fingerprint.

3. 3

   Parse organic results defensively

   Google's result markup uses generated, frequently-rotated class names, so never anchor your parser on them. The stable structure is: the results live under div#search > div#rso, each organic result is an <a> whose href is a real destination and which contains an <h3> title. Anchoring on that survives most layout churn, and you should always guard against the soft-block case where #search is missing entirely.

   Python (BeautifulSoup)Copy

   from bs4 import BeautifulSoup
   
   def parse_serp(html):
       soup = BeautifulSoup(html, "html.parser")
       rso = soup.select_one("div#search div#rso")
       if rso is None:
           raise BlockedError("No #rso container — likely a consent wall or soft block")
       results = []
       for h3 in rso.select("a > h3"):
           link = h3.find_parent("a")
           href = link.get("href", "")
           if href.startswith("http"):
               results.append({"title": h3.get_text(strip=True), "url": href})
       return results

   Field note: Always assert a non-empty result set before writing to your database. A 200 OK with zero parsed results is almost never a keyword with no rankings — it is a soft block, and silently storing it is how rank-tracking dashboards end up quietly wrong for weeks.

4. 4

   Use a sticky session only when paging one keyword

   Per-request rotation is the right default. The exception is paging deep results for a single keyword (start=0,10,20…): switching IPs mid-keyword can yield slightly different result sets, so pin one exit for that short burst with a session id and lifetime, then drop back to rotation for the next keyword.

   Python (sticky)Copy

   # Same exit IP for ~10 minutes while paging one keyword
   def sticky_proxy(keyword_id):
       user = f"aethyn-XXXXX-country-de-session-kw{keyword_id}-lifetime-10"
       url = f"http://{user}:PASSWORD@proxy.aethyn.io:5499"
       return {"http": url, "https": url}

   Field note: Keep the lifetime as short as the job needs (a couple of minutes is plenty for three pages). A long-lived sticky session is just a slow way to over-use one IP — the opposite of what keeps SERP collection healthy.

5. 5

   Fall back to a real browser for consent walls (Playwright)

   In the EU, Google often redirects to consent.google.com before showing results. A headless browser through the same residential proxy can accept consent and render the SERP exactly as a real visit would — and it carries a genuine browser fingerprint, sidestepping the TLS/JA3 problem of plain HTTP clients. Reserve this for queries that get walled, since a browser costs far more per request than requests.

   Python (Playwright)Copy

   from playwright.sync_api import sync_playwright
   
   with sync_playwright() as p:
       browser = p.chromium.launch(proxy={
           "server": "http://proxy.aethyn.io:5499",
           "username": "aethyn-XXXXX-country-de",
           "password": "PASSWORD",
       })
       page = browser.new_page(locale="de-DE")
       page.goto("https://www.google.com/search?q=residential+proxies&gl=de&hl=de")
       # Accept the EU consent dialog if it appears
       btn = page.query_selector("button:has-text('Accept all'), button:has-text('Alle akzeptieren')")
       if btn:
           btn.click()
       page.wait_for_selector("div#search", timeout=15000)
       html = page.content()
       browser.close()

   Field note: Set the browser locale to match the proxy country (locale="de-DE" with a German exit). A German IP rendering a US-English browser is an obvious mismatch and a soft signal that you are automating.

6. 6

   Treat blocks as a throttle signal, not a retry loop

   Detect the block states explicitly — HTTP 429, a redirect to /sorry/index, or a missing #rso — and respond by slowing down and rotating, never by retrying immediately on the same path. Exponential backoff with jitter plus per-request rotation lets the next attempt leave from a fresh IP after the pressure has eased. If 429s spike across many IPs at once, that is your system telling you global concurrency is too high.

   Python (block detection + backoff)Copy

   import time, random
   
   class BlockedError(Exception):
       pass
   
   def is_blocked(resp):
       if resp.status_code == 429:
           return True
       if "/sorry/index" in resp.url or "consent.google.com" in resp.url:
           return True
       return "detected unusual traffic" in resp.text
   
   def fetch_with_retry(query, attempts=4):
       for i in range(attempts):
           resp = fetch_page(query)            # rotates IP each call
           if not is_blocked(resp):
               return parse_serp(resp.text)
           time.sleep(min(60, 2 ** i) + random.random())  # back off + jitter
       raise BlockedError(f"Still blocked after {attempts} attempts: {query}")

   Field note: Track your block rate as a first-class metric. A healthy SERP pipeline sits near zero; a slow climb usually means your concurrency crept up or a parser change is misreading good pages as blocks. Alert on it before the data silently degrades.

Best practices that keep scrapers reliable

• Spread volume across the pool — a wide, slow fan-out beats a few fast IPs every time
• Keep -country aligned with gl/hl, and the browser locale aligned with both
• Send a full, consistent header set (UA + Accept-Language + Accept), not just a User-Agent
• Anchor parsers on stable structure (#search > #rso, a > h3), never on generated class names
• Validate a non-empty result set before persisting, and monitor block rate as a metric
• Cache SERPs and re-check on a cadence (daily/weekly) instead of re-querying constantly
• Stay within Google's Terms of Service and collect only public, non-personal data

Common mistakes that burn proxy budget

• Storing a 200 OK with zero parsed results as 'no rankings' — it is almost always a soft block
• Mismatching exit country and gl/hl, collecting a SERP no real user in that market sees
• Sending a desktop User-Agent with no Accept-Language — an instant bot tell
• Retrying on the same IP after a 429 instead of backing off and rotating
• Assuming &num=100 still works; deep result sets need start= pagination
• Hard-coding today's CSS class names and re-breaking every Google layout tweak

Common questions about this scraper problem