What goes wrong when this scraper fails
An SEO team reports rankings to clients across a dozen countries and, for local businesses, dozens of cities. Checking from the office gives a single personalized, location-biased view that matches no client's actual market. Worse, the SERP itself has changed: ads, the local pack, 'People also ask', and AI Overviews now occupy the space organic results used to, so a naive 'position 3' can mean wildly different visibility than it did two years ago. Accurate tracking has to control location, neutralize personalization, and account for what the modern SERP actually looks like.
Why this failure mode happens
Search results are localized by the requester's IP and the gl/hl parameters, and personalized by account history and prior searches. A central server sees exactly one biased view. Layer on velocity-based throttling for repeated automated queries from datacenter IPs, and the data is both unrepresentative and unreliable. And because Google increasingly injects SERP features above the organic block, position numbers are only meaningful if you also record what surrounds them.
Challenges that make this hard to automate
- Rankings that legitimately differ by country and, for local intent, by city
- Personalization (history, login) biasing any single-location check
- Velocity throttling that corrupts repeated automated queries
- Keeping IP geography, gl/hl, and city all consistent
- Counting position correctly when ads, local packs, and AI Overviews intrude
Approaches that usually fail
- Manual incognito checks — still biased by your own IP and impossible to schedule
- Datacenter proxies — throttled and geographically wrong for the market
- Generic rank tools — convenient but often coarse on geo and opaque on method
- Trusting 'incognito' to remove bias — it hides history but not IP-based localization
When residential proxies fix this — and when they cannot
Querying from residential IPs in each target market returns the localized rankings real users in that market see, and rotating across the pool keeps per-query velocity low so scheduled tracking stays clean and unblocked. For local-intent terms, city-level exits capture the neighborhood-by-neighborhood variation that national tracking simply cannot.
How Aethyn residential proxies help here
Rank tracking is recurring, geo-specific, and volume-sensitive. Aethyn maps country (and Elite city) targeting onto a single rotating endpoint so one tracker sweeps every market you report on.
- Country targeting aligned to gl/hl for accurate national rankings
- Elite city/ISP targeting for local-pack and local-intent terms
- 195+ countries for full international rank coverage
- Per-request rotation to keep query velocity low across large keyword sets
- Premium pricing that keeps daily/weekly scheduled tracking affordable
How to implement this with residential proxies
- 1
Define the keyword × market matrix
Map each keyword to the countries — and, for local intent, the cities — you report on, with the matching language. This matrix is the unit of tracking: 'best running shoes' in (de, de-DE) is a different measurement from the same term in (us, en-US), and you'll store and trend them separately.
Field note: Tag each row with search intent. Local-intent terms ('dentist', 'gym near me') need city-level tracking; informational terms usually only need country. Tracking everything at city level is expensive and noisy for terms where the city doesn't change the result.
- 2
Query each market with a neutralized, locale-matched request
Set the proxy country and the engine's gl/hl to the same market, and neutralize personalization: send no logged-in cookies and add pws=0 to drop personalized results. The IP says where you are, gl/hl says which market's index to use, and pws=0 says 'don't bias this by anyone's history'.
Python (requests)import requests def serp(keyword, country, hl, start=0): proxy = f"http://aethyn-XXXXX-country-{country}:PASSWORD@proxy.aethyn.io:2099" proxies = {"http": proxy, "https": proxy} params = {"q": keyword, "gl": country, "hl": hl, "pws": 0, "start": start} headers = {"Accept-Language": f"{hl},{hl};q=0.9", "User-Agent": "Mozilla/5.0 ... Chrome/124.0 Safari/537.36"} # fresh client => no carried-over cookies/personalization return requests.get("https://www.google.com/search", params=params, headers=headers, proxies=proxies, timeout=30) r = serp("residential proxies", "de", "de-DE") print(r.status_code)Field note: Use a fresh, cookie-less request per check. A persisted session quietly accumulates personalization signals, so your 'market ranking' slowly drifts toward whatever this particular session has been doing.
- 3
Count position the way the SERP actually renders
Decide and document your position model before parsing: are ads, the local pack, and AI Overviews counted, or only the classic organic (#rso a > h3) results? Find your target domain in that ordered list and record the rank — plus the SERP features present — so the number is interpretable later.
Python (find position)from urllib.parse import urlparse from bs4 import BeautifulSoup def find_position(html, target_domain): soup = BeautifulSoup(html, "html.parser") rso = soup.select_one("div#search div#rso") if rso is None: raise BlockedError("No #rso — soft block or consent wall") for i, h3 in enumerate(rso.select("a > h3"), start=1): href = h3.find_parent("a").get("href", "") if urlparse(href).netloc.endswith(target_domain): return i # organic position return None # not in this page of resultsField note: Record the SERP features alongside the position (ai_overview, local_pack, ads_count). A page can hold position 1 organically and still lose clicks to an AI Overview above it — tracking only the number hides that story from your reporting.
- 4
Schedule, rotate, and trend
Run on a daily or weekly cadence, rotate IPs per query to keep velocity low, paginate with start=10,20 for deeper positions, and back off on any challenge. Persist (keyword, market, position, features, timestamp) so reports show movement — and so a drop caused by a new SERP feature is distinguishable from an actual ranking loss.
Best practices that keep scrapers reliable
- Track per market (country, and city for local intent), never as one global number
- Neutralize personalization: cookie-less requests plus pws=0
- Keep exit IP, gl/hl, and city all aligned
- Document and apply a consistent position model (what counts as rank 1)
- Record SERP features alongside position, not just the number
- Rotate per query, store time-stamped history, and back off on challenges
Common mistakes that burn proxy budget
- Reporting one location's rankings as if they were global
- Mismatching IP country and gl/hl, measuring a market no user sees
- Carrying cookies/personalization across checks and drifting the data
- Counting position inconsistently across runs so trends are meaningless
- Ignoring AI Overviews/local packs and misreading lost clicks as lost rank
- Querying too fast and throttling the very pipeline you depend on