How to Scrape Google Maps at Scale

What goes wrong when this scraper fails

A lead-gen team wants every 'HVAC contractor' across 200 US metros; a local-SEO agency needs to see how a client ranks in the Map Pack from each suburb it serves. Both depend on Maps returning what a resident would see. Pull those queries from a single server and Maps either throttles the feed, redirects to a consent wall, or returns a listing set skewed to the datacenter's location — so the 'Chicago' results come back quietly half-populated with the wrong city.

Why this failure mode happens

Maps personalizes ranking by the viewport and the requester's inferred location, then layers Google's standard anti-automation on top: velocity scoring, consent gating (consent.google.com in the EU), and detection of headless or automated browsers. A datacenter IP fails on two fronts at once — it is low-trust and geographically wrong — so even when it is not outright blocked, the listings it returns do not represent any real local searcher.

Challenges that make this hard to automate

• Ranking is location-sensitive, so the exit IP's city must match the market, not just the country
• The feed is JavaScript-rendered and lazy-loaded — raw HTTP returns almost nothing usable
• Scrolling must continue to the end-of-list sentinel, or you silently truncate the results
• Reviews are personal data, creating GDPR/CCPA obligations distinct from business facts
• Business names repeat across locations, so naive name-based dedupe merges distinct places

Approaches that usually fail

• The official Places API — clean and reliable, but metered, field-limited, and costly across hundreds of metros
• Datacenter proxies — wrong geography and throttled before a city sweep even finishes
• A single headless browser on one IP — blocked after a handful of searches
• Manual collection in Maps — accurate but impossible across 200 cities on a schedule

When residential proxies fix this — and when they cannot

Residential IPs in the target city make each search read as a local user and return the listings residents actually see, while a large rotating pool keeps per-IP velocity low enough to sweep many metros without throttling. City-level exits are precisely what datacenter proxies and VPNs cannot replicate — and for Map Pack work, city precision is the entire point.

How Aethyn residential proxies help here

Local data collection needs two things at once: geographic precision and enough IP diversity to query hundreds of areas without tripping limits. Aethyn's Elite tier provides both, controlled through the username.

• Elite city/ISP targeting (-city-) so 'coffee in Chicago' returns Chicago's listings, not a datacenter's
• A large rotating pool to sweep hundreds of metros while keeping per-IP velocity low
• Sticky sessions (-session-…-lifetime-) to hold one exit through a single scroll-and-paginate pass
• 195+ country coverage for international local data from one credential
• Standard auth that drops straight into Playwright/Puppeteer for the JavaScript feed

How to implement this with residential proxies

1. 1

   Target the city and bias the viewport

   On Elite, pin country and city in the username so the request originates locally. Then bias the map itself by appending /@lat,lng,zoomz to the search URL — the IP sets who you appear to be, the coordinates set where the map is centered. Together they pin the result set to a real neighborhood.

   cURLCopy

   # City-targeted exit + viewport centered on downtown Chicago (zoom 13)
   curl -x "http://aethyn-XXXXX-country-us-city-chicago:PASSWORD@proxy.aethyn.io:5499" \
     "https://www.google.com/maps/search/coffee+shops/@41.8781,-87.6298,13z"

   Field note: Keep a small lookup of lat/lng centroids per metro you crawl. Pairing the right coordinates with a city-matched IP is what separates a clean local data set from one subtly contaminated by a neighboring region's listings.

2. 2

   Render and scroll the feed to the end (Playwright)

   Maps is fully JavaScript-driven, so use a headless browser through the residential proxy. The catch most scrapers miss: the results live in a scrollable side panel (div[role='feed']), not the window — you must scroll that element repeatedly until Google appends the 'You've reached the end of the list' sentinel, otherwise you only ever capture the first dozen listings.

   Python (Playwright)Copy

   from playwright.sync_api import sync_playwright
   
   def collect_place_urls(query, lat, lng, max_rounds=40):
       url = f"https://www.google.com/maps/search/{query}/@{lat},{lng},13z"
       with sync_playwright() as p:
           browser = p.chromium.launch(proxy={
               "server": "http://proxy.aethyn.io:5499",
               "username": "aethyn-XXXXX-country-us-city-chicago",
               "password": "PASSWORD",
           })
           page = browser.new_page(locale="en-US")
           page.goto(url)
           feed = page.wait_for_selector("div[role='feed']", timeout=20000)
           for _ in range(max_rounds):
               page.eval_on_selector("div[role='feed']",
                                     "el => el.scrollBy(0, el.scrollHeight)")
               page.wait_for_timeout(1500)
               if page.get_by_text("reached the end of the list").count():
                   break
           links = page.eval_on_selector_all(
               "div[role='feed'] a[href*='/maps/place/']",
               "els => els.map(e => e.href)")
           browser.close()
           return list(dict.fromkeys(links))  # de-dupe, preserve order

   Field note: Cap the scroll rounds and watch for the sentinel — never loop forever. If the feed stops growing well before the sentinel, that is usually a soft throttle, not the genuine end: drop the session, rotate, and resume the metro later.

3. 3

   Extract structured fields per listing

   Open each place URL (reusing the same sticky session) and read the fields off stable, role-based selectors rather than churned class names. Capture the place_id from the URL as your dedupe key — names like 'Starbucks' appear thousands of times and will collide.

   Python (extraction)Copy

   import re
   
   def parse_place(page):
       name = page.locator("h1").first.inner_text()
       rating = page.locator("div.fontDisplayLarge").first.inner_text()  # e.g. "4.6"
       reviews = page.get_by_text(re.compile(r"\d[\d,]* reviews")).first.inner_text()
       category = page.locator("button[jsaction*='category']").first.inner_text()
       place_id = re.search(r"!1s(0x[0-9a-f:]+)", page.url)
       return {
           "place_id": place_id.group(1) if place_id else None,
           "name": name,
           "rating": float(rating.replace(",", ".")) if rating else None,
           "reviews": int(re.sub(r"[^\d]", "", reviews) or 0),
           "category": category,
           "url": page.url,
       }

   Field note: Selectors like fontDisplayLarge drift over time — wrap each field read in a try/except and log misses. A field that suddenly goes null across every listing is your early-warning that Maps changed its layout, not that every business lost its rating.

4. 4

   Hold one IP per pass with a sticky session

   A metro sweep is a multi-step flow — search, scroll, then open each place — so pin one exit for that pass with a session id and a short lifetime. Rotate to a fresh exit for the next metro to keep per-IP velocity low across the whole job.

   Username (sticky)Copy

   aethyn-XXXXX-country-us-city-chicago-session-metro042-lifetime-20

   Field note: Derive the session id from the metro (session-metro042) so retries of the same city reuse a coherent identity, while different cities never share one. It makes runs reproducible and easier to debug.

5. 5

   Collect reviews only with care

   Ratings and review counts are business facts and safe to store. Individual review text and author names are personal data — if you genuinely need them, collect only what your lawful basis covers, minimize what you retain, and honor deletion. For most ranking and lead-gen work, the aggregates are enough.

Best practices that keep scrapers reliable

• Pair a city-matched Elite IP with a /@lat,lng viewport for true local results
• Scroll the feed element to the end-of-list sentinel — don't stop at the first screen
• Dedupe on place_id, never on business name
• Read fields off role-based selectors and wrap each in try/except for layout drift
• Re-crawl on a cadence and store deltas; ratings move slowly, listings churn
• Treat ratings/counts as facts but review text/authors as personal data

Common mistakes that burn proxy budget

• Scrolling the window instead of the feed panel, capturing only the first dozen listings
• Stopping scroll before the sentinel and silently truncating each metro
• Deduping on name, merging 'Starbucks' locations into one record
• Querying a city with a datacenter IP and trusting the skewed listing set
• Hoarding review text with no lawful basis or retention limit
• Treating a stalled feed as the genuine end of results rather than a soft throttle

Common questions about this scraper problem