How to Collect Local Business Data

What goes wrong when this scraper fails

Agencies, data vendors, and sales teams need a comprehensive, accurate list of businesses in an area — names, addresses, phones, categories, hours, websites, and ratings. Two structural problems make this hard. First, every local source caps results per search, so a single 'restaurants in Chicago' query returns a fraction of reality. Second, any one source is incomplete and partly stale, so a trustworthy dataset has to merge several and resolve conflicts — which only works if records are matched correctly. And because results are localized, querying from the wrong place returns the wrong businesses.

Why this failure mode happens

Local platforms (maps and directories) localize results to the searcher's location and cap how many they return per query to keep responses small and relevant. Each source also has its own coverage gaps, category taxonomy, and update cadence, so no single one is authoritative. Building a complete picture therefore means many geographically-scoped searches across multiple sources, then entity resolution to collapse duplicates — none of which works from a single mislocated IP.

Challenges that make this hard to automate

• Per-search result caps that hide most businesses in an area
• Localized results that depend on the searcher's location
• Incomplete, partially-stale coverage in any single source
• Matching the same business across sources with messy NAP data
• Reviewer and personal data mixed into business listings

Approaches that usually fail

• Scraping one source once — incomplete and quickly outdated
• A single broad search per city — capped, missing most of the long tail
• Buying a static list — stale the day it's delivered
• Manual directory compilation — accurate but impossibly slow at scale

When residential proxies fix this — and when they cannot

Residential IPs in the target area return the local results an actual resident sees and carry the reputation to query maps and directories reliably. A rotating pool lets you run the hundreds of category × grid-cell searches needed for full coverage across multiple sources without any single IP hitting velocity limits, so the dataset is both complete and current.

How Aethyn residential proxies help here

Local collection is geography-first and runs many searches across sources. Aethyn handles both location precision and volume through the username.

• City-level targeting so results match the exact local market you're mapping
• Elite high-trust IPs for map sources that defend aggressively
• A large rotating pool for hundreds of grid × category searches
• Sticky sessions for multi-step listing-detail flows
• Per-byte metering so recurring refreshes of the dataset stay predictable

How to implement this with residential proxies

1. 1

   Grid the area and sweep categories

   Because each search caps results, divide your target area into a grid of cells and run each business category within each cell. This converts one capped query into many smaller, complete ones. Size cells to density — tighter grids downtown, looser in rural areas.

   Python (geo grid)Copy

   def grid(sw, ne, step_km=2.0):
       import math
       (s, w), (no, e) = sw, ne
       dlat = step_km / 111.0
       lat = s
       while lat < no:
           dlng = step_km / (111.0 * max(0.1, math.cos(math.radians(lat))))
           lng = w
           while lng < e:
               yield (lat, lng)
               lng += dlng
           lat += dlat
   
   # For each cell + category, issue one residential-proxied search,
   # then dedupe results across overlapping cells.

   Field note: If a single cell still returns a capped page for a category, subdivide just that cell. Dense downtown blocks need a finer grid than suburbs — adapt instead of using one global cell size.

2. 2

   Geo-target the exit IP to the area

   Local results depend on where the searcher appears to be. Set a city-targeted exit IP in the area you're collecting so you get the businesses a resident would see, not a national or wrong-region view.

   cURLCopy

   curl -x "http://aethyn-XXXXX-country-us-city-chicago:PASSWORD@proxy.aethyn.io:5499" \
     "https://www.google.com/maps/search/coffee+shops/@41.8781,-87.6298,14z"

   Field note: Keep the IP geography and the search coordinates in the same place. A Chicago search from a Dallas IP can return blended or down-ranked local results — consistency is what makes coverage trustworthy.

3. 3

   Normalize NAP as the matching key

   Name, Address, Phone is the identity of a local business. Normalize each aggressively — lowercase, strip punctuation, standardize the address with a parser, and reduce phones to E.164 digits — so the same business from two sources collapses to one entity.

   Python (NAP normalization)Copy

   import re
   
   def norm_phone(p):
       return "+" + re.sub(r"\D", "", p or "")
   
   def norm_name(n):
       n = (n or "").lower()
       n = re.sub(r"\b(inc|llc|ltd|co|corp)\b", "", n)
       return re.sub(r"[^a-z0-9]+", " ", n).strip()
   
   def nap_key(rec):
       return (norm_name(rec["name"]), norm_phone(rec["phone"]))

   Field note: Phone is the strongest single matcher — it's far less ambiguous than a name or a loosely-formatted address. When phone matches but names differ slightly, it's almost always the same business under a brand variation.

4. 4

   Reconcile multiple sources into one record

   Merge records sharing a NAP key across sources, and on conflicts prefer the more authoritative or more recent field (e.g. hours from the maps source, website from the official listing). Keep source provenance per field so you can audit where each value came from.

   Field note: Use a fuzzy fallback (token-set ratio on name + address proximity) for records that don't match on phone. Set a high threshold and route borderline pairs to a review queue rather than auto-merging — a bad merge is worse than a duplicate.

5. 5

   Refresh on a cadence and track changes

   Local data drifts — businesses close, move, and rebrand. Re-sweep on a schedule through rotating residential IPs, diff against the prior dataset, and flag closures and moves. Collect business facts (NAP, hours, category, rating count) and avoid reviewer identities.

Best practices that keep scrapers reliable

• Grid the area and run category searches per cell to beat caps
• City-target the exit IP to match the local market
• Normalize NAP aggressively and match primarily on phone
• Reconcile sources with per-field provenance and a review queue
• Refresh on a cadence and diff to catch closures/moves
• Collect business facts; minimize reviewer and personal data

Common mistakes that burn proxy budget

• Running one broad city search and capturing only the capped top slice
• Collecting from a single source and shipping an incomplete dataset
• Auto-merging on fuzzy name matches and corrupting records
• Searching a city from a wrong-region IP and getting blended results
• Treating reviewer names/text as fair-game business data
• Never refreshing, so the dataset rots within months

Common questions about this scraper problem