Aggregating Public Real-Estate Listing Data Across Zillow, Realtor.com, and Rightmove

What goes wrong when this scraper fails

Market-research, valuation, and property-search products all need the same public facts about a home: its list price, bed/bath count, square footage, location, how long it has been on the market, and its price-change history. None of the three major portals - Zillow (US), Realtor.com (US), and Rightmove (UK) - exposes a public, general-purpose API for this inventory, so the data has to come off the public search-results and detail pages. Each portal defends those pages differently and serves geographically distinct inventory, which turns a conceptually simple aggregation job into a per-site anti-bot and geo-routing problem.

Why this failure mode happens

Listing data is the portals' core commercial asset, so they invest heavily in blocking automated collection. Zillow fronts its pages with PerimeterX (now HUMAN), which fingerprints the TLS handshake (JA3), inspects header ordering and completeness, and watches interaction timing before deciding whether to serve a px-captcha - it is comfortably the most aggressive of the three. Realtor.com runs Akamai Bot Manager with sensor-data challenges, while Rightmove is lighter on active bot management but geofences hard to the UK and will localize or empty a response from a foreign IP. On top of that, all three localize by exit IP, so an out-of-country request returns the wrong regional context rather than an honest error.

Challenges that make this hard to automate

• Zillow's PerimeterX layer scores multiple signals together (JA3/TLS fingerprint, header order, timing, prior IP reputation), so improving one signal in isolation rarely lifts the block - it often takes both a residential exit and a realistic browser fingerprint.
• Geo-correctness is mandatory and silent when wrong: a US exit on Rightmove or a UK exit on Zillow returns localized or empty pages that parse to zero results without any error status.
• The useful fields live in embedded JSON that changes shape between page templates - Zillow's __NEXT_DATA__ payload and Realtor.com's model blob get restructured periodically, breaking brittle key paths.
• Soft blocks return HTTP 200 with a challenge or an empty listing array, so naive success-checks (status == 200) record garbage instead of retrying.
• Price history and days-on-market are frequently lazy-loaded or gated behind an extra XHR, so a single page fetch misses exactly the temporal fields AVM models care about most.

Approaches that usually fail

• Buying a licensed MLS/RETS or RESO Web API feed - authoritative and compliant, but expensive, access-gated to licensed brokerages, and regionally fragmented across hundreds of MLSs.
• Cheap datacenter proxies - almost universally burned on Zillow and Realtor.com, whose bot managers flag datacenter ASNs on the first request.
• Full headless-browser farms (Puppeteer/Playwright at scale) - they render everything but are slow, resource-heavy, and still get fingerprinted by PerimeterX unless carefully hardened.
• Off-the-shelf 'real estate scraper API' resellers - convenient but opaque about geo-routing and freshness, and you inherit their block rate and their ToS exposure with no control.

When residential proxies fix this — and when they cannot

Residential proxies solve the two hardest parts of this job at once: reputation and geography. Routing each request through a residential IP in the correct country (US for Zillow and Realtor.com, GB for Rightmove) means the portals see an ordinary consumer connection and serve the correct localized inventory, instead of flagging a datacenter ASN or redirecting a foreign visitor. Per-request rotation spreads velocity across many exits so no single IP accumulates the request pattern that PerimeterX and Akamai score against, and city-level targeting lets you request the exact metro a valuation model needs. When Zillow escalates to px-captcha anyway, moving that traffic to a cleaner, higher-trust pool is usually enough to clear it without solving challenges.

How Aethyn residential proxies help here

Aethyn's residential network is built for exactly this kind of geo-partitioned, reputation-sensitive collection. You point every request at proxy.aethyn.io, encode the country (and optionally city) into the username, and the Premium pool on port 2099 handles the routine load; when Zillow starts throwing PerimeterX challenges you switch that one site's traffic to the Elite pool on port 5499 without changing anything else in your pipeline. Sticky sessions keep a multi-step flow (search page then detail pages) on one exit when you need continuity, while the default per-request rotation keeps velocity low per IP.

• Country and city targeting encoded directly in the username (aethyn-XXXXX-country-us-city-chicago) so US listings come from US exits and UK listings from GB exits, with no separate geo config.
• Two quality tiers on one endpoint: Premium (port 2099) for Realtor.com and Rightmove, Elite (port 5499) to escalate Zillow when px-captcha appears - same credentials, just a different port.
• Per-request rotation by default so consecutive requests never stack on one IP, which is what keeps you under PerimeterX and Akamai velocity thresholds.
• Optional sticky sessions (-session-TOKEN-lifetime-10, up to 30 minutes) to hold one exit across a search-to-detail burst when a site ties results to a session.
• Large residential pool across US and UK metros so you can shard a metro-by-metro AVM refresh across many distinct exits.
• Standard HTTP proxy auth that drops into requests, Playwright, cURL, and any HTTP client with zero SDK lock-in.

How to implement this with residential proxies

1. 1

   Route each portal to the correct country exit

   Geo-correctness comes first, because every other signal is wasted if the exit country is wrong. Encode the country in the proxy username: US exits for Zillow and Realtor.com, GB exits for Rightmove. Default to Premium on port 2099. Rotate per request by omitting any session token. Build a small resolver so each target maps to its own proxy string, and never send a Rightmove request through a US IP or vice versa.

   pythonCopy

   import requests
   
   HOST = "proxy.aethyn.io"
   PREMIUM_PORT = 2099  # recommended tier for this guide
   
   def proxy_for(country_code):
       # country_code is a 2-letter lowercase ISO code, e.g. 'us' or 'gb'
       user = f"aethyn-XXXXX-country-{country_code}"
       url = f"http://{user}:PASSWORD@{HOST}:{PREMIUM_PORT}"
       return {"http": url, "https": url}
   
   TARGETS = {
       "zillow": "us",
       "realtor": "us",
       "rightmove": "gb",
   }
   
   resp = requests.get(
       "https://www.rightmove.co.uk/property-for-sale/find.html?searchLocation=London",
       proxies=proxy_for(TARGETS["rightmove"]),
       timeout=30,
   )
   print(resp.status_code, len(resp.content))

   Field note: Log the exit country the portal thinks you are in (Rightmove's currency, Zillow's default region) on the first request of every run. A silent geo mismatch is the single most common cause of a clean 200 that parses to zero listings.

2. 2

   Send a complete, realistic header set

   A bare User-Agent with no Accept-Language or Accept-Encoding is an obvious bot tell, and PerimeterX weights header completeness and order. Send a full, coherent header set that matches a real browser, and make Accept-Language consistent with the exit country - en-US for the US portals, en-GB for Rightmove. Keep the header ordering stable and browser-like rather than whatever your HTTP client emits by default.

   pythonCopy

   US_HEADERS = {
       "User-Agent": (
           "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
           "(KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36"
       ),
       "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8",
       "Accept-Language": "en-US,en;q=0.9",
       "Accept-Encoding": "gzip, deflate, br",
       "Upgrade-Insecure-Requests": "1",
       "Sec-Fetch-Dest": "document",
       "Sec-Fetch-Mode": "navigate",
       "Sec-Fetch-Site": "none",
   }
   
   UK_HEADERS = {**US_HEADERS, "Accept-Language": "en-GB,en;q=0.9"}
   
   r = requests.get(
       "https://www.realtor.com/realestateandhomes-search/Austin_TX",
       headers=US_HEADERS,
       proxies=proxy_for("us"),
       timeout=30,
   )
   print(r.status_code)

   Field note: Match Accept-Language to the exit: an en-US header arriving on Rightmove from a UK IP is an inconsistency that costs you reputation. Keep one header profile per country and never mix them.

3. 3

   Parse the embedded JSON state, not the rendered HTML

   The clean, typed fields live in the page's embedded state, not in the visual markup. Zillow ships a Next.js __NEXT_DATA__ script tag; Realtor.com embeds a large model blob. Extract and json.loads that payload to get list price, beds, baths, square footage, location, days-on-market, and price history as structured data - and only the public property facts. Do not extract agent or owner personal contact details even when they appear in the same blob.

   pythonCopy

   import json
   from bs4 import BeautifulSoup
   
   def extract_next_data(html):
       soup = BeautifulSoup(html, "html.parser")
       tag = soup.find("script", id="__NEXT_DATA__")
       if not tag or not tag.string:
           return None  # likely a challenge/soft-block page, not a listing
       return json.loads(tag.string)
   
   def public_fields(prop):
       # Keep only public, non-personal property facts
       return {
           "price": prop.get("price"),
           "beds": prop.get("bedrooms"),
           "baths": prop.get("bathrooms"),
           "sqft": prop.get("livingArea"),
           "zipcode": prop.get("zipcode"),
           "days_on_market": prop.get("daysOnZillow"),
           "price_history": [
               {"date": h.get("date"), "price": h.get("price"), "event": h.get("event")}
               for h in (prop.get("priceHistory") or [])
           ],
       }

   Field note: Pin your parser to the JSON key paths, not to CSS classes. Zillow and Realtor.com reshuffle rendered markup constantly, but the embedded state schema is far more stable - and when it does change, a KeyError is a much louder failure than a silently-missing HTML element.

4. 4

   Detect soft blocks and empty results before writing

   All three portals return HTTP 200 for challenges and empty pages, so status alone is meaningless. After parsing, assert that you actually got listings and that the challenge markers are absent. Treat a missing __NEXT_DATA__ tag, a px-captcha reference, or a zero-length results array as a block - never as a real empty page - and route it to a retry rather than to your database.

   pythonCopy

   class SoftBlock(Exception):
       pass
   
   PX_MARKERS = ("px-captcha", "/px/", "Access to this page has been denied")
   
   def assert_real_results(html, listings):
       if any(m in html for m in PX_MARKERS):
           raise SoftBlock("perimeterx_challenge")
       if not listings:
           # 200 OK with zero listings is almost never a genuine empty page
           raise SoftBlock("empty_result_set")
       return listings
   
   try:
       data = extract_next_data(r.text)
       listings = (data or {}).get("props", {}).get("searchResults", []) if data else []
       assert_real_results(r.text, listings)
       write_to_db(listings)
   except SoftBlock as e:
       schedule_retry(reason=str(e))

   Field note: Alert on your soft-block rate per portal, not just on hard errors. A creeping empty-result rate on Zillow is the earliest sign PerimeterX has started scoring your traffic, and it shows up days before you see outright 403s.

5. 5

   Escalate Zillow to the Elite pool when PerimeterX challenges

   Premium on port 2099 is enough for Realtor.com and Rightmove and for routine Zillow traffic. When your soft-block detector starts flagging px-captcha on Zillow, move only Zillow's traffic to the Elite pool on port 5499 - same credentials and username format, just a higher-trust exit pool. Keep everything else on 2099 so you are not paying Elite rates for sites that do not need it.

   pythonCopy

   ELITE_PORT = 5499  # escalation pool for Zillow px-captcha
   
   def zillow_proxy(elite=False):
       port = ELITE_PORT if elite else PREMIUM_PORT
       user = "aethyn-XXXXX-country-us"
       url = f"http://{user}:PASSWORD@{HOST}:{port}"
       return {"http": url, "https": url}
   
   def fetch_zillow(url, headers):
       r = requests.get(url, headers=headers, proxies=zillow_proxy(False), timeout=30)
       if any(m in r.text for m in PX_MARKERS):
           # escalate this request to the Elite pool and retry once
           r = requests.get(url, headers=headers, proxies=zillow_proxy(True), timeout=30)
       return r

   Field note: Escalate per-site, not globally. Only Zillow reliably justifies the Elite pool of these three; sending Rightmove through 5499 just burns cleaner IPs on a site that clears fine on 2099.

6. 6

   Use sticky sessions for search-to-detail bursts, rotate everywhere else

   Per-request rotation is the right default - it keeps velocity low per IP. But when you walk a search-results page and then fetch several detail pages that a portal ties to the same session, pin that short burst to one exit with a session token. Give it a lifetime just long enough for the burst (a few minutes, max 30), then let it expire so you return to fresh rotating exits. Add human-like jitter between requests inside the burst.

   pythonCopy

   import random, time
   
   def sticky_proxy(country, token, minutes=10):
       user = f"aethyn-XXXXX-country-{country}-session-{token}-lifetime-{minutes}"
       url = f"http://{user}:PASSWORD@{HOST}:{PREMIUM_PORT}"
       return {"http": url, "https": url}
   
   def crawl_search_then_details(search_url, detail_urls, headers):
       token = f"burst{random.randint(1000, 9999)}"
       px = sticky_proxy("us", token, minutes=10)
       out = [requests.get(search_url, headers=headers, proxies=px, timeout=30)]
       for u in detail_urls:
           time.sleep(random.uniform(1.5, 4.0))  # jitter, not a fixed cadence
           out.append(requests.get(u, headers=headers, proxies=px, timeout=30))
       return out

   Field note: Keep session lifetimes as short as the burst needs. A 30-minute sticky session dragged across hundreds of requests concentrates velocity on one IP - exactly the pattern reputation scoring is built to catch. Short bursts then rotate.

Best practices that keep scrapers reliable

• Collect only public, non-personal listing facts - price, beds/baths, sqft, location, days-on-market, price history. Do not harvest agent or owner names, phone numbers, or emails, respect each site's robots directives and Terms of Service, and consult counsel for your jurisdiction.
• Always match the exit country to the portal: US for Zillow and Realtor.com, GB for Rightmove, verified on the first request of every run.
• Assert a non-empty, challenge-free parse before every database write so soft blocks become retries instead of silent gaps in your AVM training data.
• Default to per-request rotation on Premium (2099); reserve sticky sessions for genuine multi-step bursts and keep their lifetimes short.
• Escalate to the Elite pool (5499) only for the specific site and specific requests that are being challenged - Zillow in practice - not across the whole pipeline.
• Parse the embedded JSON state and pin to key paths, so schema changes fail loudly rather than corrupting fields silently.
• Throttle with randomized jitter and cap velocity per exit; keep total volume reasonable and proportionate to genuine research need rather than maxing throughput.

Common mistakes that burn proxy budget

• Treating HTTP 200 as success - on all three portals a challenge or empty page returns 200, so status-only checks record garbage.
• Using a US exit for Rightmove (or a UK exit for Zillow) and getting silently localized or empty responses instead of the intended inventory.
• Sending a bare User-Agent with no Accept-Language or Sec-Fetch headers, which PerimeterX flags immediately on Zillow.
• Scraping rendered HTML by CSS class instead of the embedded JSON, so routine markup changes quietly break field extraction.
• Running every site through the Elite pool 'to be safe,' burning premium IPs on Realtor.com and Rightmove that clear fine on Premium.
• Holding one long sticky session across hundreds of requests, concentrating velocity on a single IP exactly where reputation scoring looks.

Common questions about this scraper problem