Scrape Public Real-Estate Listings: Zillow, Realtor.com, Rightmove

What goes wrong when this scraper fails

Market-research, valuation, and property-search products all need the same public facts about a home: its list price, bed/bath count, square footage, location, how long it has been on the market, and its price-change history. None of the three major portals - Zillow (US), Realtor.com (US), and Rightmove (UK) - exposes a public, general-purpose API for this inventory, so the data has to come off the public search-results and detail pages. Each portal defends those pages differently and serves geographically distinct inventory, which turns a conceptually simple aggregation job into a per-site anti-bot and geo-routing problem.

Why this failure mode happens

Listing data is the portals' core commercial asset, so they invest heavily in blocking automated collection. Zillow fronts its pages with PerimeterX (now HUMAN), which fingerprints the TLS handshake (JA3), inspects header ordering and completeness, and watches interaction timing before deciding whether to serve a px-captcha - it is comfortably the most aggressive of the three. Realtor.com runs Akamai Bot Manager with sensor-data challenges, while Rightmove is lighter on active bot management but geofences hard to the UK and will localize or empty a response from a foreign IP. On top of that, all three localize by exit IP, so an out-of-country request returns the wrong regional context rather than an honest error.

Challenges that make this hard to automate

Zillow's PerimeterX layer scores multiple signals together (JA3/TLS fingerprint, header order, timing, prior IP reputation), so improving one signal in isolation rarely lifts the block - it often takes both a residential exit and a realistic browser fingerprint.
Geo-correctness is mandatory and silent when wrong: a US exit on Rightmove or a UK exit on Zillow returns localized or empty pages that parse to zero results without any error status.
The useful fields live in embedded JSON that changes shape between page templates - Zillow's __NEXT_DATA__ payload and Realtor.com's model blob get restructured periodically, breaking brittle key paths.
Soft blocks return HTTP 200 with a challenge or an empty listing array, so naive success-checks (status == 200) record garbage instead of retrying.
Price history and days-on-market are frequently lazy-loaded or gated behind an extra XHR, so a single page fetch misses exactly the temporal fields AVM models care about most.

Approaches that usually fail

Buying a licensed MLS/RETS or RESO Web API feed - authoritative and compliant, but expensive, access-gated to licensed brokerages, and regionally fragmented across hundreds of MLSs.
Cheap datacenter proxies - almost universally burned on Zillow and Realtor.com, whose bot managers flag datacenter ASNs on the first request.
Full headless-browser farms (Puppeteer/Playwright at scale) - they render everything but are slow, resource-heavy, and still get fingerprinted by PerimeterX unless carefully hardened.
Off-the-shelf 'real estate scraper API' resellers - convenient but opaque about geo-routing and freshness, and you inherit their block rate and their ToS exposure with no control.

When residential proxies fix this — and when they cannot

Residential proxies solve the two hardest parts of this job at once: reputation and geography. Routing each request through a residential IP in the correct country (US for Zillow and Realtor.com, GB for Rightmove) means the portals see an ordinary consumer connection and serve the correct localized inventory, instead of flagging a datacenter ASN or redirecting a foreign visitor. Per-request rotation spreads velocity across many exits so no single IP accumulates the request pattern that PerimeterX and Akamai score against, and city-level targeting lets you request the exact metro a valuation model needs. When Zillow escalates to px-captcha anyway, moving that traffic to a cleaner, higher-trust pool is usually enough to clear it without solving challenges.

How Aethyn residential proxies help here

Aethyn's residential network is built for exactly this kind of geo-partitioned, reputation-sensitive collection. You point every request at proxy.aethyn.io, encode the country (and optionally city) into the username, and the Premium pool on port 2099 handles the routine load; when Zillow starts throwing PerimeterX challenges you switch that one site's traffic to the Elite pool on port 5499 without changing anything else in your pipeline. Sticky sessions keep a multi-step flow (search page then detail pages) on one exit when you need continuity, while the default per-request rotation keeps velocity low per IP.

Country and city targeting encoded directly in the username (aethyn-XXXXX-country-us-city-chicago) so US listings come from US exits and UK listings from GB exits, with no separate geo config.
Two quality tiers on one endpoint: Premium (port 2099) for Realtor.com and Rightmove, Elite (port 5499) to escalate Zillow when px-captcha appears - same credentials, just a different port.
Per-request rotation by default so consecutive requests never stack on one IP, which is what keeps you under PerimeterX and Akamai velocity thresholds.
Optional sticky sessions (-session-TOKEN-lifetime-10, up to 30 minutes) to hold one exit across a search-to-detail burst when a site ties results to a session.
Large residential pool across US and UK metros so you can shard a metro-by-metro AVM refresh across many distinct exits.
Standard HTTP proxy auth that drops into requests, Playwright, cURL, and any HTTP client with zero SDK lock-in.

How to implement this with residential proxies

1
Route each portal to the correct country exit
Geo-correctness comes first, because every other signal is wasted if the exit country is wrong. Encode the country in the proxy username: US exits for Zillow and Realtor.com, GB exits for Rightmove. Default to Premium on port 2099. Rotate per request by omitting any session token. Build a small resolver so each target maps to its own proxy string, and never send a Rightmove request through a US IP or vice versa.
python
```
import requests

HOST = "proxy.aethyn.io"
PREMIUM_PORT = 2099  # recommended tier for this guide

def proxy_for(country_code):
    # country_code is a 2-letter lowercase ISO code, e.g. 'us' or 'gb'
    user = f"aethyn-XXXXX-country-{country_code}"
    url = f"http://{user}:PASSWORD@{HOST}:{PREMIUM_PORT}"
    return {"http": url, "https": url}

TARGETS = {
    "zillow": "us",
    "realtor": "us",
    "rightmove": "gb",
}

resp = requests.get(
    "https://www.rightmove.co.uk/property-for-sale/find.html?searchLocation=London",
    proxies=proxy_for(TARGETS["rightmove"]),
    timeout=30,
)
print(resp.status_code, len(resp.content))
```
Field note: Log the exit country the portal thinks you are in (Rightmove's currency, Zillow's default region) on the first request of every run. A silent geo mismatch is the single most common cause of a clean 200 that parses to zero listings.

Send a complete, realistic header set

A bare User-Agent with no Accept-Language or Accept-Encoding is an obvious bot tell, and PerimeterX weights header completeness and order. Send a full, coherent header set that matches a real browser, and make Accept-Language consistent with the exit country - en-US for the US portals, en-GB for Rightmove. Keep the header ordering stable and browser-like rather than whatever your HTTP client emits by default.

python

US_HEADERS = {
    "User-Agent": (
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
        "(KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36"
    ),
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8",
    "Accept-Language": "en-US,en;q=0.9",
    "Accept-Encoding": "gzip, deflate, br",
    "Upgrade-Insecure-Requests": "1",
    "Sec-Fetch-Dest": "document",
    "Sec-Fetch-Mode": "navigate",
    "Sec-Fetch-Site": "none",
}

UK_HEADERS = {**US_HEADERS, "Accept-Language": "en-GB,en;q=0.9"}

r = requests.get(
    "https://www.realtor.com/realestateandhomes-search/Austin_TX",
    headers=US_HEADERS,
    proxies=proxy_for("us"),
    timeout=30,
)
print(r.status_code)

Field note: Match Accept-Language to the exit: an en-US header arriving on Rightmove from a UK IP is an inconsistency that costs you reputation. Keep one header profile per country and never mix them.

Parse the embedded JSON state, not the rendered HTML

The clean, typed fields live in the page's embedded state, not in the visual markup. Zillow ships a Next.js __NEXT_DATA__ script tag; Realtor.com embeds a large model blob. Extract and json.loads that payload to get list price, beds, baths, square footage, location, days-on-market, and price history as structured data - and only the public property facts. Do not extract agent or owner personal contact details even when they appear in the same blob.

python

import json
from bs4 import BeautifulSoup

def extract_next_data(html):
    soup = BeautifulSoup(html, "html.parser")
    tag = soup.find("script", id="__NEXT_DATA__")
    if not tag or not tag.string:
        return None  # likely a challenge/soft-block page, not a listing
    return json.loads(tag.string)

def public_fields(prop):
    # Keep only public, non-personal property facts
    return {
        "price": prop.get("price"),
        "beds": prop.get("bedrooms"),
        "baths": prop.get("bathrooms"),
        "sqft": prop.get("livingArea"),
        "zipcode": prop.get("zipcode"),
        "days_on_market": prop.get("daysOnZillow"),
        "price_history": [
            {"date": h.get("date"), "price": h.get("price"), "event": h.get("event")}
            for h in (prop.get("priceHistory") or [])
        ],
    }

Field note: Pin your parser to the JSON key paths, not to CSS classes. Zillow and Realtor.com reshuffle rendered markup constantly, but the embedded state schema is far more stable - and when it does change, a KeyError is a much louder failure than a silently-missing HTML element.

4
Detect soft blocks and empty results before writing
All three portals return HTTP 200 for challenges and empty pages, so status alone is meaningless. After parsing, assert that you actually got listings and that the challenge markers are absent. Treat a missing __NEXT_DATA__ tag, a px-captcha reference, or a zero-length results array as a block - never as a real empty page - and route it to a retry rather than to your database.
python
```
class SoftBlock(Exception):
    pass

PX_MARKERS = ("px-captcha", "/px/", "Access to this page has been denied")

def assert_real_results(html, listings):
    if any(m in html for m in PX_MARKERS):
        raise SoftBlock("perimeterx_challenge")
    if not listings:
        # 200 OK with zero listings is almost never a genuine empty page
        raise SoftBlock("empty_result_set")
    return listings

try:
    data = extract_next_data(r.text)
    listings = (data or {}).get("props", {}).get("searchResults", []) if data else []
    assert_real_results(r.text, listings)
    write_to_db(listings)
except SoftBlock as e:
    schedule_retry(reason=str(e))
```
Field note: Alert on your soft-block rate per portal, not just on hard errors. A creeping empty-result rate on Zillow is the earliest sign PerimeterX has started scoring your traffic, and it shows up days before you see outright 403s.
5
Escalate Zillow to the Elite pool when PerimeterX challenges
Premium on port 2099 is enough for Realtor.com and Rightmove and for routine Zillow traffic. When your soft-block detector starts flagging px-captcha on Zillow, move only Zillow's traffic to the Elite pool on port 5499 - same credentials and username format, just a higher-trust exit pool. Keep everything else on 2099 so you are not paying Elite rates for sites that do not need it.
python
```
ELITE_PORT = 5499  # escalation pool for Zillow px-captcha

def zillow_proxy(elite=False):
    port = ELITE_PORT if elite else PREMIUM_PORT
    user = "aethyn-XXXXX-country-us"
    url = f"http://{user}:PASSWORD@{HOST}:{port}"
    return {"http": url, "https": url}

def fetch_zillow(url, headers):
    r = requests.get(url, headers=headers, proxies=zillow_proxy(False), timeout=30)
    if any(m in r.text for m in PX_MARKERS):
        # escalate this request to the Elite pool and retry once
        r = requests.get(url, headers=headers, proxies=zillow_proxy(True), timeout=30)
    return r
```
Field note: Escalate per-site, not globally. Only Zillow reliably justifies the Elite pool of these three; sending Rightmove through 5499 just burns cleaner IPs on a site that clears fine on 2099.
6
Use sticky sessions for search-to-detail bursts, rotate everywhere else
Per-request rotation is the right default - it keeps velocity low per IP. But when you walk a search-results page and then fetch several detail pages that a portal ties to the same session, pin that short burst to one exit with a session token. Give it a lifetime just long enough for the burst (a few minutes, max 30), then let it expire so you return to fresh rotating exits. Add human-like jitter between requests inside the burst.
python
```
import random, time

def sticky_proxy(country, token, minutes=10):
    user = f"aethyn-XXXXX-country-{country}-session-{token}-lifetime-{minutes}"
    url = f"http://{user}:PASSWORD@{HOST}:{PREMIUM_PORT}"
    return {"http": url, "https": url}

def crawl_search_then_details(search_url, detail_urls, headers):
    token = f"burst{random.randint(1000, 9999)}"
    px = sticky_proxy("us", token, minutes=10)
    out = [requests.get(search_url, headers=headers, proxies=px, timeout=30)]
    for u in detail_urls:
        time.sleep(random.uniform(1.5, 4.0))  # jitter, not a fixed cadence
        out.append(requests.get(u, headers=headers, proxies=px, timeout=30))
    return out
```
Field note: Keep session lifetimes as short as the burst needs. A 30-minute sticky session dragged across hundreds of requests concentrates velocity on one IP - exactly the pattern reputation scoring is built to catch. Short bursts then rotate.

Best practices that keep scrapers reliable

Collect only public, non-personal listing facts - price, beds/baths, sqft, location, days-on-market, price history. Do not harvest agent or owner names, phone numbers, or emails, respect each site's robots directives and Terms of Service, and consult counsel for your jurisdiction.
Always match the exit country to the portal: US for Zillow and Realtor.com, GB for Rightmove, verified on the first request of every run.
Assert a non-empty, challenge-free parse before every database write so soft blocks become retries instead of silent gaps in your AVM training data.
Default to per-request rotation on Premium (2099); reserve sticky sessions for genuine multi-step bursts and keep their lifetimes short.
Escalate to the Elite pool (5499) only for the specific site and specific requests that are being challenged - Zillow in practice - not across the whole pipeline.
Parse the embedded JSON state and pin to key paths, so schema changes fail loudly rather than corrupting fields silently.
Throttle with randomized jitter and cap velocity per exit; keep total volume reasonable and proportionate to genuine research need rather than maxing throughput.

Common mistakes that burn proxy budget

Treating HTTP 200 as success - on all three portals a challenge or empty page returns 200, so status-only checks record garbage.
Using a US exit for Rightmove (or a UK exit for Zillow) and getting silently localized or empty responses instead of the intended inventory.
Sending a bare User-Agent with no Accept-Language or Sec-Fetch headers, which PerimeterX flags immediately on Zillow.
Scraping rendered HTML by CSS class instead of the embedded JSON, so routine markup changes quietly break field extraction.
Running every site through the Elite pool 'to be safe,' burning premium IPs on Realtor.com and Rightmove that clear fine on Premium.
Holding one long sticky session across hundreds of requests, concentrating velocity on a single IP exactly where reputation scoring looks.

Common questions about this scraper problem

Is scraping real-estate listing data from these portals legal?

Collecting public, non-personal factual listing data (price, beds/baths, sqft, location, days-on-market, price history) is generally lower-risk than collecting personal data, but it is still governed by each site's Terms of Service and by local law. Do not harvest agent or owner personal contact details, keep your request volume reasonable and proportionate, respect robots directives, and consult counsel for your specific jurisdiction and use case before running at scale.

Why is Zillow so much harder than Realtor.com or Rightmove?

Zillow fronts its pages with PerimeterX (HUMAN), which scores the TLS/JA3 fingerprint, header completeness and order, and interaction timing together - not just request count. That combined scoring means a stock headless-browser fingerprint gets challenged even at low volume, which is why Zillow is the one site here that regularly justifies the Elite pool.

Which port should I use for each portal?

Default everything to Premium on port 2099 - it clears Realtor.com and Rightmove comfortably and handles routine Zillow traffic. When your soft-block detector starts flagging px-captcha on Zillow, move only Zillow's requests to the Elite pool on port 5499. The username format and credentials are identical; you only change the port.

Do I need a headless browser, or is requests enough?

For Realtor.com and Rightmove, plain HTTP requests with a complete header set and a residential exit are usually enough, and parsing the embedded JSON avoids rendering entirely. Zillow's price-history and some detail fields can be lazy-loaded, so a hardened Playwright session through the proxy is sometimes needed - but try the JSON-state approach first, it is faster and more stable.

How do I get accurate US-metro or UK-region data?

Encode the country and optionally the city in the proxy username, for example aethyn-XXXXX-country-us-city-chicago for US metros or aethyn-XXXXX-country-gb for UK-wide Rightmove coverage. The portals localize by exit IP, so a city-targeted residential exit returns the correct regional inventory that a generic exit would not.

Why do I get a 200 response but zero listings?

That is almost always a soft block or a geo mismatch, not a genuinely empty page. Check for a missing __NEXT_DATA__ tag or px-captcha markers (a challenge), and confirm your exit country matches the portal. Treat any zero-length result array as a block to retry, never as data to store.

How should I handle price history and days-on-market specifically?

These temporal fields are the most valuable for AVM work and the most often missed, because they are frequently in a separate XHR or a lazy-loaded section rather than the initial HTML. Pull them from the embedded JSON state where present, and where a portal gates them behind an extra request, fetch that endpoint within the same sticky session so it is tied to the same exit.

How do I keep my block rate low over time?

Rotate per request by default so velocity stays low per IP, send a complete country-consistent header set, add randomized jitter between requests, and cap concurrency per exit. Monitor your soft-block rate per portal - a creeping empty-result rate on Zillow is the earliest warning that PerimeterX has started scoring you, well before you see hard 403s.

Can I resell or redistribute the data I collect?

That is a separate and stricter question from collection. Redistribution of listing data can implicate the portals' Terms of Service, database rights, and copyright, and licensed feeds (MLS/RESO) carry their own redistribution terms. Keep collected data for internal research/modeling unless you have confirmed redistribution rights with counsel.

Should I use sticky sessions or per-request rotation?

Per-request rotation is the default and the safer choice for reputation, because no single IP accumulates a scoreable pattern. Use a short sticky session (a few minutes, up to 30) only when a multi-step flow - a search page followed by several detail pages - must stay on one exit, then let it expire and return to rotation.

What goes wrong when this scraper fails

Why this failure mode happens

Challenges that make this hard to automate

Zillow's PerimeterX layer scores multiple signals together (JA3/TLS fingerprint, header order, timing, prior IP reputation), so improving one signal in isolation rarely lifts the block - it often takes both a residential exit and a realistic browser fingerprint.
Geo-correctness is mandatory and silent when wrong: a US exit on Rightmove or a UK exit on Zillow returns localized or empty pages that parse to zero results without any error status.
The useful fields live in embedded JSON that changes shape between page templates - Zillow's __NEXT_DATA__ payload and Realtor.com's model blob get restructured periodically, breaking brittle key paths.
Soft blocks return HTTP 200 with a challenge or an empty listing array, so naive success-checks (status == 200) record garbage instead of retrying.
Price history and days-on-market are frequently lazy-loaded or gated behind an extra XHR, so a single page fetch misses exactly the temporal fields AVM models care about most.

Approaches that usually fail

Buying a licensed MLS/RETS or RESO Web API feed - authoritative and compliant, but expensive, access-gated to licensed brokerages, and regionally fragmented across hundreds of MLSs.
Cheap datacenter proxies - almost universally burned on Zillow and Realtor.com, whose bot managers flag datacenter ASNs on the first request.
Full headless-browser farms (Puppeteer/Playwright at scale) - they render everything but are slow, resource-heavy, and still get fingerprinted by PerimeterX unless carefully hardened.
Off-the-shelf 'real estate scraper API' resellers - convenient but opaque about geo-routing and freshness, and you inherit their block rate and their ToS exposure with no control.

When residential proxies fix this — and when they cannot

How Aethyn residential proxies help here

Country and city targeting encoded directly in the username (aethyn-XXXXX-country-us-city-chicago) so US listings come from US exits and UK listings from GB exits, with no separate geo config.
Two quality tiers on one endpoint: Premium (port 2099) for Realtor.com and Rightmove, Elite (port 5499) to escalate Zillow when px-captcha appears - same credentials, just a different port.
Per-request rotation by default so consecutive requests never stack on one IP, which is what keeps you under PerimeterX and Akamai velocity thresholds.
Optional sticky sessions (-session-TOKEN-lifetime-10, up to 30 minutes) to hold one exit across a search-to-detail burst when a site ties results to a session.
Large residential pool across US and UK metros so you can shard a metro-by-metro AVM refresh across many distinct exits.
Standard HTTP proxy auth that drops into requests, Playwright, cURL, and any HTTP client with zero SDK lock-in.

How to implement this with residential proxies

1
Route each portal to the correct country exit
Geo-correctness comes first, because every other signal is wasted if the exit country is wrong. Encode the country in the proxy username: US exits for Zillow and Realtor.com, GB exits for Rightmove. Default to Premium on port 2099. Rotate per request by omitting any session token. Build a small resolver so each target maps to its own proxy string, and never send a Rightmove request through a US IP or vice versa.
python
```
import requests

HOST = "proxy.aethyn.io"
PREMIUM_PORT = 2099  # recommended tier for this guide

def proxy_for(country_code):
    # country_code is a 2-letter lowercase ISO code, e.g. 'us' or 'gb'
    user = f"aethyn-XXXXX-country-{country_code}"
    url = f"http://{user}:PASSWORD@{HOST}:{PREMIUM_PORT}"
    return {"http": url, "https": url}

TARGETS = {
    "zillow": "us",
    "realtor": "us",
    "rightmove": "gb",
}

resp = requests.get(
    "https://www.rightmove.co.uk/property-for-sale/find.html?searchLocation=London",
    proxies=proxy_for(TARGETS["rightmove"]),
    timeout=30,
)
print(resp.status_code, len(resp.content))
```
Field note: Log the exit country the portal thinks you are in (Rightmove's currency, Zillow's default region) on the first request of every run. A silent geo mismatch is the single most common cause of a clean 200 that parses to zero listings.

Send a complete, realistic header set

python

US_HEADERS = {
    "User-Agent": (
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
        "(KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36"
    ),
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8",
    "Accept-Language": "en-US,en;q=0.9",
    "Accept-Encoding": "gzip, deflate, br",
    "Upgrade-Insecure-Requests": "1",
    "Sec-Fetch-Dest": "document",
    "Sec-Fetch-Mode": "navigate",
    "Sec-Fetch-Site": "none",
}

UK_HEADERS = {**US_HEADERS, "Accept-Language": "en-GB,en;q=0.9"}

r = requests.get(
    "https://www.realtor.com/realestateandhomes-search/Austin_TX",
    headers=US_HEADERS,
    proxies=proxy_for("us"),
    timeout=30,
)
print(r.status_code)

Parse the embedded JSON state, not the rendered HTML

python

import json
from bs4 import BeautifulSoup

def extract_next_data(html):
    soup = BeautifulSoup(html, "html.parser")
    tag = soup.find("script", id="__NEXT_DATA__")
    if not tag or not tag.string:
        return None  # likely a challenge/soft-block page, not a listing
    return json.loads(tag.string)

def public_fields(prop):
    # Keep only public, non-personal property facts
    return {
        "price": prop.get("price"),
        "beds": prop.get("bedrooms"),
        "baths": prop.get("bathrooms"),
        "sqft": prop.get("livingArea"),
        "zipcode": prop.get("zipcode"),
        "days_on_market": prop.get("daysOnZillow"),
        "price_history": [
            {"date": h.get("date"), "price": h.get("price"), "event": h.get("event")}
            for h in (prop.get("priceHistory") or [])
        ],
    }

4
Detect soft blocks and empty results before writing
All three portals return HTTP 200 for challenges and empty pages, so status alone is meaningless. After parsing, assert that you actually got listings and that the challenge markers are absent. Treat a missing __NEXT_DATA__ tag, a px-captcha reference, or a zero-length results array as a block - never as a real empty page - and route it to a retry rather than to your database.
python
```
class SoftBlock(Exception):
    pass

PX_MARKERS = ("px-captcha", "/px/", "Access to this page has been denied")

def assert_real_results(html, listings):
    if any(m in html for m in PX_MARKERS):
        raise SoftBlock("perimeterx_challenge")
    if not listings:
        # 200 OK with zero listings is almost never a genuine empty page
        raise SoftBlock("empty_result_set")
    return listings

try:
    data = extract_next_data(r.text)
    listings = (data or {}).get("props", {}).get("searchResults", []) if data else []
    assert_real_results(r.text, listings)
    write_to_db(listings)
except SoftBlock as e:
    schedule_retry(reason=str(e))
```
Field note: Alert on your soft-block rate per portal, not just on hard errors. A creeping empty-result rate on Zillow is the earliest sign PerimeterX has started scoring your traffic, and it shows up days before you see outright 403s.
5
Escalate Zillow to the Elite pool when PerimeterX challenges
Premium on port 2099 is enough for Realtor.com and Rightmove and for routine Zillow traffic. When your soft-block detector starts flagging px-captcha on Zillow, move only Zillow's traffic to the Elite pool on port 5499 - same credentials and username format, just a higher-trust exit pool. Keep everything else on 2099 so you are not paying Elite rates for sites that do not need it.
python
```
ELITE_PORT = 5499  # escalation pool for Zillow px-captcha

def zillow_proxy(elite=False):
    port = ELITE_PORT if elite else PREMIUM_PORT
    user = "aethyn-XXXXX-country-us"
    url = f"http://{user}:PASSWORD@{HOST}:{port}"
    return {"http": url, "https": url}

def fetch_zillow(url, headers):
    r = requests.get(url, headers=headers, proxies=zillow_proxy(False), timeout=30)
    if any(m in r.text for m in PX_MARKERS):
        # escalate this request to the Elite pool and retry once
        r = requests.get(url, headers=headers, proxies=zillow_proxy(True), timeout=30)
    return r
```
Field note: Escalate per-site, not globally. Only Zillow reliably justifies the Elite pool of these three; sending Rightmove through 5499 just burns cleaner IPs on a site that clears fine on 2099.
6
Use sticky sessions for search-to-detail bursts, rotate everywhere else
Per-request rotation is the right default - it keeps velocity low per IP. But when you walk a search-results page and then fetch several detail pages that a portal ties to the same session, pin that short burst to one exit with a session token. Give it a lifetime just long enough for the burst (a few minutes, max 30), then let it expire so you return to fresh rotating exits. Add human-like jitter between requests inside the burst.
python
```
import random, time

def sticky_proxy(country, token, minutes=10):
    user = f"aethyn-XXXXX-country-{country}-session-{token}-lifetime-{minutes}"
    url = f"http://{user}:PASSWORD@{HOST}:{PREMIUM_PORT}"
    return {"http": url, "https": url}

def crawl_search_then_details(search_url, detail_urls, headers):
    token = f"burst{random.randint(1000, 9999)}"
    px = sticky_proxy("us", token, minutes=10)
    out = [requests.get(search_url, headers=headers, proxies=px, timeout=30)]
    for u in detail_urls:
        time.sleep(random.uniform(1.5, 4.0))  # jitter, not a fixed cadence
        out.append(requests.get(u, headers=headers, proxies=px, timeout=30))
    return out
```
Field note: Keep session lifetimes as short as the burst needs. A 30-minute sticky session dragged across hundreds of requests concentrates velocity on one IP - exactly the pattern reputation scoring is built to catch. Short bursts then rotate.

Best practices that keep scrapers reliable

Collect only public, non-personal listing facts - price, beds/baths, sqft, location, days-on-market, price history. Do not harvest agent or owner names, phone numbers, or emails, respect each site's robots directives and Terms of Service, and consult counsel for your jurisdiction.
Always match the exit country to the portal: US for Zillow and Realtor.com, GB for Rightmove, verified on the first request of every run.
Assert a non-empty, challenge-free parse before every database write so soft blocks become retries instead of silent gaps in your AVM training data.
Default to per-request rotation on Premium (2099); reserve sticky sessions for genuine multi-step bursts and keep their lifetimes short.
Escalate to the Elite pool (5499) only for the specific site and specific requests that are being challenged - Zillow in practice - not across the whole pipeline.
Parse the embedded JSON state and pin to key paths, so schema changes fail loudly rather than corrupting fields silently.
Throttle with randomized jitter and cap velocity per exit; keep total volume reasonable and proportionate to genuine research need rather than maxing throughput.

Common mistakes that burn proxy budget

Treating HTTP 200 as success - on all three portals a challenge or empty page returns 200, so status-only checks record garbage.
Using a US exit for Rightmove (or a UK exit for Zillow) and getting silently localized or empty responses instead of the intended inventory.
Sending a bare User-Agent with no Accept-Language or Sec-Fetch headers, which PerimeterX flags immediately on Zillow.
Scraping rendered HTML by CSS class instead of the embedded JSON, so routine markup changes quietly break field extraction.
Running every site through the Elite pool 'to be safe,' burning premium IPs on Realtor.com and Rightmove that clear fine on Premium.
Holding one long sticky session across hundreds of requests, concentrating velocity on a single IP exactly where reputation scoring looks.

Common questions about this scraper problem

Is scraping real-estate listing data from these portals legal?

Why is Zillow so much harder than Realtor.com or Rightmove?

Which port should I use for each portal?

Do I need a headless browser, or is requests enough?

How do I get accurate US-metro or UK-region data?

Why do I get a 200 response but zero listings?

How should I handle price history and days-on-market specifically?

How do I keep my block rate low over time?

Can I resell or redistribute the data I collect?

Should I use sticky sessions or per-request rotation?

Aggregating Public Real-Estate Listing Data Across Zillow, Realtor.com, and Rightmove

Key takeaways

On this page

What goes wrong when this scraper fails

Challenges that make this hard to automate

How Aethyn residential proxies help here

How to implement this with residential proxies

Route each portal to the correct country exit

Send a complete, realistic header set

Parse the embedded JSON state, not the rendered HTML

Detect soft blocks and empty results before writing

Escalate Zillow to the Elite pool when PerimeterX challenges

Use sticky sessions for search-to-detail bursts, rotate everywhere else

Best practices that keep scrapers reliable

Common questions about this scraper problem

Solve this with Aethyn residential proxies

Aggregating Public Real-Estate Listing Data Across Zillow, Realtor.com, and Rightmove

Key takeaways

On this page

What goes wrong when this scraper fails

Challenges that make this hard to automate

How Aethyn residential proxies help here

How to implement this with residential proxies

Route each portal to the correct country exit

Send a complete, realistic header set

Parse the embedded JSON state, not the rendered HTML

Detect soft blocks and empty results before writing

Escalate Zillow to the Elite pool when PerimeterX challenges

Use sticky sessions for search-to-detail bursts, rotate everywhere else

Best practices that keep scrapers reliable

Common questions about this scraper problem

Solve this with Aethyn residential proxies

Key takeaways

On this page

What goes wrong when this scraper fails

Challenges that make this hard to automate

How Aethyn residential proxies help here

How to implement this with residential proxies

Route each portal to the correct country exit

Send a complete, realistic header set

Parse the embedded JSON state, not the rendered HTML

Detect soft blocks and empty results before writing

Escalate Zillow to the Elite pool when PerimeterX challenges

Use sticky sessions for search-to-detail bursts, rotate everywhere else

Best practices that keep scrapers reliable

Common questions about this scraper problem

Related guides & resources

Solve this with Aethyn residential proxies

Key takeaways

On this page

What goes wrong when this scraper fails

Challenges that make this hard to automate

How Aethyn residential proxies help here

How to implement this with residential proxies

Route each portal to the correct country exit

Send a complete, realistic header set

Parse the embedded JSON state, not the rendered HTML

Detect soft blocks and empty results before writing

Escalate Zillow to the Elite pool when PerimeterX challenges

Use sticky sessions for search-to-detail bursts, rotate everywhere else

Best practices that keep scrapers reliable

Common questions about this scraper problem

Related guides & resources

Solve this with Aethyn residential proxies