Scrape Craigslist & Mercari Listings at Scale (Proxy Guide)

What goes wrong when this scraper fails

Market-research teams want a normalized view of what things sell for and how fast: Craigslist for local, cash-economy classifieds across metros, and Mercari for nationwide C2C resale pricing. Neither offers a public research API, so the data has to come from the same public pages and endpoints a browser hits. Craigslist serves fast, minimal HTML but aggressively blocks IPs that behave non-humanly, and Mercari hides its catalog behind a JSON API that is individually rate-limited and fingerprinted. Aggregating both into one price-and-trend dataset means handling two completely different anti-automation postures in a single pipeline.

Why this failure mode happens

Both sites are magnets for spam, scraping-for-resale, and fraud, so they invest heavily in keeping automated traffic out. Craigslist keeps its stack deliberately simple - light HTML, minimal JavaScript - which lets it spend its defense budget on IP reputation scoring and per-IP velocity limits that return 403 and 429 within seconds. Mercari, being a modern SPA, funnels everything through an internal JSON endpoint (search, item detail, price history) that carries its own bot-detection: a client-generated DPoP request-signing token, header and TLS fingerprinting, and per-operation throttles. The result is that naive scraping fails differently on each - Craigslist hard-blocks the IP, Mercari quietly returns empty or error-wrapped JSON with a 200 status.

Challenges that make this hard to automate

Craigslist's geo-partitioning means one logical crawl is really hundreds of subdomain crawls (sfbay, chicago, newyork...), each with its own listing pages and pagination, and each best fetched from a matching-region exit.
Craigslist's velocity limiter trips on burst patterns, not just totals - ten fast requests from one IP will 403 where the same ten spaced out and rotated would not.
Mercari gives no useful HTML; you must reverse the JSON API shape and reproduce its DPoP-signed request, and those endpoints change field names and signing parameters without notice, silently breaking parsers.
Soft blocks look like success: Craigslist may serve a near-empty results page and Mercari a 200 with an empty item array, so a length check on the parsed output is the only reliable block signal.
Price and condition normalization across two schemas - Craigslist's free-text prices and Mercari's structured price/condition/shipping fields - requires defensive parsing before anything is comparable.

Approaches that usually fail

Running the crawler from a single server IP or a small datacenter range - works for a few hundred requests, then Craigslist reputation-flags the range and every metro starts returning 403.
A fixed pool of cheap datacenter proxies - Craigslist scores whole ASNs, so once one IP in the block is burned the neighbors inherit the bad reputation and 429s cascade.
Hammering Mercari's JSON endpoint from Python requests with a copied User-Agent - misses the header/TLS fingerprint and DPoP signing expectations, so responses come back empty or challenged.
Full headless-browser automation for both sites - unnecessary and slow for Craigslist's static HTML, and for Mercari it is easier and cheaper to speak to the JSON API directly than to render the SPA.

When residential proxies fix this — and when they cannot

Residential proxies give each request a clean, real consumer IP with good reputation, which is exactly the signal Craigslist's reputation scorer rewards and datacenter ranges fail. Per-request rotation spreads a metro crawl across many exits so no single IP accumulates the velocity that trips Craigslist's limiter, and country/city targeting lets you fetch chicago.craigslist.org from a Chicago-region exit so the geography is coherent. For Mercari, rotating residential IPs paired with a complete, browser-realistic header set (including a valid DPoP token) keeps individual JSON operations under their per-IP throttle and out of the fingerprint-based challenge path.

How Aethyn residential proxies help here

Aethyn's Premium residential pool is built for exactly this shape of job: broad, geo-aware crawling across many exits with clean reputation. You compose the behavior entirely in the proxy username, so a Craigslist metro crawl and a Mercari JSON pull differ only by which country and city you encode - no separate infrastructure per site.

Premium residential exits on port 2099 with the reputation profile Craigslist's scorer treats as human, avoiding the ASN-wide flagging that burns datacenter pools.
Country and city targeting in the username (aethyn-XXXXX-country-us-city-chicago) so each Craigslist subdomain is fetched from a geographically coherent exit.
Per-request rotation by default - just omit the session token - so a metro's pagination spreads across many IPs and never stacks velocity on one.
Optional sticky sessions (-session-TOKEN-lifetime-MINUTES, up to 30) for the rare case where a short paginated Mercari burst must stay on one exit.
A single endpoint, proxy.aethyn.io, for both sites - your pipeline changes only the username, not its transport or connection code.
Large enough exit diversity to keep both Craigslist velocity limits and Mercari per-operation throttles comfortably below their trip thresholds at research volume.

How to implement this with residential proxies

Build the Craigslist subdomain plan and route each metro through a matching-region exit

Craigslist has no global search - the catalog is partitioned into per-metro subdomains, so start by enumerating the metros you care about and their subdomain hosts. For each metro, route the request through an exit in the same country and, where it matters, the same city, so the exit geography matches the subdomain. Rotate on every request by omitting any session token; consecutive fetches then never stack on one IP, which is what keeps you under Craigslist's velocity limiter.

python

import requests

# account id and password are literal placeholders - substitute your own
PROXY_USER = "aethyn-XXXXX-country-us-city-chicago"
PROXY_PASS = "PASSWORD"
PROXY_URL = "http://" + PROXY_USER + ":" + PROXY_PASS + "@proxy.aethyn.io:2099"
proxies = {"http": PROXY_URL, "https": PROXY_URL}

HEADERS = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
                  "(KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36",
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
    "Accept-Language": "en-US,en;q=0.9",
    "Referer": "https://chicago.craigslist.org/",
}

# public search results only - item, price, location, category
url = "https://chicago.craigslist.org/search/sss"  # sss = general for-sale
resp = requests.get(url, params={"query": "road bike"}, headers=HEADERS,
                    proxies=proxies, timeout=20)
print(resp.status_code, len(resp.text))

Field note: Keep a fixed map of metro subdomain to proxy city (chicago.craigslist.org -> -city-chicago) and never fetch a subdomain from a mismatched region. A New York exit hitting the Seattle subdomain is a coherence tell that adds to your reputation risk for zero benefit.

Parse Craigslist result HTML defensively and treat empty pages as soft blocks

Craigslist HTML is light and stable but not guaranteed - class names shift and a throttled response can return the page shell with zero listings. Parse into item, price, location, and category only, and assert a non-empty result set before you trust the page. A 200 with no parsed rows on a query you know has inventory is a soft block, not an empty market.

python

from bs4 import BeautifulSoup

def parse_craigslist(html):
    soup = BeautifulSoup(html, "html.parser")
    rows = soup.select("li.cl-static-search-result, li.result-row")
    items = []
    for r in rows:
        title_el = r.select_one(".title, .result-title")
        price_el = r.select_one(".price, .result-price")
        loc_el = r.select_one(".location, .result-hood")
        if not title_el:
            continue
        items.append({
            "item": title_el.get_text(strip=True),
            "price": price_el.get_text(strip=True) if price_el else None,
            "location": loc_el.get_text(strip=True).strip("() ") if loc_el else None,
            "category": "for-sale",
        })
    return items

listings = parse_craigslist(resp.text)
if not listings:
    raise RuntimeError("empty result set - likely a soft block, rotate and retry")
print(len(listings), "listings")

Field note: Never persist a zero-row parse as a real result. Wire empty output straight into your retry-with-fresh-exit path, because silently storing soft blocks is how a price dashboard quietly reports a collapsing market that never actually moved.

3
Handle Craigslist 403/429 with backoff and forced exit rotation
When Craigslist decides an IP is too loud it returns 403 or 429 fast, sometimes within one request on a bad exit. Because rotation is per-request by default, the simplest recovery is to retry the same URL - a new exit is drawn automatically. Add exponential backoff so you are not immediately re-loading a flagged pattern, and cap retries so one poisoned metro does not stall the whole crawl.
python
```
import time

def fetch_craigslist(url, params, headers, proxies, max_retries=4):
    delay = 2.0
    for attempt in range(max_retries):
        r = requests.get(url, params=params, headers=headers,
                         proxies=proxies, timeout=20)
        if r.status_code == 200 and "result" in r.text:
            return r
        if r.status_code in (403, 429):
            # per-request rotation gives a fresh exit on the next call
            time.sleep(delay)
            delay *= 2
            continue
        r.raise_for_status()
    raise RuntimeError("exhausted retries on " + url)
```
Field note: Do not raise the retry ceiling to brute-force through blocks - if a metro 429s past four fresh exits, you are querying too fast overall. Lower your per-metro concurrency instead; velocity, not attempt count, is what Craigslist scores.

Talk to Mercari's JSON search endpoint directly with a complete, signed header set

Mercari's public catalog is served by an internal JSON API, so skip HTML rendering and request the search endpoint directly through a rotating US residential exit. The difference between a full response and an empty one is almost entirely the header set - a bare User-Agent with no Accept-Language, no X-Platform hint, and no DPoP token is an instant bot tell. Send a realistic, complete set, request JSON explicitly, and mirror the DPoP header your browser produces for the same call.

python

import requests

PROXY_USER = "aethyn-XXXXX-country-us"
PROXY_PASS = "PASSWORD"
PROXY_URL = "http://" + PROXY_USER + ":" + PROXY_PASS + "@proxy.aethyn.io:2099"
proxies = {"http": PROXY_URL, "https": PROXY_URL}

JSON_HEADERS = {
    "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) "
                  "AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36",
    "Accept": "application/json, text/plain, */*",
    "Accept-Language": "en-US,en;q=0.9",
    "Content-Type": "application/json",
    "X-Platform": "web",
    "Origin": "https://www.mercari.com",
    "Referer": "https://www.mercari.com/search/",
    # Mercari signs each call with a client-generated DPoP token; capture the
    # exact value from your browser network tab and pass it through here.
    "DPoP": "REPLACE_WITH_OBSERVED_DPOP_TOKEN",
}

# public search - item, price, condition, category only
payload = {"keyword": "nintendo switch", "pageSize": 60, "page": 0}
resp = requests.post("https://www.mercari.com/v1/api",
                     json=payload, headers=JSON_HEADERS,
                     proxies=proxies, timeout=20)
print(resp.status_code, resp.headers.get("content-type"))

Field note: Capture the exact request shape from your browser's network tab first - the DPoP header, X-Platform, and any operation name - and mirror it faithfully. Mercari's endpoints move field names and signing parameters between releases, so pin your parser to observed responses and re-verify whenever your empty-rate climbs.

Validate Mercari JSON, extract non-personal fields, and detect empty-body soft blocks

Mercari's soft block is a 200 status wrapping an empty item array or an error object rather than an HTTP error code. Parse the JSON, confirm the item list is actually populated, and pull only the public, non-personal fields - item name, price, condition, category, and general location. If the body is empty on a query you know has stock, treat it as a block and rotate.

python

def parse_mercari(resp):
    if resp.status_code != 200:
        return None  # trigger retry / rotation
    try:
        data = resp.json()
    except ValueError:
        return None  # non-JSON body = challenge page, rotate
    items = data.get("items") or data.get("data", {}).get("items") or []
    if not items:
        return None  # 200 with empty body = soft block
    out = []
    for it in items:
        out.append({
            "item": it.get("name"),
            "price": it.get("price"),
            "condition": it.get("itemCondition", {}).get("name"),
            "category": it.get("categoryName"),
            # do NOT collect seller name, id, or any contact field
        })
    return out

rows = parse_mercari(resp)
if rows is None:
    raise RuntimeError("empty or challenged Mercari response - rotate exit")

Field note: Explicitly drop seller identity fields at the parse boundary, not later in the pipeline. Filtering personal data at the point of extraction means it never lands in your store, which is both the responsible default and far easier to defend than after-the-fact deletion.

6
Normalize both schemas and pace the aggregate crawl under both throttles
Once you have Craigslist and Mercari rows, normalize them into one price-and-trend schema: coerce free-text Craigslist prices to numeric, carry Mercari's structured condition and shipping through, and stamp every row with source and collection time. Pace the whole job so each site stays comfortably under its limits - modest per-metro concurrency for Craigslist velocity, modest per-operation rate for Mercari - rather than running both flat out.
python
```
import re, time, datetime

def to_cents(price_text):
    if price_text is None:
        return None
    m = re.search(r"[\d,]+(?:\.\d{2})?", str(price_text))
    if not m:
        return None
    return int(round(float(m.group().replace(",", "")) * 100))

def normalize(row, source):
    return {
        "source": source,
        "item": row.get("item"),
        "price_cents": to_cents(row.get("price")),
        "condition": row.get("condition"),
        "location": row.get("location"),
        "category": row.get("category"),
        "collected_at": datetime.datetime.utcnow().isoformat() + "Z",
    }

# gentle pacing between requests keeps you under both sites' throttles
for row in craigslist_rows:
    record = normalize(row, "craigslist")
    time.sleep(1.5)  # per-request spacing on top of concurrency limits
```
Field note: Log source, exit region, and empty-rate per site to a rolling metric. A rising Craigslist empty-rate means back off velocity; a rising Mercari empty-rate usually means the JSON schema shifted under you - the two failure modes need opposite responses, so keep their signals separate.

Best practices that keep scrapers reliable

Collect only public, non-personal listing fields - item, price, location, category, condition. Never harvest seller names, emails, phone numbers, or handles, never post or message, and respect each site's robots.txt and Terms of Service; consult counsel for your jurisdiction.
Match exit geography to the target: Craigslist city subdomains fetched from same-region exits (aethyn-XXXXX-country-us-city-chicago), Mercari from US exits, so the request geography is always coherent.
Default to per-request rotation for breadth crawling and reserve sticky sessions only for short paginated bursts that genuinely need one exit.
Assert a non-empty parsed result before storing anything - a 200 with zero rows is a soft block on both sites, not an empty market.
Pace by velocity, not by total count: cap per-metro concurrency for Craigslist and per-operation rate for Mercari, and lower concurrency before raising retry ceilings.
Send a complete, realistic header set on every request - Accept-Language, Referer, and the JSON/X-Platform/DPoP headers Mercari expects - because a bare User-Agent is a classic bot tell on both surfaces.
Pin Mercari parsers to observed JSON responses and re-verify whenever the empty-rate climbs, since the internal endpoint renames fields and rotates signing parameters between releases without notice.

Common mistakes that burn proxy budget

Crawling from a single server or datacenter range - Craigslist reputation-scores the ASN and 403s the whole block once one IP gets loud.
Fetching a Craigslist metro subdomain from a mismatched-region exit, adding a geographic-coherence tell for no benefit.
Storing zero-row responses as real empty results, quietly corrupting price and trend dashboards with soft blocks.
Sending a minimal User-Agent with no Accept-Language, X-Platform, or DPoP token, which reads as a bot and returns empty Mercari bodies.
Brute-forcing past 429s by cranking retries and concurrency instead of lowering request velocity - the exact signal Craigslist blocks on.
Harvesting seller contact details or attempting to message/post - outside the responsible public-data scope and a Terms of Service violation on both sites.

Common questions about this scraper problem

Is scraping Craigslist and Mercari listings legal?

Collecting public, non-personal listing data - item, price, location, category - for research is broadly lower-risk than harvesting personal data, but it is still governed by each site's Terms of Service. Keep volume reasonable, take only non-personal fields, never post or interact, respect robots.txt, and consult counsel for your specific jurisdiction and use case before scaling.

Why does Craigslist block me after only a handful of requests?

Craigslist scores IP reputation and per-IP velocity, not just total request count. A datacenter IP starts with poor reputation and gets a fast 403, and any IP - even a clean one - that bursts requests quickly trips the velocity limiter. Rotating clean residential exits per request and slowing your pace fixes both.

Do I need a headless browser for either site?

No. Craigslist serves light static HTML you can parse directly with requests and BeautifulSoup, and Mercari's data lives in a JSON API you can call directly - rendering the SPA is slower and unnecessary. Skip the browser unless a specific endpoint starts requiring client-side execution you cannot replicate, such as regenerating a fresh DPoP token.

Why does Mercari return a 200 with no items?

That is a soft block. Mercari wraps throttled or fingerprint-flagged requests in a 200 status with an empty item array or an error object rather than a clean HTTP error. Detect it by asserting the item list is populated, and rotate to a fresh exit with a complete, DPoP-signed header set when it is empty.

Should I use sticky sessions or per-request rotation?

Per-request rotation is the right default for both sites when crawling breadth - it keeps velocity off any single IP. Use a sticky session (-session-TOKEN-lifetime-MINUTES, up to 30 minutes) only for a short paginated burst that must stay on one exit, then drop back to rotation.

How do I match proxy geography to Craigslist's city subdomains?

Keep a fixed map from subdomain to proxy city and encode it in the username - chicago.craigslist.org through aethyn-XXXXX-country-us-city-chicago. The exit region should always match the metro you are querying so the request geography is coherent, which is one less anomaly for the reputation scorer.

Which Aethyn tier and port should I use?

Use the Premium tier on port 2099. Its residential exits carry the reputation profile Craigslist's scorer treats as human and give enough exit diversity to stay under Mercari's per-operation throttles at research volume. The full URL looks like http://aethyn-XXXXX-country-us:PASSWORD@proxy.aethyn.io:2099.

How do I keep price data comparable across the two sites?

Normalize at the parse boundary: coerce Craigslist's free-text prices to numeric cents, carry Mercari's structured price, condition, and shipping through, and stamp every row with source and collection time. Defensive parsing before storage is what makes cross-site benchmarking meaningful.

My Mercari parser suddenly returns nothing - what changed?

Most often the JSON schema shifted - Mercari renames fields and rotates signing parameters between releases without notice. Distinguish this from a block by checking whether the raw body is populated but your field paths miss; if so, re-capture the request in the browser network tab, refresh the DPoP token, and re-pin your parser. A rising empty-rate on Mercari specifically usually means schema drift, not throttling.

How much volume is safe to run?

There is no fixed number - safety is about velocity and reputation, not a daily total. Keep per-metro concurrency modest for Craigslist and per-operation rate modest for Mercari, watch your empty-rate as the real feedback signal, and back off the moment it climbs rather than pushing through blocks.

What goes wrong when this scraper fails

Why this failure mode happens

Challenges that make this hard to automate

Craigslist's geo-partitioning means one logical crawl is really hundreds of subdomain crawls (sfbay, chicago, newyork...), each with its own listing pages and pagination, and each best fetched from a matching-region exit.
Craigslist's velocity limiter trips on burst patterns, not just totals - ten fast requests from one IP will 403 where the same ten spaced out and rotated would not.
Mercari gives no useful HTML; you must reverse the JSON API shape and reproduce its DPoP-signed request, and those endpoints change field names and signing parameters without notice, silently breaking parsers.
Soft blocks look like success: Craigslist may serve a near-empty results page and Mercari a 200 with an empty item array, so a length check on the parsed output is the only reliable block signal.
Price and condition normalization across two schemas - Craigslist's free-text prices and Mercari's structured price/condition/shipping fields - requires defensive parsing before anything is comparable.

Approaches that usually fail

Running the crawler from a single server IP or a small datacenter range - works for a few hundred requests, then Craigslist reputation-flags the range and every metro starts returning 403.
A fixed pool of cheap datacenter proxies - Craigslist scores whole ASNs, so once one IP in the block is burned the neighbors inherit the bad reputation and 429s cascade.
Hammering Mercari's JSON endpoint from Python requests with a copied User-Agent - misses the header/TLS fingerprint and DPoP signing expectations, so responses come back empty or challenged.
Full headless-browser automation for both sites - unnecessary and slow for Craigslist's static HTML, and for Mercari it is easier and cheaper to speak to the JSON API directly than to render the SPA.

When residential proxies fix this — and when they cannot

How Aethyn residential proxies help here

Premium residential exits on port 2099 with the reputation profile Craigslist's scorer treats as human, avoiding the ASN-wide flagging that burns datacenter pools.
Country and city targeting in the username (aethyn-XXXXX-country-us-city-chicago) so each Craigslist subdomain is fetched from a geographically coherent exit.
Per-request rotation by default - just omit the session token - so a metro's pagination spreads across many IPs and never stacks velocity on one.
Optional sticky sessions (-session-TOKEN-lifetime-MINUTES, up to 30) for the rare case where a short paginated Mercari burst must stay on one exit.
A single endpoint, proxy.aethyn.io, for both sites - your pipeline changes only the username, not its transport or connection code.
Large enough exit diversity to keep both Craigslist velocity limits and Mercari per-operation throttles comfortably below their trip thresholds at research volume.

How to implement this with residential proxies

Build the Craigslist subdomain plan and route each metro through a matching-region exit

python

import requests

# account id and password are literal placeholders - substitute your own
PROXY_USER = "aethyn-XXXXX-country-us-city-chicago"
PROXY_PASS = "PASSWORD"
PROXY_URL = "http://" + PROXY_USER + ":" + PROXY_PASS + "@proxy.aethyn.io:2099"
proxies = {"http": PROXY_URL, "https": PROXY_URL}

HEADERS = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
                  "(KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36",
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
    "Accept-Language": "en-US,en;q=0.9",
    "Referer": "https://chicago.craigslist.org/",
}

# public search results only - item, price, location, category
url = "https://chicago.craigslist.org/search/sss"  # sss = general for-sale
resp = requests.get(url, params={"query": "road bike"}, headers=HEADERS,
                    proxies=proxies, timeout=20)
print(resp.status_code, len(resp.text))

Parse Craigslist result HTML defensively and treat empty pages as soft blocks

python

from bs4 import BeautifulSoup

def parse_craigslist(html):
    soup = BeautifulSoup(html, "html.parser")
    rows = soup.select("li.cl-static-search-result, li.result-row")
    items = []
    for r in rows:
        title_el = r.select_one(".title, .result-title")
        price_el = r.select_one(".price, .result-price")
        loc_el = r.select_one(".location, .result-hood")
        if not title_el:
            continue
        items.append({
            "item": title_el.get_text(strip=True),
            "price": price_el.get_text(strip=True) if price_el else None,
            "location": loc_el.get_text(strip=True).strip("() ") if loc_el else None,
            "category": "for-sale",
        })
    return items

listings = parse_craigslist(resp.text)
if not listings:
    raise RuntimeError("empty result set - likely a soft block, rotate and retry")
print(len(listings), "listings")

3
Handle Craigslist 403/429 with backoff and forced exit rotation
When Craigslist decides an IP is too loud it returns 403 or 429 fast, sometimes within one request on a bad exit. Because rotation is per-request by default, the simplest recovery is to retry the same URL - a new exit is drawn automatically. Add exponential backoff so you are not immediately re-loading a flagged pattern, and cap retries so one poisoned metro does not stall the whole crawl.
python
```
import time

def fetch_craigslist(url, params, headers, proxies, max_retries=4):
    delay = 2.0
    for attempt in range(max_retries):
        r = requests.get(url, params=params, headers=headers,
                         proxies=proxies, timeout=20)
        if r.status_code == 200 and "result" in r.text:
            return r
        if r.status_code in (403, 429):
            # per-request rotation gives a fresh exit on the next call
            time.sleep(delay)
            delay *= 2
            continue
        r.raise_for_status()
    raise RuntimeError("exhausted retries on " + url)
```
Field note: Do not raise the retry ceiling to brute-force through blocks - if a metro 429s past four fresh exits, you are querying too fast overall. Lower your per-metro concurrency instead; velocity, not attempt count, is what Craigslist scores.

Talk to Mercari's JSON search endpoint directly with a complete, signed header set

python

import requests

PROXY_USER = "aethyn-XXXXX-country-us"
PROXY_PASS = "PASSWORD"
PROXY_URL = "http://" + PROXY_USER + ":" + PROXY_PASS + "@proxy.aethyn.io:2099"
proxies = {"http": PROXY_URL, "https": PROXY_URL}

JSON_HEADERS = {
    "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) "
                  "AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36",
    "Accept": "application/json, text/plain, */*",
    "Accept-Language": "en-US,en;q=0.9",
    "Content-Type": "application/json",
    "X-Platform": "web",
    "Origin": "https://www.mercari.com",
    "Referer": "https://www.mercari.com/search/",
    # Mercari signs each call with a client-generated DPoP token; capture the
    # exact value from your browser network tab and pass it through here.
    "DPoP": "REPLACE_WITH_OBSERVED_DPOP_TOKEN",
}

# public search - item, price, condition, category only
payload = {"keyword": "nintendo switch", "pageSize": 60, "page": 0}
resp = requests.post("https://www.mercari.com/v1/api",
                     json=payload, headers=JSON_HEADERS,
                     proxies=proxies, timeout=20)
print(resp.status_code, resp.headers.get("content-type"))

Validate Mercari JSON, extract non-personal fields, and detect empty-body soft blocks

python

def parse_mercari(resp):
    if resp.status_code != 200:
        return None  # trigger retry / rotation
    try:
        data = resp.json()
    except ValueError:
        return None  # non-JSON body = challenge page, rotate
    items = data.get("items") or data.get("data", {}).get("items") or []
    if not items:
        return None  # 200 with empty body = soft block
    out = []
    for it in items:
        out.append({
            "item": it.get("name"),
            "price": it.get("price"),
            "condition": it.get("itemCondition", {}).get("name"),
            "category": it.get("categoryName"),
            # do NOT collect seller name, id, or any contact field
        })
    return out

rows = parse_mercari(resp)
if rows is None:
    raise RuntimeError("empty or challenged Mercari response - rotate exit")

6
Normalize both schemas and pace the aggregate crawl under both throttles
Once you have Craigslist and Mercari rows, normalize them into one price-and-trend schema: coerce free-text Craigslist prices to numeric, carry Mercari's structured condition and shipping through, and stamp every row with source and collection time. Pace the whole job so each site stays comfortably under its limits - modest per-metro concurrency for Craigslist velocity, modest per-operation rate for Mercari - rather than running both flat out.
python
```
import re, time, datetime

def to_cents(price_text):
    if price_text is None:
        return None
    m = re.search(r"[\d,]+(?:\.\d{2})?", str(price_text))
    if not m:
        return None
    return int(round(float(m.group().replace(",", "")) * 100))

def normalize(row, source):
    return {
        "source": source,
        "item": row.get("item"),
        "price_cents": to_cents(row.get("price")),
        "condition": row.get("condition"),
        "location": row.get("location"),
        "category": row.get("category"),
        "collected_at": datetime.datetime.utcnow().isoformat() + "Z",
    }

# gentle pacing between requests keeps you under both sites' throttles
for row in craigslist_rows:
    record = normalize(row, "craigslist")
    time.sleep(1.5)  # per-request spacing on top of concurrency limits
```
Field note: Log source, exit region, and empty-rate per site to a rolling metric. A rising Craigslist empty-rate means back off velocity; a rising Mercari empty-rate usually means the JSON schema shifted under you - the two failure modes need opposite responses, so keep their signals separate.

Best practices that keep scrapers reliable

Collect only public, non-personal listing fields - item, price, location, category, condition. Never harvest seller names, emails, phone numbers, or handles, never post or message, and respect each site's robots.txt and Terms of Service; consult counsel for your jurisdiction.
Match exit geography to the target: Craigslist city subdomains fetched from same-region exits (aethyn-XXXXX-country-us-city-chicago), Mercari from US exits, so the request geography is always coherent.
Default to per-request rotation for breadth crawling and reserve sticky sessions only for short paginated bursts that genuinely need one exit.
Assert a non-empty parsed result before storing anything - a 200 with zero rows is a soft block on both sites, not an empty market.
Pace by velocity, not by total count: cap per-metro concurrency for Craigslist and per-operation rate for Mercari, and lower concurrency before raising retry ceilings.
Send a complete, realistic header set on every request - Accept-Language, Referer, and the JSON/X-Platform/DPoP headers Mercari expects - because a bare User-Agent is a classic bot tell on both surfaces.
Pin Mercari parsers to observed JSON responses and re-verify whenever the empty-rate climbs, since the internal endpoint renames fields and rotates signing parameters between releases without notice.

Common mistakes that burn proxy budget

Crawling from a single server or datacenter range - Craigslist reputation-scores the ASN and 403s the whole block once one IP gets loud.
Fetching a Craigslist metro subdomain from a mismatched-region exit, adding a geographic-coherence tell for no benefit.
Storing zero-row responses as real empty results, quietly corrupting price and trend dashboards with soft blocks.
Sending a minimal User-Agent with no Accept-Language, X-Platform, or DPoP token, which reads as a bot and returns empty Mercari bodies.
Brute-forcing past 429s by cranking retries and concurrency instead of lowering request velocity - the exact signal Craigslist blocks on.
Harvesting seller contact details or attempting to message/post - outside the responsible public-data scope and a Terms of Service violation on both sites.

Common questions about this scraper problem

Is scraping Craigslist and Mercari listings legal?

Why does Craigslist block me after only a handful of requests?

Do I need a headless browser for either site?

Why does Mercari return a 200 with no items?

Should I use sticky sessions or per-request rotation?

How do I match proxy geography to Craigslist's city subdomains?

Which Aethyn tier and port should I use?

How do I keep price data comparable across the two sites?

My Mercari parser suddenly returns nothing - what changed?

How much volume is safe to run?

Aggregating Public Craigslist and Mercari Listings for Price and Trend Research

Key takeaways

On this page

What goes wrong when this scraper fails

Challenges that make this hard to automate

How Aethyn residential proxies help here

How to implement this with residential proxies

Build the Craigslist subdomain plan and route each metro through a matching-region exit

Parse Craigslist result HTML defensively and treat empty pages as soft blocks

Handle Craigslist 403/429 with backoff and forced exit rotation

Talk to Mercari's JSON search endpoint directly with a complete, signed header set

Validate Mercari JSON, extract non-personal fields, and detect empty-body soft blocks

Normalize both schemas and pace the aggregate crawl under both throttles

Best practices that keep scrapers reliable

Common questions about this scraper problem

Solve this with Aethyn residential proxies

Aggregating Public Craigslist and Mercari Listings for Price and Trend Research

Key takeaways

On this page

What goes wrong when this scraper fails

Challenges that make this hard to automate

How Aethyn residential proxies help here

How to implement this with residential proxies

Build the Craigslist subdomain plan and route each metro through a matching-region exit

Parse Craigslist result HTML defensively and treat empty pages as soft blocks

Handle Craigslist 403/429 with backoff and forced exit rotation

Talk to Mercari's JSON search endpoint directly with a complete, signed header set

Validate Mercari JSON, extract non-personal fields, and detect empty-body soft blocks

Normalize both schemas and pace the aggregate crawl under both throttles

Best practices that keep scrapers reliable

Common questions about this scraper problem

Solve this with Aethyn residential proxies

Key takeaways

On this page

What goes wrong when this scraper fails

Challenges that make this hard to automate

How Aethyn residential proxies help here

How to implement this with residential proxies

Build the Craigslist subdomain plan and route each metro through a matching-region exit

Parse Craigslist result HTML defensively and treat empty pages as soft blocks

Handle Craigslist 403/429 with backoff and forced exit rotation

Talk to Mercari's JSON search endpoint directly with a complete, signed header set

Validate Mercari JSON, extract non-personal fields, and detect empty-body soft blocks

Normalize both schemas and pace the aggregate crawl under both throttles

Best practices that keep scrapers reliable

Common questions about this scraper problem

Related guides & resources

Solve this with Aethyn residential proxies

Key takeaways

On this page

What goes wrong when this scraper fails

Challenges that make this hard to automate

How Aethyn residential proxies help here

How to implement this with residential proxies

Build the Craigslist subdomain plan and route each metro through a matching-region exit

Parse Craigslist result HTML defensively and treat empty pages as soft blocks

Handle Craigslist 403/429 with backoff and forced exit rotation

Talk to Mercari's JSON search endpoint directly with a complete, signed header set

Validate Mercari JSON, extract non-personal fields, and detect empty-body soft blocks

Normalize both schemas and pace the aggregate crawl under both throttles

Best practices that keep scrapers reliable

Common questions about this scraper problem

Related guides & resources

Solve this with Aethyn residential proxies