Aggregating Public Craigslist and Mercari Listings for Price and Trend Research

What goes wrong when this scraper fails

Market-research teams want a normalized view of what things sell for and how fast: Craigslist for local, cash-economy classifieds across metros, and Mercari for nationwide C2C resale pricing. Neither offers a public research API, so the data has to come from the same public pages and endpoints a browser hits. Craigslist serves fast, minimal HTML but aggressively blocks IPs that behave non-humanly, and Mercari hides its catalog behind a JSON API that is individually rate-limited and fingerprinted. Aggregating both into one price-and-trend dataset means handling two completely different anti-automation postures in a single pipeline.

Why this failure mode happens

Both sites are magnets for spam, scraping-for-resale, and fraud, so they invest heavily in keeping automated traffic out. Craigslist keeps its stack deliberately simple - light HTML, minimal JavaScript - which lets it spend its defense budget on IP reputation scoring and per-IP velocity limits that return 403 and 429 within seconds. Mercari, being a modern SPA, funnels everything through an internal JSON endpoint (search, item detail, price history) that carries its own bot-detection: a client-generated DPoP request-signing token, header and TLS fingerprinting, and per-operation throttles. The result is that naive scraping fails differently on each - Craigslist hard-blocks the IP, Mercari quietly returns empty or error-wrapped JSON with a 200 status.

Challenges that make this hard to automate

• Craigslist's geo-partitioning means one logical crawl is really hundreds of subdomain crawls (sfbay, chicago, newyork...), each with its own listing pages and pagination, and each best fetched from a matching-region exit.
• Craigslist's velocity limiter trips on burst patterns, not just totals - ten fast requests from one IP will 403 where the same ten spaced out and rotated would not.
• Mercari gives no useful HTML; you must reverse the JSON API shape and reproduce its DPoP-signed request, and those endpoints change field names and signing parameters without notice, silently breaking parsers.
• Soft blocks look like success: Craigslist may serve a near-empty results page and Mercari a 200 with an empty item array, so a length check on the parsed output is the only reliable block signal.
• Price and condition normalization across two schemas - Craigslist's free-text prices and Mercari's structured price/condition/shipping fields - requires defensive parsing before anything is comparable.

Approaches that usually fail

• Running the crawler from a single server IP or a small datacenter range - works for a few hundred requests, then Craigslist reputation-flags the range and every metro starts returning 403.
• A fixed pool of cheap datacenter proxies - Craigslist scores whole ASNs, so once one IP in the block is burned the neighbors inherit the bad reputation and 429s cascade.
• Hammering Mercari's JSON endpoint from Python requests with a copied User-Agent - misses the header/TLS fingerprint and DPoP signing expectations, so responses come back empty or challenged.
• Full headless-browser automation for both sites - unnecessary and slow for Craigslist's static HTML, and for Mercari it is easier and cheaper to speak to the JSON API directly than to render the SPA.

When residential proxies fix this — and when they cannot

Residential proxies give each request a clean, real consumer IP with good reputation, which is exactly the signal Craigslist's reputation scorer rewards and datacenter ranges fail. Per-request rotation spreads a metro crawl across many exits so no single IP accumulates the velocity that trips Craigslist's limiter, and country/city targeting lets you fetch chicago.craigslist.org from a Chicago-region exit so the geography is coherent. For Mercari, rotating residential IPs paired with a complete, browser-realistic header set (including a valid DPoP token) keeps individual JSON operations under their per-IP throttle and out of the fingerprint-based challenge path.

How Aethyn residential proxies help here

Aethyn's Premium residential pool is built for exactly this shape of job: broad, geo-aware crawling across many exits with clean reputation. You compose the behavior entirely in the proxy username, so a Craigslist metro crawl and a Mercari JSON pull differ only by which country and city you encode - no separate infrastructure per site.

• Premium residential exits on port 2099 with the reputation profile Craigslist's scorer treats as human, avoiding the ASN-wide flagging that burns datacenter pools.
• Country and city targeting in the username (aethyn-XXXXX-country-us-city-chicago) so each Craigslist subdomain is fetched from a geographically coherent exit.
• Per-request rotation by default - just omit the session token - so a metro's pagination spreads across many IPs and never stacks velocity on one.
• Optional sticky sessions (-session-TOKEN-lifetime-MINUTES, up to 30) for the rare case where a short paginated Mercari burst must stay on one exit.
• A single endpoint, proxy.aethyn.io, for both sites - your pipeline changes only the username, not its transport or connection code.
• Large enough exit diversity to keep both Craigslist velocity limits and Mercari per-operation throttles comfortably below their trip thresholds at research volume.

How to implement this with residential proxies

1. 1

   Build the Craigslist subdomain plan and route each metro through a matching-region exit

   Craigslist has no global search - the catalog is partitioned into per-metro subdomains, so start by enumerating the metros you care about and their subdomain hosts. For each metro, route the request through an exit in the same country and, where it matters, the same city, so the exit geography matches the subdomain. Rotate on every request by omitting any session token; consecutive fetches then never stack on one IP, which is what keeps you under Craigslist's velocity limiter.

   pythonCopy

   import requests
   
   # account id and password are literal placeholders - substitute your own
   PROXY_USER = "aethyn-XXXXX-country-us-city-chicago"
   PROXY_PASS = "PASSWORD"
   PROXY_URL = "http://" + PROXY_USER + ":" + PROXY_PASS + "@proxy.aethyn.io:2099"
   proxies = {"http": PROXY_URL, "https": PROXY_URL}
   
   HEADERS = {
       "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
                     "(KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36",
       "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
       "Accept-Language": "en-US,en;q=0.9",
       "Referer": "https://chicago.craigslist.org/",
   }
   
   # public search results only - item, price, location, category
   url = "https://chicago.craigslist.org/search/sss"  # sss = general for-sale
   resp = requests.get(url, params={"query": "road bike"}, headers=HEADERS,
                       proxies=proxies, timeout=20)
   print(resp.status_code, len(resp.text))

   Field note: Keep a fixed map of metro subdomain to proxy city (chicago.craigslist.org -> -city-chicago) and never fetch a subdomain from a mismatched region. A New York exit hitting the Seattle subdomain is a coherence tell that adds to your reputation risk for zero benefit.

2. 2

   Parse Craigslist result HTML defensively and treat empty pages as soft blocks

   Craigslist HTML is light and stable but not guaranteed - class names shift and a throttled response can return the page shell with zero listings. Parse into item, price, location, and category only, and assert a non-empty result set before you trust the page. A 200 with no parsed rows on a query you know has inventory is a soft block, not an empty market.

   pythonCopy

   from bs4 import BeautifulSoup
   
   def parse_craigslist(html):
       soup = BeautifulSoup(html, "html.parser")
       rows = soup.select("li.cl-static-search-result, li.result-row")
       items = []
       for r in rows:
           title_el = r.select_one(".title, .result-title")
           price_el = r.select_one(".price, .result-price")
           loc_el = r.select_one(".location, .result-hood")
           if not title_el:
               continue
           items.append({
               "item": title_el.get_text(strip=True),
               "price": price_el.get_text(strip=True) if price_el else None,
               "location": loc_el.get_text(strip=True).strip("() ") if loc_el else None,
               "category": "for-sale",
           })
       return items
   
   listings = parse_craigslist(resp.text)
   if not listings:
       raise RuntimeError("empty result set - likely a soft block, rotate and retry")
   print(len(listings), "listings")

   Field note: Never persist a zero-row parse as a real result. Wire empty output straight into your retry-with-fresh-exit path, because silently storing soft blocks is how a price dashboard quietly reports a collapsing market that never actually moved.

3. 3

   Handle Craigslist 403/429 with backoff and forced exit rotation

   When Craigslist decides an IP is too loud it returns 403 or 429 fast, sometimes within one request on a bad exit. Because rotation is per-request by default, the simplest recovery is to retry the same URL - a new exit is drawn automatically. Add exponential backoff so you are not immediately re-loading a flagged pattern, and cap retries so one poisoned metro does not stall the whole crawl.

   pythonCopy

   import time
   
   def fetch_craigslist(url, params, headers, proxies, max_retries=4):
       delay = 2.0
       for attempt in range(max_retries):
           r = requests.get(url, params=params, headers=headers,
                            proxies=proxies, timeout=20)
           if r.status_code == 200 and "result" in r.text:
               return r
           if r.status_code in (403, 429):
               # per-request rotation gives a fresh exit on the next call
               time.sleep(delay)
               delay *= 2
               continue
           r.raise_for_status()
       raise RuntimeError("exhausted retries on " + url)

   Field note: Do not raise the retry ceiling to brute-force through blocks - if a metro 429s past four fresh exits, you are querying too fast overall. Lower your per-metro concurrency instead; velocity, not attempt count, is what Craigslist scores.

4. 4

   Talk to Mercari's JSON search endpoint directly with a complete, signed header set

   Mercari's public catalog is served by an internal JSON API, so skip HTML rendering and request the search endpoint directly through a rotating US residential exit. The difference between a full response and an empty one is almost entirely the header set - a bare User-Agent with no Accept-Language, no X-Platform hint, and no DPoP token is an instant bot tell. Send a realistic, complete set, request JSON explicitly, and mirror the DPoP header your browser produces for the same call.

   pythonCopy

   import requests
   
   PROXY_USER = "aethyn-XXXXX-country-us"
   PROXY_PASS = "PASSWORD"
   PROXY_URL = "http://" + PROXY_USER + ":" + PROXY_PASS + "@proxy.aethyn.io:2099"
   proxies = {"http": PROXY_URL, "https": PROXY_URL}
   
   JSON_HEADERS = {
       "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) "
                     "AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36",
       "Accept": "application/json, text/plain, */*",
       "Accept-Language": "en-US,en;q=0.9",
       "Content-Type": "application/json",
       "X-Platform": "web",
       "Origin": "https://www.mercari.com",
       "Referer": "https://www.mercari.com/search/",
       # Mercari signs each call with a client-generated DPoP token; capture the
       # exact value from your browser network tab and pass it through here.
       "DPoP": "REPLACE_WITH_OBSERVED_DPOP_TOKEN",
   }
   
   # public search - item, price, condition, category only
   payload = {"keyword": "nintendo switch", "pageSize": 60, "page": 0}
   resp = requests.post("https://www.mercari.com/v1/api",
                        json=payload, headers=JSON_HEADERS,
                        proxies=proxies, timeout=20)
   print(resp.status_code, resp.headers.get("content-type"))

   Field note: Capture the exact request shape from your browser's network tab first - the DPoP header, X-Platform, and any operation name - and mirror it faithfully. Mercari's endpoints move field names and signing parameters between releases, so pin your parser to observed responses and re-verify whenever your empty-rate climbs.

5. 5

   Validate Mercari JSON, extract non-personal fields, and detect empty-body soft blocks

   Mercari's soft block is a 200 status wrapping an empty item array or an error object rather than an HTTP error code. Parse the JSON, confirm the item list is actually populated, and pull only the public, non-personal fields - item name, price, condition, category, and general location. If the body is empty on a query you know has stock, treat it as a block and rotate.

   pythonCopy

   def parse_mercari(resp):
       if resp.status_code != 200:
           return None  # trigger retry / rotation
       try:
           data = resp.json()
       except ValueError:
           return None  # non-JSON body = challenge page, rotate
       items = data.get("items") or data.get("data", {}).get("items") or []
       if not items:
           return None  # 200 with empty body = soft block
       out = []
       for it in items:
           out.append({
               "item": it.get("name"),
               "price": it.get("price"),
               "condition": it.get("itemCondition", {}).get("name"),
               "category": it.get("categoryName"),
               # do NOT collect seller name, id, or any contact field
           })
       return out
   
   rows = parse_mercari(resp)
   if rows is None:
       raise RuntimeError("empty or challenged Mercari response - rotate exit")

   Field note: Explicitly drop seller identity fields at the parse boundary, not later in the pipeline. Filtering personal data at the point of extraction means it never lands in your store, which is both the responsible default and far easier to defend than after-the-fact deletion.

6. 6

   Normalize both schemas and pace the aggregate crawl under both throttles

   Once you have Craigslist and Mercari rows, normalize them into one price-and-trend schema: coerce free-text Craigslist prices to numeric, carry Mercari's structured condition and shipping through, and stamp every row with source and collection time. Pace the whole job so each site stays comfortably under its limits - modest per-metro concurrency for Craigslist velocity, modest per-operation rate for Mercari - rather than running both flat out.

   pythonCopy

   import re, time, datetime
   
   def to_cents(price_text):
       if price_text is None:
           return None
       m = re.search(r"[\d,]+(?:\.\d{2})?", str(price_text))
       if not m:
           return None
       return int(round(float(m.group().replace(",", "")) * 100))
   
   def normalize(row, source):
       return {
           "source": source,
           "item": row.get("item"),
           "price_cents": to_cents(row.get("price")),
           "condition": row.get("condition"),
           "location": row.get("location"),
           "category": row.get("category"),
           "collected_at": datetime.datetime.utcnow().isoformat() + "Z",
       }
   
   # gentle pacing between requests keeps you under both sites' throttles
   for row in craigslist_rows:
       record = normalize(row, "craigslist")
       time.sleep(1.5)  # per-request spacing on top of concurrency limits

   Field note: Log source, exit region, and empty-rate per site to a rolling metric. A rising Craigslist empty-rate means back off velocity; a rising Mercari empty-rate usually means the JSON schema shifted under you - the two failure modes need opposite responses, so keep their signals separate.

Best practices that keep scrapers reliable

• Collect only public, non-personal listing fields - item, price, location, category, condition. Never harvest seller names, emails, phone numbers, or handles, never post or message, and respect each site's robots.txt and Terms of Service; consult counsel for your jurisdiction.
• Match exit geography to the target: Craigslist city subdomains fetched from same-region exits (aethyn-XXXXX-country-us-city-chicago), Mercari from US exits, so the request geography is always coherent.
• Default to per-request rotation for breadth crawling and reserve sticky sessions only for short paginated bursts that genuinely need one exit.
• Assert a non-empty parsed result before storing anything - a 200 with zero rows is a soft block on both sites, not an empty market.
• Pace by velocity, not by total count: cap per-metro concurrency for Craigslist and per-operation rate for Mercari, and lower concurrency before raising retry ceilings.
• Send a complete, realistic header set on every request - Accept-Language, Referer, and the JSON/X-Platform/DPoP headers Mercari expects - because a bare User-Agent is a classic bot tell on both surfaces.
• Pin Mercari parsers to observed JSON responses and re-verify whenever the empty-rate climbs, since the internal endpoint renames fields and rotates signing parameters between releases without notice.

Common mistakes that burn proxy budget

• Crawling from a single server or datacenter range - Craigslist reputation-scores the ASN and 403s the whole block once one IP gets loud.
• Fetching a Craigslist metro subdomain from a mismatched-region exit, adding a geographic-coherence tell for no benefit.
• Storing zero-row responses as real empty results, quietly corrupting price and trend dashboards with soft blocks.
• Sending a minimal User-Agent with no Accept-Language, X-Platform, or DPoP token, which reads as a bot and returns empty Mercari bodies.
• Brute-forcing past 429s by cranking retries and concurrency instead of lowering request velocity - the exact signal Craigslist blocks on.
• Harvesting seller contact details or attempting to message/post - outside the responsible public-data scope and a Terms of Service violation on both sites.

Common questions about this scraper problem