Scrape Quora and Pinterest Public Data with Proxies

What goes wrong when this scraper fails

Quora and Pinterest are two of the richest sources of public user-generated signal on the open web - Quora for how real people phrase questions and reason through answers, Pinterest for which visual and product trends are gaining momentum. But both are built to serve humans in a browser, not machines at scale. Quora sits behind Cloudflare and pushes logged-out visitors toward a login wall while rate-limiting on velocity; Pinterest renders nothing useful in initial HTML and instead lazy-loads pins through internal JSON endpoints guarded by bot detection. Collecting this data responsibly means reading only the public surface, at a civilized pace, without authenticating.

Why this failure mode happens

Both companies have strong commercial incentives to keep their content on-platform: Quora monetizes engaged logged-in sessions and Pinterest sells shopping and ad placement against its feed. Aggressive anti-automation - Cloudflare managed challenges, TLS/JA3 fingerprinting, per-IP velocity limits, and device fingerprinting on the internal APIs - is how they protect that. On top of that, both feeds are personalized by geography and inferred interest, so the same URL returns different content depending on the exit IP's location and session state, which is precisely the surface that trips naive scrapers into inconsistent, partially-blocked datasets.

Challenges that make this hard to automate

Quora fronts nearly everything with Cloudflare, so a datacenter IP or thin fingerprint draws a managed challenge or a 403 before you ever see an answer.
Pinterest returns almost no content in server-rendered HTML - the pins arrive asynchronously via internal JSON resource endpoints that expect specific headers and a valid CSRF token.
Both platforms rate-limit on request velocity per IP, so a fast single-IP crawl degrades to challenges within a few hundred requests.
Feeds are geo- and interest-personalized, meaning results drift by exit location and prior session state, hurting reproducibility across a dataset.
Distinguishing a genuinely empty public page from a soft block (login wall, Cloudflare interstitial, empty JSON) requires content-aware validation, not just HTTP status checks.

Approaches that usually fail

Running a single-server scraper from a cloud datacenter IP - which Cloudflare flags almost immediately on Quora and which Pinterest's bot detection rate-limits fast.
Driving a full headless browser for every page to render infinite scroll - correct but brutally slow and expensive at dataset scale, and still fingerprintable.
Buying scattered datacenter proxies and rotating blindly, without matching geography or sending complete headers, so the block rate stays high and data drifts by location.
Leaning on unofficial third-party 'Quora API' or Pinterest export tools that break every few weeks when the internal endpoints change and offer no control over exit geography.

When residential proxies fix this — and when they cannot

Residential rotating proxies put each request behind a real consumer IP with genuine reputation, which is what Cloudflare and Pinterest's detection weigh most heavily - a clean residential exit clears challenges that any datacenter IP fails. Per-request rotation spreads a crawl across thousands of IPs so no single address builds the velocity that triggers throttling, while country-pinning lets you hold the personalized feed constant for a reproducible dataset. For the short bursts where Pinterest hands you a CSRF token that must be reused across a few paginated calls, a sticky session keeps those calls on one exit so the token stays valid.

How Aethyn residential proxies help here

Aethyn's residential network is built for exactly this shape of job: high-reputation exits, per-request rotation by default, and country- or city-level targeting through a single username format. You point your existing requests or Playwright code at proxy.aethyn.io and encode the geography and session behavior in the username - no SDK, no rewrites. The Premium tier (port 2099) gives you the residential pool and rotation control that Quora's Cloudflare and Pinterest's bot detection respect.

Large residential pool with per-request rotation on port 2099, so no single IP accumulates the velocity that triggers Quora throttling or Pinterest rate limits.
Country and city targeting in the username (-country-us, -city-chicago) to hold Quora's and Pinterest's geo-personalized feeds constant across a dataset.
Sticky sessions up to 30 minutes (-session-TOKEN-lifetime-N) for the short bursts where a Pinterest CSRF token or paginated resource cursor must stay on one exit.
High-reputation residential exits that clear Cloudflare managed challenges on Quora far more reliably than datacenter IPs.
Simple username/password proxy auth that drops into requests, cURL, and Playwright unchanged - encode rotation and geo entirely in the credential string.
Predictable, transparent egress so you can pace crawls at a civilized rate and respect each site's robots.txt and Terms of Service.

How to implement this with residential proxies

1
Route every request through a rotating residential exit
Start by putting all traffic behind Aethyn's residential pool on the Premium port 2099. Use per-request rotation as the default - omit any session token so consecutive requests never stack on the same IP, which is what keeps Quora's velocity-based throttling and Pinterest's rate limiter from ever seeing a burst. Pin the country so the personalized feed stays constant for your dataset.
python
```
import requests

# Premium tier -> port 2099. Per-request rotation: no session token.
PROXY = "http://aethyn-XXXXX-country-us:PASSWORD@proxy.aethyn.io:2099"
proxies = {"http": PROXY, "https": PROXY}

resp = requests.get(
    "https://www.quora.com/robots.txt",
    proxies=proxies,
    timeout=20,
)
print(resp.status_code)
print(resp.text[:500])  # read and honor the crawl directives before anything else
```
Field note: Fetch and actually parse robots.txt first, per host. Both Quora and Pinterest list disallowed paths - staying out of them is the difference between responsible public collection and abuse, and it keeps you off the paths most aggressively defended.
2
Send a complete, realistic header set
A bare User-Agent with no Accept-Language or Accept header is a classic bot tell, and on Cloudflare-fronted Quora it is often enough on its own to draw a challenge. Send a full, coherent header set that matches a real browser, and keep Accept-Language consistent with the country you are exiting from so the fingerprint and geo agree.
python
```
HEADERS = {
    "User-Agent": (
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
        "AppleWebKit/537.36 (KHTML, like Gecko) "
        "Chrome/125.0.0.0 Safari/537.36"
    ),
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8",
    "Accept-Language": "en-US,en;q=0.9",
    "Accept-Encoding": "gzip, deflate, br",
    "Upgrade-Insecure-Requests": "1",
    "Sec-Fetch-Dest": "document",
    "Sec-Fetch-Mode": "navigate",
    "Sec-Fetch-Site": "none",
}

r = requests.get(
    "https://www.quora.com/topic/Machine-Learning",
    headers=HEADERS,
    proxies=proxies,
    timeout=20,
)
print(r.status_code, len(r.text))
```
Field note: Keep Accept-Language aligned with your exit country. A US residential IP sending Accept-Language: ru-RU is an incoherent fingerprint, and incoherence is exactly what fingerprint-based detection scores against you - matching one signal while breaking another is worse than matching none.
3
Detect Cloudflare challenges and soft blocks before parsing
On Quora a 200 OK can still be a Cloudflare interstitial or a login-wall stub with no real answers, and on Pinterest an empty resource payload looks identical to a valid one at the HTTP layer. Build a content-aware guard that classifies the response before you trust it, so you retry on a fresh IP instead of silently storing garbage.
python
```
def classify(resp):
    body = resp.text.lower()
    header_names = {k.lower() for k in resp.headers}
    if resp.status_code in (403, 429):
        return "blocked"
    if "cf-mitigated" in header_names or \
       "just a moment" in body or "cdn-cgi/challenge" in body:
        return "challenge"
    if "log in to continue" in body or "login_wall" in body:
        return "login_wall"
    return "ok"

state = classify(r)
if state != "ok":
    # rotate (a new per-request IP happens automatically) and back off
    print("soft block:", state)
else:
    pass  # safe to parse
```
Field note: Treat challenge, login_wall, and empty-result as retry-with-backoff, never as data. A 200 with zero parsed answers is almost never a real empty page - store it and your training set quietly fills with blank rows you will not notice until the model underperforms.

Pull Pinterest pins from the internal JSON resource endpoints

Pinterest's HTML is a shell; the pins load through internal JSON resource routes that the infinite scroll calls as you scroll. Hitting those endpoints directly is far cheaper than rendering a browser, but they require an x-csrftoken (read from the csrftoken cookie) and the x-requested-with header, and they page via a bookmark cursor. Because the CSRF token and cursor must stay coherent across a paginated burst, use a short sticky session so those calls share one exit.

python

import json

# Sticky session for a paginated burst: token + cursor stay on one IP.
STICKY = "http://aethyn-XXXXX-country-us-session-pinboard1-lifetime-10:PASSWORD@proxy.aethyn.io:2099"
sproxies = {"http": STICKY, "https": STICKY}

s = requests.Session()
s.proxies = sproxies
s.headers.update(HEADERS)

# Prime cookies (gets a csrftoken) against a public search URL.
s.get("https://www.pinterest.com/search/pins/?q=minimalist%20kitchen", timeout=20)
csrf = s.cookies.get("csrftoken", "")

api_headers = {
    "x-csrftoken": csrf,
    "x-requested-with": "XMLHttpRequest",
    "x-app-version": "latest",
    "accept": "application/json, text/javascript, */*; q=0.01",
}

options = {"query": "minimalist kitchen", "scope": "pins", "bookmarks": [""]}
params = {
    "source_url": "/search/pins/?q=minimalist%20kitchen",
    "data": json.dumps({"options": options, "context": {}}),
}

resp = s.get(
    "https://www.pinterest.com/resource/BaseSearchResource/get/",
    headers=api_headers, params=params, timeout=25,
)
payload = resp.json()
results = payload.get("resource_response", {}).get("data", {}).get("results", [])
print("pins:", len(results))
bookmark = payload.get("resource", {}).get("options", {}).get("bookmarks", [""])[0]

Field note: Reuse the returned bookmark as the next page's cursor, and keep the whole pagination loop inside one sticky session (lifetime up to 30 min). Rotating IP mid-pagination invalidates the CSRF token and the cursor context, and Pinterest returns an empty results array that your classifier will correctly reject.

Render infinite scroll with Playwright only where you must

Some Quora topic and question pages, and a few Pinterest surfaces, only expose their public content after client-side rendering. Reserve a headless browser for those cases - it is 5 to 10x slower and heavier than a direct request, so use it as a fallback, not the default. Route the browser through the same residential exit so its traffic shares the IP reputation and geography of your HTTP crawl.

python

from playwright.sync_api import sync_playwright

PROXY_SERVER = "http://proxy.aethyn.io:2099"
PROXY_USER = "aethyn-XXXXX-country-gb"   # per-request rotation, GB feed
PROXY_PASS = "PASSWORD"

with sync_playwright() as p:
    browser = p.chromium.launch(
        headless=True,
        proxy={"server": PROXY_SERVER, "username": PROXY_USER, "password": PROXY_PASS},
    )
    page = browser.new_page(locale="en-GB")
    page.goto("https://www.quora.com/topic/Data-Science", wait_until="networkidle", timeout=45000)

    # Scroll to trigger lazy-loaded public content, bounded so we stay civilized.
    for _ in range(5):
        page.mouse.wheel(0, 4000)
        page.wait_for_timeout(1200)

    # Collect only public question text; never touch profile or contact data.
    questions = page.eval_on_selector_all(
        "a[href*='/']",
        "els => els.map(e => e.innerText).filter(t => t.endsWith('?'))",
    )
    print(len(questions), "public questions")
    browser.close()

Field note: Set the browser locale to match your exit country (en-GB for a GB IP). A mismatched locale versus IP geolocation is a fingerprint contradiction that Cloudflare scores, and it also skews the personalized feed you are trying to hold constant across the dataset.

Pace the crawl, back off on blocks, and validate before writing

Sustained collection is about rhythm, not raw speed. Cap concurrency, add jitter between requests, and apply exponential backoff whenever your classifier reports a challenge or 429 - a polite crawl at a few requests per second across a rotating pool outlasts a fast one that gets an IP range flagged. Only write records that pass a non-empty, schema-valid check.

python

import time, random

def fetch_with_backoff(url, max_tries=4):
    delay = 2.0
    for attempt in range(max_tries):
        r = requests.get(url, headers=HEADERS, proxies=proxies, timeout=20)
        state = classify(r)
        if state == "ok":
            return r
        time.sleep(delay + random.uniform(0, 1.5))  # jitter
        delay *= 2  # exponential backoff; next request is a fresh IP
    return None

def save_answer(rec):
    # Defensive validation: reject soft blocks and personal data.
    if not rec.get("question") or not rec.get("answer_text"):
        return False
    assert "author_email" not in rec and "author_profile" not in rec
    # ... write public, non-personal record to your store ...
    return True

r = fetch_with_backoff("https://www.quora.com/topic/Machine-Learning")
if r:
    print("got", len(r.text), "bytes of public content")

Field note: Log the block-rate per country and per hour. A sudden climb usually means you are pushing velocity on one geo too hard - throttle that lane rather than the whole crawl, and you keep the reputation of your exits intact for the long-running dataset build.

Best practices that keep scrapers reliable

Collect only public, non-personal content - public questions, answers, pins, boards, and topic/trend data. Never harvest personal profile fields, contact details, or anything behind authentication, and respect each site's robots.txt and Terms of Service; consult counsel for your jurisdiction.
Rotate per request by default and only reach for a sticky session when a CSRF token or pagination cursor must stay on one exit for a short burst.
Pin the exit country per dataset so the geo-personalized feeds on Quora and Pinterest stay reproducible across runs.
Send a complete, coherent header set - User-Agent, Accept, Accept-Language, Sec-Fetch-* - with Accept-Language matching the exit geography.
Prefer Pinterest's internal JSON resource endpoints over rendered HTML, and reserve Playwright for the pages that genuinely require client-side rendering.
Classify every response for Cloudflare challenges, login walls, and empty payloads before parsing, and back off exponentially on blocks with jitter.
Pace crawls to a civilized rate and monitor block-rate per country and hour so you throttle a hot lane instead of burning the whole pool's reputation.

Common mistakes that burn proxy budget

Crawling from datacenter IPs - Cloudflare flags them on Quora almost instantly and Pinterest rate-limits them fast; residential reputation is the signal that matters most.
Storing 200 OK responses without content validation, so Cloudflare interstitials, login-wall stubs, and empty JSON silently poison the dataset.
Sending a bare User-Agent with no Accept-Language or Sec-Fetch headers, which is a textbook bot fingerprint on both sites.
Rotating the IP mid-pagination on Pinterest, which invalidates the CSRF token and cursor and returns empty results.
Rendering every page with a headless browser instead of hitting Pinterest's JSON endpoints directly - 5 to 10x slower and more fingerprintable for no benefit.
Attempting to defeat Quora's login wall or scraping personal profile data, which crosses the line from public collection into ToS violation and personal-data harvesting.

Common questions about this scraper problem

Is scraping Quora and Pinterest legal?

Collecting genuinely public, non-personal content is generally lower-risk, but it still has to respect each site's Terms of Service and robots.txt. Do not circumvent authentication, do not harvest personal profile or contact data, keep your request volume reasonable, and consult counsel for your specific jurisdiction and use case - this guide is not legal advice.

Why does Quora return a 403 or a 'Just a moment' page from my server?

That is a Cloudflare managed challenge. It fires on the combination of IP reputation, request velocity, and browser fingerprint - a datacenter IP with a thin header set is the classic trigger. Route through a residential exit, send a complete header set, and slow down; fixing only one of those three rarely clears it.

Do I need a headless browser for Pinterest?

Usually no. Pinterest loads pins through internal JSON resource endpoints (like BaseSearchResource and BoardFeedResource) that you can call directly with the right x-csrftoken and x-requested-with headers. Reserve Playwright for the few surfaces that only render client-side, because it is far slower and heavier at dataset scale.

Why do I get different pins or answers on each run?

Both feeds are geo- and interest-personalized, so results shift with the exit IP's location and any session state. Pin your country in the proxy username (for example -country-us) per dataset to hold the feed constant, and expect some drift on trending surfaces that update continuously.

When should I use a sticky session instead of per-request rotation?

Only for short bursts that must stay coherent on one exit - most commonly a Pinterest pagination loop where the CSRF token and bookmark cursor have to persist. Use -session-TOKEN-lifetime-N (up to 30 minutes) for that, and go back to per-request rotation for everything else.

How do I tell a soft block from a genuinely empty page?

Check the body, not just the status code. A Cloudflare interstitial contains markers like 'just a moment' or a cdn-cgi/challenge path, a login wall says 'log in to continue', and a soft-blocked Pinterest call returns an empty results array. Classify these as retry-with-backoff and never write them as data.

What request rate is safe?

There is no published number, but a few requests per second spread across a rotating residential pool, with jitter and exponential backoff on any 429 or challenge, is a civilized pace that sustains for long dataset builds. Monitor block-rate per country and per hour and throttle the hot lane rather than the whole crawl.

Can I use this data to train a model?

You can train on public, non-personal text and image metadata subject to the site's Terms of Service and applicable copyright and data-protection law. Keep records that identify individuals out of the set, document your provenance, and get legal review before any commercial or redistributed use of the dataset.

Which Aethyn tier and port should I use?

Premium on port 2099. It gives you the residential pool, per-request rotation, and country/city targeting that Quora's Cloudflare and Pinterest's bot detection respect. Encode rotation and geography entirely in the proxy username - no SDK required.

Why does Pinterest return empty results even with a valid token?

The most common cause is rotating the exit IP mid-pagination, which breaks the CSRF token and cursor context. Keep the whole paginated burst inside one sticky session, and make sure x-csrftoken matches the current csrftoken cookie and x-requested-with is set to XMLHttpRequest.

What goes wrong when this scraper fails

Why this failure mode happens

Challenges that make this hard to automate

Quora fronts nearly everything with Cloudflare, so a datacenter IP or thin fingerprint draws a managed challenge or a 403 before you ever see an answer.
Pinterest returns almost no content in server-rendered HTML - the pins arrive asynchronously via internal JSON resource endpoints that expect specific headers and a valid CSRF token.
Both platforms rate-limit on request velocity per IP, so a fast single-IP crawl degrades to challenges within a few hundred requests.
Feeds are geo- and interest-personalized, meaning results drift by exit location and prior session state, hurting reproducibility across a dataset.
Distinguishing a genuinely empty public page from a soft block (login wall, Cloudflare interstitial, empty JSON) requires content-aware validation, not just HTTP status checks.

Approaches that usually fail

Running a single-server scraper from a cloud datacenter IP - which Cloudflare flags almost immediately on Quora and which Pinterest's bot detection rate-limits fast.
Driving a full headless browser for every page to render infinite scroll - correct but brutally slow and expensive at dataset scale, and still fingerprintable.
Buying scattered datacenter proxies and rotating blindly, without matching geography or sending complete headers, so the block rate stays high and data drifts by location.
Leaning on unofficial third-party 'Quora API' or Pinterest export tools that break every few weeks when the internal endpoints change and offer no control over exit geography.

When residential proxies fix this — and when they cannot

How Aethyn residential proxies help here

Large residential pool with per-request rotation on port 2099, so no single IP accumulates the velocity that triggers Quora throttling or Pinterest rate limits.
Country and city targeting in the username (-country-us, -city-chicago) to hold Quora's and Pinterest's geo-personalized feeds constant across a dataset.
Sticky sessions up to 30 minutes (-session-TOKEN-lifetime-N) for the short bursts where a Pinterest CSRF token or paginated resource cursor must stay on one exit.
High-reputation residential exits that clear Cloudflare managed challenges on Quora far more reliably than datacenter IPs.
Simple username/password proxy auth that drops into requests, cURL, and Playwright unchanged - encode rotation and geo entirely in the credential string.
Predictable, transparent egress so you can pace crawls at a civilized rate and respect each site's robots.txt and Terms of Service.

How to implement this with residential proxies

1
Route every request through a rotating residential exit
Start by putting all traffic behind Aethyn's residential pool on the Premium port 2099. Use per-request rotation as the default - omit any session token so consecutive requests never stack on the same IP, which is what keeps Quora's velocity-based throttling and Pinterest's rate limiter from ever seeing a burst. Pin the country so the personalized feed stays constant for your dataset.
python
```
import requests

# Premium tier -> port 2099. Per-request rotation: no session token.
PROXY = "http://aethyn-XXXXX-country-us:PASSWORD@proxy.aethyn.io:2099"
proxies = {"http": PROXY, "https": PROXY}

resp = requests.get(
    "https://www.quora.com/robots.txt",
    proxies=proxies,
    timeout=20,
)
print(resp.status_code)
print(resp.text[:500])  # read and honor the crawl directives before anything else
```
Field note: Fetch and actually parse robots.txt first, per host. Both Quora and Pinterest list disallowed paths - staying out of them is the difference between responsible public collection and abuse, and it keeps you off the paths most aggressively defended.
2
Send a complete, realistic header set
A bare User-Agent with no Accept-Language or Accept header is a classic bot tell, and on Cloudflare-fronted Quora it is often enough on its own to draw a challenge. Send a full, coherent header set that matches a real browser, and keep Accept-Language consistent with the country you are exiting from so the fingerprint and geo agree.
python
```
HEADERS = {
    "User-Agent": (
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
        "AppleWebKit/537.36 (KHTML, like Gecko) "
        "Chrome/125.0.0.0 Safari/537.36"
    ),
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8",
    "Accept-Language": "en-US,en;q=0.9",
    "Accept-Encoding": "gzip, deflate, br",
    "Upgrade-Insecure-Requests": "1",
    "Sec-Fetch-Dest": "document",
    "Sec-Fetch-Mode": "navigate",
    "Sec-Fetch-Site": "none",
}

r = requests.get(
    "https://www.quora.com/topic/Machine-Learning",
    headers=HEADERS,
    proxies=proxies,
    timeout=20,
)
print(r.status_code, len(r.text))
```
Field note: Keep Accept-Language aligned with your exit country. A US residential IP sending Accept-Language: ru-RU is an incoherent fingerprint, and incoherence is exactly what fingerprint-based detection scores against you - matching one signal while breaking another is worse than matching none.
3
Detect Cloudflare challenges and soft blocks before parsing
On Quora a 200 OK can still be a Cloudflare interstitial or a login-wall stub with no real answers, and on Pinterest an empty resource payload looks identical to a valid one at the HTTP layer. Build a content-aware guard that classifies the response before you trust it, so you retry on a fresh IP instead of silently storing garbage.
python
```
def classify(resp):
    body = resp.text.lower()
    header_names = {k.lower() for k in resp.headers}
    if resp.status_code in (403, 429):
        return "blocked"
    if "cf-mitigated" in header_names or \
       "just a moment" in body or "cdn-cgi/challenge" in body:
        return "challenge"
    if "log in to continue" in body or "login_wall" in body:
        return "login_wall"
    return "ok"

state = classify(r)
if state != "ok":
    # rotate (a new per-request IP happens automatically) and back off
    print("soft block:", state)
else:
    pass  # safe to parse
```
Field note: Treat challenge, login_wall, and empty-result as retry-with-backoff, never as data. A 200 with zero parsed answers is almost never a real empty page - store it and your training set quietly fills with blank rows you will not notice until the model underperforms.

Pull Pinterest pins from the internal JSON resource endpoints

python

import json

# Sticky session for a paginated burst: token + cursor stay on one IP.
STICKY = "http://aethyn-XXXXX-country-us-session-pinboard1-lifetime-10:PASSWORD@proxy.aethyn.io:2099"
sproxies = {"http": STICKY, "https": STICKY}

s = requests.Session()
s.proxies = sproxies
s.headers.update(HEADERS)

# Prime cookies (gets a csrftoken) against a public search URL.
s.get("https://www.pinterest.com/search/pins/?q=minimalist%20kitchen", timeout=20)
csrf = s.cookies.get("csrftoken", "")

api_headers = {
    "x-csrftoken": csrf,
    "x-requested-with": "XMLHttpRequest",
    "x-app-version": "latest",
    "accept": "application/json, text/javascript, */*; q=0.01",
}

options = {"query": "minimalist kitchen", "scope": "pins", "bookmarks": [""]}
params = {
    "source_url": "/search/pins/?q=minimalist%20kitchen",
    "data": json.dumps({"options": options, "context": {}}),
}

resp = s.get(
    "https://www.pinterest.com/resource/BaseSearchResource/get/",
    headers=api_headers, params=params, timeout=25,
)
payload = resp.json()
results = payload.get("resource_response", {}).get("data", {}).get("results", [])
print("pins:", len(results))
bookmark = payload.get("resource", {}).get("options", {}).get("bookmarks", [""])[0]

Render infinite scroll with Playwright only where you must

python

from playwright.sync_api import sync_playwright

PROXY_SERVER = "http://proxy.aethyn.io:2099"
PROXY_USER = "aethyn-XXXXX-country-gb"   # per-request rotation, GB feed
PROXY_PASS = "PASSWORD"

with sync_playwright() as p:
    browser = p.chromium.launch(
        headless=True,
        proxy={"server": PROXY_SERVER, "username": PROXY_USER, "password": PROXY_PASS},
    )
    page = browser.new_page(locale="en-GB")
    page.goto("https://www.quora.com/topic/Data-Science", wait_until="networkidle", timeout=45000)

    # Scroll to trigger lazy-loaded public content, bounded so we stay civilized.
    for _ in range(5):
        page.mouse.wheel(0, 4000)
        page.wait_for_timeout(1200)

    # Collect only public question text; never touch profile or contact data.
    questions = page.eval_on_selector_all(
        "a[href*='/']",
        "els => els.map(e => e.innerText).filter(t => t.endsWith('?'))",
    )
    print(len(questions), "public questions")
    browser.close()

Pace the crawl, back off on blocks, and validate before writing

python

import time, random

def fetch_with_backoff(url, max_tries=4):
    delay = 2.0
    for attempt in range(max_tries):
        r = requests.get(url, headers=HEADERS, proxies=proxies, timeout=20)
        state = classify(r)
        if state == "ok":
            return r
        time.sleep(delay + random.uniform(0, 1.5))  # jitter
        delay *= 2  # exponential backoff; next request is a fresh IP
    return None

def save_answer(rec):
    # Defensive validation: reject soft blocks and personal data.
    if not rec.get("question") or not rec.get("answer_text"):
        return False
    assert "author_email" not in rec and "author_profile" not in rec
    # ... write public, non-personal record to your store ...
    return True

r = fetch_with_backoff("https://www.quora.com/topic/Machine-Learning")
if r:
    print("got", len(r.text), "bytes of public content")

Best practices that keep scrapers reliable

Collect only public, non-personal content - public questions, answers, pins, boards, and topic/trend data. Never harvest personal profile fields, contact details, or anything behind authentication, and respect each site's robots.txt and Terms of Service; consult counsel for your jurisdiction.
Rotate per request by default and only reach for a sticky session when a CSRF token or pagination cursor must stay on one exit for a short burst.
Pin the exit country per dataset so the geo-personalized feeds on Quora and Pinterest stay reproducible across runs.
Send a complete, coherent header set - User-Agent, Accept, Accept-Language, Sec-Fetch-* - with Accept-Language matching the exit geography.
Prefer Pinterest's internal JSON resource endpoints over rendered HTML, and reserve Playwright for the pages that genuinely require client-side rendering.
Classify every response for Cloudflare challenges, login walls, and empty payloads before parsing, and back off exponentially on blocks with jitter.
Pace crawls to a civilized rate and monitor block-rate per country and hour so you throttle a hot lane instead of burning the whole pool's reputation.

Common mistakes that burn proxy budget

Crawling from datacenter IPs - Cloudflare flags them on Quora almost instantly and Pinterest rate-limits them fast; residential reputation is the signal that matters most.
Storing 200 OK responses without content validation, so Cloudflare interstitials, login-wall stubs, and empty JSON silently poison the dataset.
Sending a bare User-Agent with no Accept-Language or Sec-Fetch headers, which is a textbook bot fingerprint on both sites.
Rotating the IP mid-pagination on Pinterest, which invalidates the CSRF token and cursor and returns empty results.
Rendering every page with a headless browser instead of hitting Pinterest's JSON endpoints directly - 5 to 10x slower and more fingerprintable for no benefit.
Attempting to defeat Quora's login wall or scraping personal profile data, which crosses the line from public collection into ToS violation and personal-data harvesting.

Common questions about this scraper problem

Is scraping Quora and Pinterest legal?

Why does Quora return a 403 or a 'Just a moment' page from my server?

Do I need a headless browser for Pinterest?

Why do I get different pins or answers on each run?

When should I use a sticky session instead of per-request rotation?

How do I tell a soft block from a genuinely empty page?

What request rate is safe?

Can I use this data to train a model?

Which Aethyn tier and port should I use?

Why does Pinterest return empty results even with a valid token?

Scraping Quora and Pinterest for Public Trend Data and Training Datasets

Key takeaways

On this page

What goes wrong when this scraper fails

Challenges that make this hard to automate

How Aethyn residential proxies help here

How to implement this with residential proxies

Route every request through a rotating residential exit

Send a complete, realistic header set

Detect Cloudflare challenges and soft blocks before parsing

Pull Pinterest pins from the internal JSON resource endpoints

Render infinite scroll with Playwright only where you must

Pace the crawl, back off on blocks, and validate before writing

Best practices that keep scrapers reliable

Common questions about this scraper problem

Solve this with Aethyn residential proxies

Scraping Quora and Pinterest for Public Trend Data and Training Datasets

Key takeaways

On this page

What goes wrong when this scraper fails

Challenges that make this hard to automate

How Aethyn residential proxies help here

How to implement this with residential proxies

Route every request through a rotating residential exit

Send a complete, realistic header set

Detect Cloudflare challenges and soft blocks before parsing

Pull Pinterest pins from the internal JSON resource endpoints

Render infinite scroll with Playwright only where you must

Pace the crawl, back off on blocks, and validate before writing

Best practices that keep scrapers reliable

Common questions about this scraper problem

Solve this with Aethyn residential proxies

Key takeaways

On this page

What goes wrong when this scraper fails

Challenges that make this hard to automate

How Aethyn residential proxies help here

How to implement this with residential proxies

Route every request through a rotating residential exit

Send a complete, realistic header set

Detect Cloudflare challenges and soft blocks before parsing

Pull Pinterest pins from the internal JSON resource endpoints

Render infinite scroll with Playwright only where you must

Pace the crawl, back off on blocks, and validate before writing

Best practices that keep scrapers reliable

Common questions about this scraper problem

Related guides & resources

Solve this with Aethyn residential proxies

Key takeaways

On this page

What goes wrong when this scraper fails

Challenges that make this hard to automate

How Aethyn residential proxies help here

How to implement this with residential proxies

Route every request through a rotating residential exit

Send a complete, realistic header set

Detect Cloudflare challenges and soft blocks before parsing

Pull Pinterest pins from the internal JSON resource endpoints

Render infinite scroll with Playwright only where you must

Pace the crawl, back off on blocks, and validate before writing

Best practices that keep scrapers reliable

Common questions about this scraper problem

Related guides & resources

Solve this with Aethyn residential proxies