Scrape Glassdoor & Indeed Job Data with Proxies

What goes wrong when this scraper fails

Labor-market analytics - salary benchmarking, hiring-demand tracking, competitor headcount signals, skills-gap analysis - runs on data that only exists on public job boards and employer-review sites. Glassdoor and Indeed are the two richest public sources for the English-speaking and European markets: Glassdoor for aggregate salary ranges and public company ratings, Indeed for live job-posting volume and descriptions. The problem is that both are among the hardest consumer web surfaces to read programmatically. Indeed sits fully behind Cloudflare and treats its job-search endpoints as high-value targets, while Glassdoor combines Cloudflare with its own bot-detection stack, soft login walls, and salary data that is deliberately gated by geography.

Why this failure mode happens

These sites monetize their data - Glassdoor sells employer-branding products and Indeed sells sponsored listings and hiring tools - so unrestricted automated access directly competes with their business. Cloudflare gives them managed challenge pages, JA3/JA4 TLS fingerprinting, and IP-reputation scoring at their edge, and both layer application-level heuristics on top: request velocity per IP, presence of a complete and consistent header set, cookie continuity, and whether the client executes JavaScript. Glassdoor additionally gates salary data by region because the numbers themselves differ by market, and it uses a soft wall to push anonymous visitors toward creating an account. The net effect is that a naive requests.get() from a cloud server gets a 403 or a Cloudflare interstitial long before it ever sees a job description.

Challenges that make this hard to automate

Cloudflare managed challenges and JA3/JA4 TLS fingerprinting flag the connection at the edge - a stock python-requests or Node client is identifiable before any HTML is served, independent of how good your headers are.
Glassdoor's soft login wall returns a 200 with a full HTML shell, but salary figures and review bodies are blurred, truncated, or replaced with a signup prompt until the session looks like a legitimate regional visitor.
Geo-gating means the same URL returns different data by country: Glassdoor salary bands are region-specific, and Indeed routes you to a locale domain and localized job set based on your exit IP.
Request velocity from a single IP trips rate limits fast - a few dozen job-search pages per minute from one address will earn a challenge or a temporary block on both sites.
Layout and markup drift frequently: Indeed rotates class names and embeds data in a __NEXT_DATA__ / mosaic JSON blob, and Glassdoor ships React-rendered content, so brittle CSS selectors silently break and quietly return empty.

Approaches that usually fail

Running the scraper from a handful of cloud/VPS IPs - fails almost immediately because datacenter ASNs are pre-flagged by Cloudflare's IP reputation database and get challenged on the first request.
Hard-coded sleep() delays between requests - reduces velocity but does nothing for the fingerprint and reputation signals that actually trigger the block, so you throttle to a crawl and still get flagged.
Headless Chrome with default settings - solves JavaScript rendering but leaks automation via navigator.webdriver, missing browser plugins, and a Playwright/Puppeteer-default TLS fingerprint that Cloudflare recognizes.
Buying access to a third-party 'Glassdoor/Indeed data' reseller - opaque freshness, no control over geo coverage, and you inherit their compliance posture and their outages with zero visibility into how the data was obtained.

When residential proxies fix this — and when they cannot

Rotating residential proxies put your requests behind real consumer IP addresses with clean reputation, which is the single signal Cloudflare weights most heavily - a residential IP from the target country starts as trusted rather than pre-flagged. Per-request rotation spreads velocity across a large pool so no single address accumulates the request rate that trips rate limits, while country- and city-targeted exits let you fetch the exact regional dataset you are researching (US salary bands from a US IP, German job results from a DE IP). For the short multi-step flows that need cookie continuity - loading a company page then its aggregate salary tab - a sticky session keeps you on one exit for a controlled 5-10 minute window before rotating away.

How Aethyn residential proxies help here

Aethyn's Elite residential pool is built for exactly this class of high-defense target. You get country and city geo-targeting so your exit IP matches the labor market you are analyzing, per-request rotation by default so velocity never concentrates, and short sticky sessions for the rare multi-hop flow - all through one proxy endpoint with a username-encoded config, no separate gateway per region.

Elite-tier residential IPs with clean consumer reputation that clear Cloudflare's edge reputation check on Glassdoor and Indeed where datacenter ranges get challenged instantly.
Country and city targeting via the username (e.g. -country-us-city-chicago) so you pull the correct region-specific Glassdoor salary bands and localized Indeed job sets.
Per-request rotation as the default - every request exits a fresh IP, so job-search pagination never stacks velocity on one address.
Sticky sessions up to 30 minutes (-session-TOKEN-lifetime-10) for the occasional company-page to salary-tab flow that must keep cookie continuity on one exit.
A single endpoint - proxy.aethyn.io:5499 for Elite - so you swap countries by editing the username, not by re-plumbing to a new host per region.
A pool large enough to run parallel workers across many markets at once without two workers colliding on the same exit IP mid-burst.

How to implement this with residential proxies

1
Configure the Elite residential endpoint and verify geo-targeting
Start by confirming your proxy config resolves to the right country before you touch Glassdoor or Indeed. Elite tier uses port 5499 on proxy.aethyn.io. The country you encode in the username determines which regional dataset both sites will serve, so verify it first - a US exit gives US salary bands and indeed.com results, a GB exit gives uk.indeed.com. Leave off any session token so you get a fresh IP per request; only add one when a multi-step flow needs continuity.
python
```
import requests

# Elite tier -> port 5499. No session token = per-request rotation.
proxy_user = "aethyn-XXXXX-country-us"
proxy_pass = "PASSWORD"
proxy_url = "http://" + proxy_user + ":" + proxy_pass + "@proxy.aethyn.io:5499"

proxies = {"http": proxy_url, "https": proxy_url}

# Confirm the exit IP's country before scraping anything real.
r = requests.get("https://ipinfo.io/json", proxies=proxies, timeout=20)
info = r.json()
print("exit ip:", info.get("ip"), "country:", info.get("country"))
assert info.get("country") == "US", "exit country mismatch - fix the username"
```
Field note: Verify the exit country programmatically at the start of every run, not just once by hand. Residential pools are large and a misconfigured username fails silently - you will happily scrape a full dataset of the wrong region and only notice when the salary numbers look off weeks later.

Send a complete, realistic header set and a matching Accept-Language

A bare User-Agent with no Accept-Language, no Accept, and no Sec-Fetch headers is a classic bot tell that Cloudflare and both sites' heuristics catch instantly. Match your Accept-Language to the country you are exiting from - a US IP sending Accept-Language: de-DE is contradictory and raises your bot score. Send the full header set a real Chrome sends, in a plausible order, and persist cookies across the session with a requests.Session.

python

import requests

session = requests.Session()
session.proxies = {"http": proxy_url, "https": proxy_url}

session.headers.update({
    "User-Agent": (
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
        "AppleWebKit/537.36 (KHTML, like Gecko) "
        "Chrome/126.0.0.0 Safari/537.36"
    ),
    "Accept": (
        "text/html,application/xhtml+xml,application/xml;q=0.9,"
        "image/avif,image/webp,*/*;q=0.8"
    ),
    "Accept-Language": "en-US,en;q=0.9",  # match your exit country
    "Accept-Encoding": "gzip, deflate, br",
    "Sec-Fetch-Dest": "document",
    "Sec-Fetch-Mode": "navigate",
    "Sec-Fetch-Site": "none",
    "Sec-Fetch-User": "?1",
    "Upgrade-Insecure-Requests": "1",
})

resp = session.get("https://www.indeed.com/jobs?q=data+engineer&l=Austin%2C+TX", timeout=30)
print(resp.status_code, len(resp.text))

Field note: Keep the Accept-Language consistent with the exit IP's country for the entire run. Mismatched locale and geo is one of the cheapest signals for a site to score against you, and it is the mistake most scrapers make when they template one header block and rotate countries underneath it.

Use a real browser fingerprint for Cloudflare-challenged pages

Glassdoor and Indeed both render key content with JavaScript and both throw Cloudflare interstitials that a plain HTTP client cannot pass. When a page returns a challenge, drive it with Playwright over the same Elite proxy so you get a genuine browser TLS fingerprint and JS execution. Route Playwright through the proxy at launch and it inherits the same geo-targeting. Reserve this for pages that actually need it - it is far heavier than requests, so use it as a fallback, not the default.

python

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(
        headless=True,
        proxy={
            "server": "http://proxy.aethyn.io:5499",
            "username": "aethyn-XXXXX-country-us",
            "password": "PASSWORD",
        },
    )
    ctx = browser.new_context(
        locale="en-US",
        user_agent=(
            "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
            "AppleWebKit/537.36 (KHTML, like Gecko) "
            "Chrome/126.0.0.0 Safari/537.36"
        ),
    )
    page = ctx.new_page()
    page.goto("https://www.glassdoor.com/Salaries/data-engineer-salary-SRCH_KO0,13.htm",
              wait_until="domcontentloaded", timeout=45000)
    # Wait for the salary content to actually render, not just the shell.
    page.wait_for_selector("[data-test='salary-estimate'], .salary", timeout=15000)
    html = page.content()
    print(len(html))
    browser.close()

Field note: Set a real locale on the context and let the residential IP do the heavy lifting - do not over-engineer stealth patches you cannot verify. A perfect browser fingerprint on a flagged datacenter IP still gets challenged, while a plausible one on a clean residential IP usually passes.

Parse defensively and detect soft blocks before trusting data

Indeed embeds its job data in a JSON blob (mosaic provider data / __NEXT_DATA__) that is far more stable than its rotating CSS class names - parse that when you can. Glassdoor ships React-rendered salary content that you extract from the rendered DOM. Whichever you use, treat a 200 with zero results or a signup-wall marker as a soft block, not an empty page. Assert on the parsed count and on the absence of challenge markers before you write anything to your store.

python

import json, re
from bs4 import BeautifulSoup

def parse_indeed_jobs(html):
    soup = BeautifulSoup(html, "html.parser")

    # Soft-block / challenge detection first.
    lowered = html.lower()
    block_markers = ("just a moment", "cf-challenge", "verify you are human", "enable javascript")
    if any(marker in lowered for marker in block_markers):
        raise RuntimeError("cloudflare challenge page - rotate IP and retry")

    # Prefer the embedded JSON blob over brittle CSS selectors.
    m = re.search(
        r'window\.mosaic\.providerData\["mosaic-provider-jobcards"\]\s*=\s*(\{.*?\});',
        html, re.DOTALL,
    )
    jobs = []
    if m:
        data = json.loads(m.group(1))
        results = (
            data.get("metaData", {})
            .get("mosaicProviderJobCardsModel", {})
            .get("results", [])
        )
        for job in results:
            jobs.append({
                "title": job.get("title"),
                "company": job.get("company"),
                "location": job.get("formattedLocation"),
                "salary": job.get("salarySnippet", {}).get("text"),
            })

    if not jobs:
        raise RuntimeError("zero jobs parsed - treat as soft block, do not persist")
    return jobs

Field note: Assert a non-empty, plausible result set before writing to your database. A 200 OK with zero jobs, or a Glassdoor salary that came back as a round placeholder, is almost never a real empty page - it is a soft block or a blurred wall, and silently storing it is how a benchmark dashboard goes quietly wrong for weeks.

5
Handle blocks and throttle with rotation and backoff
Per-request rotation already spreads your load, but you still need to react to the blocks that do land. Treat 403, 429, and any Cloudflare challenge marker as a rotation trigger: because the default config rotates per request, a plain retry already gives you a fresh exit IP. Add exponential backoff with jitter so a burst of blocks does not turn into a hammering loop, and cap retries so a genuinely dead query does not spin forever. Keep per-worker request rate modest - a few requests per second per market, not per IP.
python
```
import time, random

def fetch_with_retry(session, url, max_attempts=5):
    for attempt in range(1, max_attempts + 1):
        try:
            resp = session.get(url, timeout=30)
            if resp.status_code in (403, 429):
                raise RuntimeError("status " + str(resp.status_code))
            body = resp.text.lower()
            if "just a moment" in body or "verify you are human" in body:
                raise RuntimeError("challenge page")
            return resp
        except Exception as e:
            if attempt == max_attempts:
                raise
            # Per-request rotation means the next call exits a fresh IP anyway.
            sleep_s = min(2 ** attempt, 30) + random.uniform(0, 1.5)
            print("retry", attempt, "after", round(sleep_s, 1), "s -", e)
            time.sleep(sleep_s)
```
Field note: Do not retry instantly on the same URL without backoff - a tight retry loop against Cloudflare raises your velocity score across the whole pool subnet and makes blocks worse, not better. Exponential backoff with jitter plus the fresh exit IP per retry is what actually clears transient challenges.
6
Use sticky sessions only for multi-step regional flows
Most job-board collection is stateless - each search page is independent, so per-request rotation is correct. But some flows need cookie continuity across two or three hops on one company: load the Glassdoor company overview, then its aggregate salary tab, then its rating summary. For those, pin a short sticky session so all hops share one exit IP and one cookie jar, then let it expire. Keep the lifetime short (5-10 minutes) and never reuse a stale session token for unrelated companies.
python
```
import requests

# Sticky session: same exit IP for a short multi-hop flow. Lifetime in minutes (<=30).
session_user = "aethyn-XXXXX-country-us-session-gd-acme-lifetime-10"
sticky_url = "http://" + session_user + ":PASSWORD@proxy.aethyn.io:5499"

s = requests.Session()
s.proxies = {"http": sticky_url, "https": sticky_url}
s.headers.update({"Accept-Language": "en-US,en;q=0.9"})

# Public AGGREGATE company pages only - overview, aggregate salary tab, rating summary.
overview = s.get("https://www.glassdoor.com/Overview/Working-at-Acme-EI_IE12345.htm", timeout=30)
salaries = s.get("https://www.glassdoor.com/Salary/Acme-Salaries-E12345.htm", timeout=30)
print(overview.status_code, salaries.status_code)
# Extract only aggregate ratings and salary ranges - never individual reviewer identities.
```
Field note: Use one session token per logical flow (per company), and pick a lifetime that comfortably covers the flow but no longer - a 10-minute session for a 3-request hop is fine. Reusing one long-lived sticky IP across hundreds of companies recreates exactly the single-IP velocity problem that per-request rotation exists to solve.

Best practices that keep scrapers reliable

Collect only public, non-personal aggregate data - job descriptions, salary ranges, star ratings, and company-level summaries. Never harvest individual reviewer or applicant identities, and never circumvent authentication or paywalls. Respect each site's robots.txt and Terms of Service, keep volume reasonable, and consult counsel for your jurisdiction.
Match your exit IP country and city to the labor market you are analyzing - Glassdoor salary bands and Indeed job sets are genuinely region-specific, so a mismatched geo produces a wrong dataset, not just a blocked one.
Verify the exit country programmatically at the start of every run and assert on it, so a misconfigured username fails loudly instead of silently scraping the wrong region.
Prefer embedded JSON blobs (Indeed's mosaic provider data / __NEXT_DATA__) over CSS selectors - the JSON schema is far more stable than the rotating class names, so your parser survives layout drift.
Treat every fetch as guilty until proven clean: detect Cloudflare challenge markers and soft-wall signups, and assert a non-empty plausible result set before persisting anything.
Reserve headless-browser fingerprinting for pages that actually throw challenges; run the bulk of collection through lightweight requests over the same proxy to keep throughput high and cost low.
Cache aggressively and re-fetch on a cadence that matches how fast the data actually changes - salary bands move slowly, job postings turn over daily - so you minimize total request volume against both sites.

Common mistakes that burn proxy budget

Running from datacenter/VPS IPs and blaming your header set when Cloudflare challenges you - the IP reputation is flagged before headers are ever evaluated, so no amount of header tuning fixes a datacenter exit.
Rotating exit countries while keeping a single hard-coded Accept-Language, producing contradictory locale/geo signals that raise your bot score on every request.
Trusting a 200 OK blindly - storing a Glassdoor page whose salary numbers are blurred behind the soft wall, or an Indeed page that returned zero jobs from a silent challenge.
Pinning one long-lived sticky session for the whole crawl, which concentrates velocity on a single IP and recreates the exact rate-limit problem per-request rotation is designed to avoid.
Building the parser on brittle CSS class names that Indeed rotates, instead of the stable embedded JSON, so the scraper silently returns empty after the next layout change.
Retrying blocked requests in a tight loop with no backoff, which spikes velocity across the pool and escalates temporary challenges into longer blocks.

Common questions about this scraper problem

Is scraping Glassdoor and Indeed legal?

Collecting public, non-personal data such as job descriptions, aggregate salary ranges, and public company ratings is generally treated differently from harvesting private or personal data, but the exact answer depends on your jurisdiction and the sites' Terms of Service. Stick to public aggregate data, never circumvent authentication or paywalls, keep your request volume reasonable, respect robots.txt and ToS, and consult a lawyer for your specific use case and country. This guide is about market analytics on aggregate data, not collecting information about individual people.

Why do I get a Cloudflare 'Just a moment...' page even with perfect headers?

Because Cloudflare scores the connection before it looks at your headers. JA3/JA4 TLS fingerprinting and IP-reputation checks happen at the edge, so a stock HTTP client or a datacenter IP gets challenged regardless of how realistic your header set is. The fix is a clean residential exit IP plus, for the pages that render content in JavaScript, a real browser fingerprint via Playwright over that same proxy.

Why are the Glassdoor salary numbers blurred or missing even though the page loaded?

That is Glassdoor's soft login wall. It returns a full 200 HTML shell but gates the actual salary figures and review bodies behind a signup prompt for anonymous or low-trust sessions. A clean residential IP from the correct region, a complete header set, and cookie continuity usually let the public aggregate figures render - but never try to defeat a genuine authentication or paywall to reach data that is not public.

Why does Indeed return different jobs than I expect?

Indeed serves locale-specific results and routes you to a country domain (indeed.com, de.indeed.com, uk.indeed.com) based on your exit IP and locale. If your proxy exits a different country than the market you are researching, you get that country's job set. Match the exit IP country and your Accept-Language to the target market.

Should I use per-request rotation or sticky sessions?

Per-request rotation is the default and correct choice for the vast majority of job-board collection, because each search page is independent and rotating spreads velocity so no single IP trips a rate limit. Use a short sticky session only for a multi-step flow on one company - for example loading a company overview then its aggregate salary tab - where cookie continuity matters. Keep sticky lifetimes short (5-10 minutes) and one token per flow.

Do I need a headless browser, or is requests enough?

Use requests for everything you can - it is far faster and cheaper - and fall back to Playwright only for pages that throw a Cloudflare challenge or render key content in JavaScript, such as Glassdoor salary tabs. A good pattern is to try the lightweight path first and escalate to a browser only when you detect a challenge marker.

How do I avoid storing bad data from soft blocks?

Never trust a 200 by itself. Check for challenge markers ('just a moment', 'verify you are human'), check for signup-wall placeholders, and assert that you parsed a non-empty, plausible result set before writing to your store. A zero-result job search or a round-number placeholder salary is almost always a block, not a real empty page.

What tier and port should I use for this?

Elite tier is the right choice for Glassdoor and Indeed because both are high-defense Cloudflare targets, and Elite gives you the cleanest residential reputation. The Elite endpoint is proxy.aethyn.io on port 5499, with country and city encoded in the username - for example aethyn-XXXXX-country-us-city-chicago.

How fast can I go without getting blocked?

Velocity is scored per IP, so with per-request rotation your effective ceiling is much higher than a single-IP crawl - but keep each worker to a few requests per second per market and use backoff on any block. If you start seeing challenges cluster, you are pushing the pool subnet too hard; slow down and let the exponential backoff with jitter do its job.

Can I collect individual reviews or applicant details?

No - and you should not want to. This guide is deliberately scoped to public, non-personal aggregate data: job descriptions, aggregate salary bands, and company-level ratings. Individual reviewer or applicant identities are personal data, collecting them raises serious privacy-law exposure (GDPR, CCPA, and others), and it is not what labor-market intelligence needs. Keep it aggregate.

Scraping Glassdoor and Indeed for Labor-Market Intelligence: A Practitioner's Guide to Public Hiring Data at Scale

Key takeaways

On this page

What goes wrong when this scraper fails

Challenges that make this hard to automate

How Aethyn residential proxies help here

How to implement this with residential proxies

Configure the Elite residential endpoint and verify geo-targeting

Send a complete, realistic header set and a matching Accept-Language

Use a real browser fingerprint for Cloudflare-challenged pages

Parse defensively and detect soft blocks before trusting data

Handle blocks and throttle with rotation and backoff

Use sticky sessions only for multi-step regional flows

Best practices that keep scrapers reliable

Common questions about this scraper problem

Solve this with Aethyn residential proxies

Scraping Glassdoor and Indeed for Labor-Market Intelligence: A Practitioner's Guide to Public Hiring Data at Scale

Key takeaways

On this page

What goes wrong when this scraper fails

Challenges that make this hard to automate

How Aethyn residential proxies help here

How to implement this with residential proxies

Configure the Elite residential endpoint and verify geo-targeting

Send a complete, realistic header set and a matching Accept-Language

Use a real browser fingerprint for Cloudflare-challenged pages

Parse defensively and detect soft blocks before trusting data

Handle blocks and throttle with rotation and backoff

Use sticky sessions only for multi-step regional flows

Best practices that keep scrapers reliable

Common questions about this scraper problem

Solve this with Aethyn residential proxies

Key takeaways

On this page

What goes wrong when this scraper fails

Challenges that make this hard to automate

How Aethyn residential proxies help here

How to implement this with residential proxies

Configure the Elite residential endpoint and verify geo-targeting

Send a complete, realistic header set and a matching Accept-Language

Use a real browser fingerprint for Cloudflare-challenged pages

Parse defensively and detect soft blocks before trusting data

Handle blocks and throttle with rotation and backoff

Use sticky sessions only for multi-step regional flows

Best practices that keep scrapers reliable

Common questions about this scraper problem

Related guides & resources

Solve this with Aethyn residential proxies

Key takeaways

On this page

What goes wrong when this scraper fails

Challenges that make this hard to automate

How Aethyn residential proxies help here

How to implement this with residential proxies

Configure the Elite residential endpoint and verify geo-targeting

Send a complete, realistic header set and a matching Accept-Language

Use a real browser fingerprint for Cloudflare-challenged pages

Parse defensively and detect soft blocks before trusting data

Handle blocks and throttle with rotation and backoff

Use sticky sessions only for multi-step regional flows

Best practices that keep scrapers reliable

Common questions about this scraper problem

Related guides & resources

Solve this with Aethyn residential proxies