Scrape Public TikTok, Twitch & Spotify Analytics at Scale

What goes wrong when this scraper fails

Influencer marketers, trend desks, and media researchers need three families of public numbers: TikTok video/hashtag/creator view and engagement counts, Twitch channel and live-stream concurrent-viewer stats, and Spotify track/artist/playlist popularity. There is no unified, generous API that hands these over at research volume, so teams end up reading the public web and web-player endpoints directly. The moment you go past a handful of manual lookups, each platform's anti-automation posture kicks in and the data either stops flowing or - worse - comes back subtly wrong.

Why this failure mode happens

Each platform protects a different asset with a different mechanism. TikTok treats its web and mobile APIs as high-value and signs every request with X-Bogus (and device/session fingerprints), so an unsigned or replayed request is rejected outright. Twitch routes gql.twitch.tv through Cloudflare and applies per-IP rate limits, so bursts from one address get 429'd or challenged. Spotify's open web player mints a short-lived anonymous access token that is itself behind bot detection, and the downstream endpoints are rate-limited per token and per IP. On top of all that, all three personalize trending and recommendation surfaces by geography, so the same URL genuinely returns different numbers depending on where the exit lands.

Challenges that make this hard to automate

TikTok signs requests with X-Bogus (and rotating signature schemes) plus mobile-client fingerprinting and device checks - forged signatures work for days, then silently break, poisoning your dataset before you notice.
Twitch's GraphQL endpoint is fronted by Cloudflare with per-IP rate limits and TLS-fingerprint checks, so naive concurrency turns into 429s and JS challenges fast.
Spotify's anonymous web-player token is short-lived and itself gated by anti-bot, so token acquisition - not the data call - is usually the thing that gets blocked.
Geo-personalization means an exit in the wrong market returns the wrong 'trending' set, so infrastructure errors masquerade as data insights.
Both soft blocks and schema drift fail with no HTTP-level signal: a soft block returns 200 with an empty or truncated payload, and layout/API churn (renamed JSON keys, moved rehydration blobs) returns a full body with the metric relocated - either one quietly corrupts dashboards for weeks if you trust the status code.

Approaches that usually fail

Hand-forging X-Bogus/signature parameters from reverse-engineered JS - fast when it works, but every TikTok deploy can invalidate the algorithm and there is no warning, so you ship bad numbers.
A pool of cheap datacenter IPs with per-request rotation - fine for low-value targets, but Cloudflare and TikTok flag datacenter ASNs immediately and geo-personalization is unreliable.
A single headless browser farm hitting everything - correct rendering but brutally slow and expensive per data point, and still gets fingerprinted if every session shares one IP.
Buying whatever third-party 'social API' aggregator is available - convenient until it silently rate-limits you, lags real-time viewer counts, or drops a platform mid-contract.

When residential proxies fix this — and when they cannot

Residential and mobile exits in the correct target market solve two problems at once: they carry the ASN reputation these platforms expect, and they place your request inside the geography that determines which trending/popularity numbers you actually get back. Per-request rotation keeps consecutive calls off the same address so you never stack velocity on one IP and trip Cloudflare or TikTok's rate limits, while sticky sessions let a browser-rendered TikTok flow that needs a stable device/session identity stay on one exit for its short lifetime. Combined with a complete, realistic header and TLS profile, a geo-correct rotating exit is what turns "works for ten lookups then dies" into steady, accurate collection at volume.

How Aethyn residential proxies help here

Aethyn's Elite tier gives you clean residential and mobile exits with precise country (and city) targeting on a single endpoint - proxy.aethyn.io:5499 - which is exactly what geo-personalized platforms like TikTok, Twitch, and Spotify demand for correct numbers. You control rotation entirely through the username: leave the session token off for per-request rotation, add one for a short sticky burst when a browser flow needs a stable identity.

Country- and city-level exit selection (-country-us, -city-chicago) so trending and popularity data reflects the market you're actually researching, not a random datacenter's locale.
Per-request rotation by default - a fresh exit on every call keeps velocity off any single IP, which is the signal Cloudflare and TikTok rate-limit on.
Sticky sessions up to 30 minutes (-session-TOKEN-lifetime-10) for TikTok Playwright flows that need device/session stability across a page render.
Residential and mobile ASN reputation on the Elite tier, so requests don't get flagged the way datacenter ranges do at Twitch's Cloudflare edge.
One endpoint and one credential format across all three platforms - no per-provider gateway juggling, so your collectors stay simple.
High concurrency headroom so you can fan out research-volume collection across markets without self-inflicted rate limits.

How to implement this with residential proxies

1
Stand up a geo-correct, per-request rotating exit pool
Everything downstream depends on requests leaving from the right country with a fresh exit each time. Build the proxy URL from the Aethyn username format and leave the session token off so every call rotates. Pick the exit country to match the market whose trending/popularity numbers you want - a US creator's real reach only shows through a -country-us exit. Verify the exit actually landed where you asked before you trust any metric.
python
```
import requests

# Elite tier -> port 5499. Per-request rotation = NO session token.
PROXY_TMPL = "http://aethyn-XXXXX-country-{cc}:PASSWORD@proxy.aethyn.io:5499"

def proxied(cc: str) -> dict:
    url = PROXY_TMPL.format(cc=cc)
    return {"http": url, "https": url}

# Confirm the exit is really in the target market before collecting.
r = requests.get(
    "https://ipinfo.io/json",
    proxies=proxied("us"),
    timeout=15,
)
info = r.json()
assert info.get("country") == "US", f"exit landed in {info.get('country')}, not US"
print(info["ip"], info["country"], info.get("city"))
```
Field note: Assert the exit country on a cheap IP-echo call at the start of every collection run. Geo-personalized platforms fail silently - a GB exit against a US trend query returns valid-looking numbers that are simply wrong, and no HTTP error ever tells you.

Pull Twitch public stream and channel stats via the public GraphQL client

Twitch's own web player talks to gql.twitch.tv with a well-known public Client-ID; you can send a raw query for public aggregate fields - live concurrent viewers, stream type, game, follower totals. Route it through a rotating exit so consecutive channel lookups never stack on one IP, and send a complete header set. Cloudflare here rate-limits per IP and inspects TLS fingerprints, so a bare requests call with one User-Agent and no Accept-Language is a classic tell.

python

import requests

TWITCH_GQL = "https://gql.twitch.tv/gql"
# Public Client-ID shipped in Twitch's own browser web player.
CLIENT_ID = "kimne78kx3ncx6brgo4mv6wki5h1ko"

QUERY = """
query($login: String!) {
  user(login: $login) {
    displayName
    followers { totalCount }
    stream { id type viewersCount game { name } }
  }
}
"""

def channel_stats(login: str, cc: str = "us") -> dict:
    proxy = f"http://aethyn-XXXXX-country-{cc}:PASSWORD@proxy.aethyn.io:5499"
    headers = {
        "Client-ID": CLIENT_ID,
        "Accept": "application/json",
        "Accept-Language": "en-US,en;q=0.9",
        "Content-Type": "application/json",
        "User-Agent": ("Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
                       "AppleWebKit/537.36 (KHTML, like Gecko) "
                       "Chrome/125.0.0.0 Safari/537.36"),
    }
    resp = requests.post(
        TWITCH_GQL,
        json={"query": QUERY, "variables": {"login": login}},
        headers=headers,
        proxies={"https": proxy},
        timeout=20,
    )
    resp.raise_for_status()
    user = resp.json()["data"]["user"]
    stream = user.get("stream")
    return {
        "channel": user["displayName"],
        "followers": user["followers"]["totalCount"],
        "live": stream is not None,
        "viewers": stream["viewersCount"] if stream else 0,
        "game": stream["game"]["name"] if stream and stream.get("game") else None,
    }

print(channel_stats("somepublicchannel"))

Field note: Distinguish live-with-zero-viewers from offline explicitly. stream is null means offline; a non-null stream with viewersCount 0 is a real live number. Collapsing both to 0 quietly destroys the signal a trend desk actually cares about.

Collect public TikTok hashtag/video metrics by rendering, not forging signatures

TikTok signs requests with X-Bogus and checks device/session fingerprints, and any hand-rolled signature you reverse-engineer will break on a future deploy without warning. The durable approach is to let a real browser produce a valid signed request for you: drive Playwright through the proxy, load the public hashtag or video page, and read the aggregate counts out of the embedded rehydration blob. Because a render is one stable session with device state, pin it to a sticky exit for its short lifetime.

python

import json
from playwright.sync_api import sync_playwright

# Sticky session so the whole render stays on one exit (device/session stability).
PROXY = {
    "server": "http://proxy.aethyn.io:5499",
    "username": "aethyn-XXXXX-country-us-session-tt42-lifetime-10",
    "password": "PASSWORD",
}

def hashtag_metrics(tag: str) -> dict:
    with sync_playwright() as p:
        browser = p.chromium.launch(proxy=PROXY, headless=True)
        ctx = browser.new_context(
            locale="en-US",
            user_agent=("Mozilla/5.0 (iPhone; CPU iPhone OS 17_4 like Mac OS X) "
                        "AppleWebKit/605.1.15 (KHTML, like Gecko) "
                        "Version/17.4 Mobile/15E148 Safari/604.1"),
        )
        page = ctx.new_page()
        page.goto(f"https://www.tiktok.com/tag/{tag}", wait_until="networkidle")
        raw = page.locator("#__UNIVERSAL_DATA_FOR_REHYDRATION__").inner_text()
        browser.close()

    scope = json.loads(raw)["__DEFAULT_SCOPE__"]
    info = scope["webapp.challenge-detail"]["challengeInfo"]
    stats = info["stats"]
    return {
        "tag": tag,
        "video_views": int(stats["viewCount"]),
        "video_count": int(stats["videoCount"]),
    }

print(hashtag_metrics("marketing"))

Field note: Let the browser mint the signature; never persist a reverse-engineered X-Bogus implementation as your primary path. It will pass for a few days after a TikTok deploy and then start returning malformed or empty payloads - which is exactly how a dataset gets silently poisoned before anyone spots it.

Read Spotify track/artist/playlist popularity via the public web-player token

Spotify's open web player fetches an anonymous access token that is itself the bot-detection choke point, so acquire the token through a clean geo-correct exit first, then reuse it across a batch of public catalog lookups. The public /v1/tracks, /v1/artists, and /v1/playlists endpoints return the popularity score and follower/aggregate fields you need. Pass market so popularity reflects the region you're researching, and refresh the token when it expires rather than hammering the token endpoint per call.

python

import requests

UA = ("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
      "(KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36")

def spotify_token(cc: str = "us") -> str:
    proxy = f"http://aethyn-XXXXX-country-{cc}:PASSWORD@proxy.aethyn.io:5499"
    r = requests.get(
        "https://open.spotify.com/get_access_token",
        params={"reason": "transport", "productType": "web-player"},
        headers={"User-Agent": UA, "App-Platform": "WebPlayer",
                 "Accept-Language": "en-US,en;q=0.9"},
        proxies={"https": proxy},
        timeout=15,
    )
    r.raise_for_status()
    return r.json()["accessToken"]

def track_popularity(track_id: str, token: str, cc: str = "us") -> dict:
    proxy = f"http://aethyn-XXXXX-country-{cc}:PASSWORD@proxy.aethyn.io:5499"
    r = requests.get(
        f"https://api.spotify.com/v1/tracks/{track_id}",
        params={"market": cc.upper()},
        headers={"Authorization": f"Bearer {token}", "User-Agent": UA},
        proxies={"https": proxy},
        timeout=15,
    )
    r.raise_for_status()
    d = r.json()
    return {
        "name": d["name"],
        "popularity": d["popularity"],       # 0-100 aggregate score
        "artists": [a["name"] for a in d["artists"]],
    }

tok = spotify_token("us")
print(track_popularity("11dFghVXANMlKmJXsNCbNl", tok))

Field note: Cache the token and reuse it across a batch; the token endpoint is more aggressively bot-gated than the data endpoints. When a data call returns 401, refresh once and retry - but if the refresh itself starts failing, rotate the exit before retrying, because that means the IP is what got flagged.

Detect soft blocks and schema drift with defensive parsing

The most dangerous failure on all three platforms is a 200 OK that carries an empty, truncated, or restructured payload - no exception, no bad status, just wrong data flowing into your warehouse. Wrap every extraction in a helper that walks the expected path, fails loudly on a missing key (layout drift) or an empty node (soft block), and refuses to persist either. Treat 'zero results' as guilty until proven innocent.

python

class SoftBlock(Exception):
    pass

def dig(payload, *path):
    """Walk a nested payload; raise loudly instead of returning silent garbage."""
    node = payload
    for key in path:
        if not isinstance(node, (dict, list)) or (
            isinstance(node, dict) and key not in node
        ):
            raise SoftBlock(f"missing key '{key}' - layout drift or block at {path}")
        node = node[key]
    if node in (None, "", [], {}, 0):
        raise SoftBlock(f"empty value at {path} - treat as block, do not persist")
    return node

def safe_store(record: dict, writer):
    # Never write a record whose core metric is missing or zero-by-block.
    if not record.get("video_views") and not record.get("viewers") \
       and not record.get("popularity"):
        raise SoftBlock(f"no live metric in record - dropping: {record}")
    writer(record)

Field note: Log the raw response length and HTTP status alongside every SoftBlock you raise. When TikTok or Spotify silently changes a JSON key, the length shift is your earliest warning - far earlier than a downstream analyst noticing the trend line went flat.

Handle throttling with backoff, rotation, and honest concurrency limits

Cloudflare (Twitch) and per-token limits (Spotify) answer overload with 429/403; the right response is exponential backoff with jitter, and - critically - a fresh exit on retry, because the address is usually what got throttled. Keep per-market concurrency modest and let per-request rotation spread load across exits rather than pushing one IP harder. Cap total attempts so a genuinely blocked target fails fast instead of grinding.

python

import random
import time
import requests

def with_backoff(make_request, attempts: int = 5):
    """make_request() must build a NEW request each call so the exit rotates."""
    for i in range(attempts):
        resp = make_request()
        if resp.status_code in (429, 403):
            wait = min((2 ** i) + random.random(), 30)
            time.sleep(wait)
            continue
        resp.raise_for_status()
        return resp
    raise RuntimeError("throttled after retries - cool down this market and rotate")

# Each call builds a fresh proxied request -> per-request rotation on retry.
def fetch_twitch(login: str, cc: str = "us"):
    proxy = f"http://aethyn-XXXXX-country-{cc}:PASSWORD@proxy.aethyn.io:5499"
    return with_backoff(lambda: requests.post(
        "https://gql.twitch.tv/gql",
        json={"query": "{ __typename }"},
        headers={"Client-ID": "kimne78kx3ncx6brgo4mv6wki5h1ko"},
        proxies={"https": proxy},
        timeout=20,
    ))

Field note: On 429/403, rotate the exit before you sleep-and-retry, not after. Backing off on the same flagged IP just wastes the delay; the whole point of per-request rotation is that your retry leaves from a clean address the throttle has never seen.

Best practices that keep scrapers reliable

Collect only public, non-personal AGGREGATE metrics - view counts, follower totals, concurrent viewers, trending tags, public playlist/popularity data. Never scrape private profiles, DMs, or anything behind a login, respect each platform's Terms of Service, keep volume reasonable, and consult counsel for your jurisdiction.
Match the exit country to the market whose numbers you want; on geo-personalized surfaces the exit location is part of data correctness, not just deliverability.
Prefer each platform's own documented public path (Twitch web Client-ID, Spotify anonymous web-player token, browser-rendered TikTok) over reverse-engineered private mobile APIs that break without warning.
Render TikTok in a real browser to get valid X-Bogus signing rather than persisting a hand-forged signature that silently rots after a deploy.
Assert a non-empty, well-shaped result before writing anything - a 200 with an empty metric object is a soft block, not a real zero.
Rotate per request by default and reserve sticky sessions for browser flows that genuinely need device/session stability across a render.
Log raw response length and status with every parse so schema drift and soft blocks surface as alerts, not as quietly wrong dashboards weeks later.

Common mistakes that burn proxy budget

Hand-forging TikTok's X-Bogus/signature and treating it as a stable primary path - it works for days after a deploy, then returns malformed or empty payloads that poison the dataset.
Ignoring exit geography and reading 'trending' from whatever country the IP happened to land in, then reporting personalized-but-wrong numbers as insights.
Persisting HTTP 200 responses with empty or truncated bodies because the status code looked fine, silently flat-lining trend charts.
Backing off and retrying on the same throttled IP instead of rotating to a fresh exit first, wasting the delay and staying blocked.
Refetching the Spotify anonymous token on every call - the token endpoint is more bot-gated than the data endpoints, so it gets you blocked faster than the data ever would.
Sending a bare User-Agent with no Accept-Language or full header set to Cloudflare-fronted Twitch, which is an obvious automation tell that draws challenges.

Common questions about this scraper problem

Is collecting this data legal, and does it violate the platforms' Terms of Service?

This guide is strictly for PUBLIC, non-personal AGGREGATE metrics - view counts, follower totals, concurrent viewers, trending tags, and public playlist popularity - collected read-only. Do not scrape private user data, circumvent any login, or automate fake engagement. Each platform's Terms of Service may restrict automated access regardless of data sensitivity, so keep volume reasonable, honor robots and ToS, and consult counsel for your specific jurisdiction and use case before running at scale.

Why can't I just reimplement TikTok's X-Bogus signature in Python?

You can, and it will work - until TikTok's next web deploy changes the algorithm, at which point it fails silently rather than erroring. Because the failure returns malformed or empty payloads instead of an HTTP error, a forged-signature pipeline poisons your dataset before anyone notices. Rendering the page in a real browser lets TikTok's own JS produce a valid signature, which survives their deploys.

Do I really need proxies in the target country, or is any working exit fine?

Country matters for correctness, not just access. TikTok, Twitch, and Spotify all personalize trending and 'popular' surfaces by geography, so a track's popularity or a hashtag's featured set genuinely differs by market. An exit in the wrong country returns valid-looking numbers that answer a different question than the one you asked.

Per-request rotation or sticky sessions - which should I use?

Default to per-request rotation so consecutive calls never stack velocity on one IP, which is what Cloudflare and TikTok rate-limit on. Use a sticky session only for browser-rendered TikTok flows that need a stable device/session identity across a single render, and keep the lifetime short (up to 30 minutes). Twitch and Spotify JSON calls should rotate every request.

Twitch returns 429 or a Cloudflare challenge - what's the fix?

Two things are usually wrong: too much velocity on one IP and an incomplete request fingerprint. Rotate to a fresh exit on each retry, add exponential backoff with jitter, and send a complete header set including Accept-Language and a realistic User-Agent. Residential/mobile ASN reputation from the Elite tier also matters, since Cloudflare treats datacenter ranges more harshly.

The Spotify token call keeps getting blocked even though data calls work - why?

The anonymous web-player token endpoint is the bot-detection choke point; it's more aggressively gated than the downstream /v1 data endpoints. Acquire the token through a clean geo-correct exit, cache it, and reuse it across a batch instead of refetching per call. If token acquisition itself starts failing, rotate the exit - the IP, not your headers, is what got flagged.

I'm getting HTTP 200 but the metrics are empty or zero. Is that a real zero?

Almost never. A 200 with an empty or truncated payload is the classic soft block, and on TikTok/Spotify it can also mean a renamed JSON key after a layout change. Assert a non-empty, well-shaped result before persisting, and log the raw response length so schema drift and soft blocks surface as alerts rather than quietly wrong dashboards.

How do I tell a soft block apart from a genuine layout/schema change?

Instrument both. A soft block tends to return a short or challenge-shaped body across many targets at once and often clears after rotating the exit; a schema change returns a full-length body where a specific expected key is missing or moved, consistently, on a clean exit. Logging response length plus the exact missing-key path lets you distinguish 'rotate the IP' from 'fix the parser.'

What concurrency can I safely run across these platforms?

There's no universal number - it depends on how many exits you rotate across and how each platform's per-IP limits behave that week. Practically, spread load with per-request rotation, keep per-market concurrency modest, and back off the moment 429/403 rates climb. It's cheaper to run steady and correct than to burst, get flagged, and re-collect.

Can I use these public numbers to measure or drive engagement automation?

This workflow is read-only analytics for influencer sizing, trend research, and media analysis. It does not cover - and you should not use it for - fake engagement, bot followers, auto-views, or any manipulation of the platforms. Collecting public aggregate metrics for measurement is a different activity from automating interactions, and only the former is in scope here.

What goes wrong when this scraper fails

Why this failure mode happens

Challenges that make this hard to automate

TikTok signs requests with X-Bogus (and rotating signature schemes) plus mobile-client fingerprinting and device checks - forged signatures work for days, then silently break, poisoning your dataset before you notice.
Twitch's GraphQL endpoint is fronted by Cloudflare with per-IP rate limits and TLS-fingerprint checks, so naive concurrency turns into 429s and JS challenges fast.
Spotify's anonymous web-player token is short-lived and itself gated by anti-bot, so token acquisition - not the data call - is usually the thing that gets blocked.
Geo-personalization means an exit in the wrong market returns the wrong 'trending' set, so infrastructure errors masquerade as data insights.
Both soft blocks and schema drift fail with no HTTP-level signal: a soft block returns 200 with an empty or truncated payload, and layout/API churn (renamed JSON keys, moved rehydration blobs) returns a full body with the metric relocated - either one quietly corrupts dashboards for weeks if you trust the status code.

Approaches that usually fail

Hand-forging X-Bogus/signature parameters from reverse-engineered JS - fast when it works, but every TikTok deploy can invalidate the algorithm and there is no warning, so you ship bad numbers.
A pool of cheap datacenter IPs with per-request rotation - fine for low-value targets, but Cloudflare and TikTok flag datacenter ASNs immediately and geo-personalization is unreliable.
A single headless browser farm hitting everything - correct rendering but brutally slow and expensive per data point, and still gets fingerprinted if every session shares one IP.
Buying whatever third-party 'social API' aggregator is available - convenient until it silently rate-limits you, lags real-time viewer counts, or drops a platform mid-contract.

When residential proxies fix this — and when they cannot

How Aethyn residential proxies help here

Country- and city-level exit selection (-country-us, -city-chicago) so trending and popularity data reflects the market you're actually researching, not a random datacenter's locale.
Per-request rotation by default - a fresh exit on every call keeps velocity off any single IP, which is the signal Cloudflare and TikTok rate-limit on.
Sticky sessions up to 30 minutes (-session-TOKEN-lifetime-10) for TikTok Playwright flows that need device/session stability across a page render.
Residential and mobile ASN reputation on the Elite tier, so requests don't get flagged the way datacenter ranges do at Twitch's Cloudflare edge.
One endpoint and one credential format across all three platforms - no per-provider gateway juggling, so your collectors stay simple.
High concurrency headroom so you can fan out research-volume collection across markets without self-inflicted rate limits.

How to implement this with residential proxies

1
Stand up a geo-correct, per-request rotating exit pool
Everything downstream depends on requests leaving from the right country with a fresh exit each time. Build the proxy URL from the Aethyn username format and leave the session token off so every call rotates. Pick the exit country to match the market whose trending/popularity numbers you want - a US creator's real reach only shows through a -country-us exit. Verify the exit actually landed where you asked before you trust any metric.
python
```
import requests

# Elite tier -> port 5499. Per-request rotation = NO session token.
PROXY_TMPL = "http://aethyn-XXXXX-country-{cc}:PASSWORD@proxy.aethyn.io:5499"

def proxied(cc: str) -> dict:
    url = PROXY_TMPL.format(cc=cc)
    return {"http": url, "https": url}

# Confirm the exit is really in the target market before collecting.
r = requests.get(
    "https://ipinfo.io/json",
    proxies=proxied("us"),
    timeout=15,
)
info = r.json()
assert info.get("country") == "US", f"exit landed in {info.get('country')}, not US"
print(info["ip"], info["country"], info.get("city"))
```
Field note: Assert the exit country on a cheap IP-echo call at the start of every collection run. Geo-personalized platforms fail silently - a GB exit against a US trend query returns valid-looking numbers that are simply wrong, and no HTTP error ever tells you.

Pull Twitch public stream and channel stats via the public GraphQL client

python

import requests

TWITCH_GQL = "https://gql.twitch.tv/gql"
# Public Client-ID shipped in Twitch's own browser web player.
CLIENT_ID = "kimne78kx3ncx6brgo4mv6wki5h1ko"

QUERY = """
query($login: String!) {
  user(login: $login) {
    displayName
    followers { totalCount }
    stream { id type viewersCount game { name } }
  }
}
"""

def channel_stats(login: str, cc: str = "us") -> dict:
    proxy = f"http://aethyn-XXXXX-country-{cc}:PASSWORD@proxy.aethyn.io:5499"
    headers = {
        "Client-ID": CLIENT_ID,
        "Accept": "application/json",
        "Accept-Language": "en-US,en;q=0.9",
        "Content-Type": "application/json",
        "User-Agent": ("Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
                       "AppleWebKit/537.36 (KHTML, like Gecko) "
                       "Chrome/125.0.0.0 Safari/537.36"),
    }
    resp = requests.post(
        TWITCH_GQL,
        json={"query": QUERY, "variables": {"login": login}},
        headers=headers,
        proxies={"https": proxy},
        timeout=20,
    )
    resp.raise_for_status()
    user = resp.json()["data"]["user"]
    stream = user.get("stream")
    return {
        "channel": user["displayName"],
        "followers": user["followers"]["totalCount"],
        "live": stream is not None,
        "viewers": stream["viewersCount"] if stream else 0,
        "game": stream["game"]["name"] if stream and stream.get("game") else None,
    }

print(channel_stats("somepublicchannel"))

Collect public TikTok hashtag/video metrics by rendering, not forging signatures

python

import json
from playwright.sync_api import sync_playwright

# Sticky session so the whole render stays on one exit (device/session stability).
PROXY = {
    "server": "http://proxy.aethyn.io:5499",
    "username": "aethyn-XXXXX-country-us-session-tt42-lifetime-10",
    "password": "PASSWORD",
}

def hashtag_metrics(tag: str) -> dict:
    with sync_playwright() as p:
        browser = p.chromium.launch(proxy=PROXY, headless=True)
        ctx = browser.new_context(
            locale="en-US",
            user_agent=("Mozilla/5.0 (iPhone; CPU iPhone OS 17_4 like Mac OS X) "
                        "AppleWebKit/605.1.15 (KHTML, like Gecko) "
                        "Version/17.4 Mobile/15E148 Safari/604.1"),
        )
        page = ctx.new_page()
        page.goto(f"https://www.tiktok.com/tag/{tag}", wait_until="networkidle")
        raw = page.locator("#__UNIVERSAL_DATA_FOR_REHYDRATION__").inner_text()
        browser.close()

    scope = json.loads(raw)["__DEFAULT_SCOPE__"]
    info = scope["webapp.challenge-detail"]["challengeInfo"]
    stats = info["stats"]
    return {
        "tag": tag,
        "video_views": int(stats["viewCount"]),
        "video_count": int(stats["videoCount"]),
    }

print(hashtag_metrics("marketing"))

Read Spotify track/artist/playlist popularity via the public web-player token

python

import requests

UA = ("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
      "(KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36")

def spotify_token(cc: str = "us") -> str:
    proxy = f"http://aethyn-XXXXX-country-{cc}:PASSWORD@proxy.aethyn.io:5499"
    r = requests.get(
        "https://open.spotify.com/get_access_token",
        params={"reason": "transport", "productType": "web-player"},
        headers={"User-Agent": UA, "App-Platform": "WebPlayer",
                 "Accept-Language": "en-US,en;q=0.9"},
        proxies={"https": proxy},
        timeout=15,
    )
    r.raise_for_status()
    return r.json()["accessToken"]

def track_popularity(track_id: str, token: str, cc: str = "us") -> dict:
    proxy = f"http://aethyn-XXXXX-country-{cc}:PASSWORD@proxy.aethyn.io:5499"
    r = requests.get(
        f"https://api.spotify.com/v1/tracks/{track_id}",
        params={"market": cc.upper()},
        headers={"Authorization": f"Bearer {token}", "User-Agent": UA},
        proxies={"https": proxy},
        timeout=15,
    )
    r.raise_for_status()
    d = r.json()
    return {
        "name": d["name"],
        "popularity": d["popularity"],       # 0-100 aggregate score
        "artists": [a["name"] for a in d["artists"]],
    }

tok = spotify_token("us")
print(track_popularity("11dFghVXANMlKmJXsNCbNl", tok))

Detect soft blocks and schema drift with defensive parsing

python

class SoftBlock(Exception):
    pass

def dig(payload, *path):
    """Walk a nested payload; raise loudly instead of returning silent garbage."""
    node = payload
    for key in path:
        if not isinstance(node, (dict, list)) or (
            isinstance(node, dict) and key not in node
        ):
            raise SoftBlock(f"missing key '{key}' - layout drift or block at {path}")
        node = node[key]
    if node in (None, "", [], {}, 0):
        raise SoftBlock(f"empty value at {path} - treat as block, do not persist")
    return node

def safe_store(record: dict, writer):
    # Never write a record whose core metric is missing or zero-by-block.
    if not record.get("video_views") and not record.get("viewers") \
       and not record.get("popularity"):
        raise SoftBlock(f"no live metric in record - dropping: {record}")
    writer(record)

Handle throttling with backoff, rotation, and honest concurrency limits

python

import random
import time
import requests

def with_backoff(make_request, attempts: int = 5):
    """make_request() must build a NEW request each call so the exit rotates."""
    for i in range(attempts):
        resp = make_request()
        if resp.status_code in (429, 403):
            wait = min((2 ** i) + random.random(), 30)
            time.sleep(wait)
            continue
        resp.raise_for_status()
        return resp
    raise RuntimeError("throttled after retries - cool down this market and rotate")

# Each call builds a fresh proxied request -> per-request rotation on retry.
def fetch_twitch(login: str, cc: str = "us"):
    proxy = f"http://aethyn-XXXXX-country-{cc}:PASSWORD@proxy.aethyn.io:5499"
    return with_backoff(lambda: requests.post(
        "https://gql.twitch.tv/gql",
        json={"query": "{ __typename }"},
        headers={"Client-ID": "kimne78kx3ncx6brgo4mv6wki5h1ko"},
        proxies={"https": proxy},
        timeout=20,
    ))

Best practices that keep scrapers reliable

Collect only public, non-personal AGGREGATE metrics - view counts, follower totals, concurrent viewers, trending tags, public playlist/popularity data. Never scrape private profiles, DMs, or anything behind a login, respect each platform's Terms of Service, keep volume reasonable, and consult counsel for your jurisdiction.
Match the exit country to the market whose numbers you want; on geo-personalized surfaces the exit location is part of data correctness, not just deliverability.
Prefer each platform's own documented public path (Twitch web Client-ID, Spotify anonymous web-player token, browser-rendered TikTok) over reverse-engineered private mobile APIs that break without warning.
Render TikTok in a real browser to get valid X-Bogus signing rather than persisting a hand-forged signature that silently rots after a deploy.
Assert a non-empty, well-shaped result before writing anything - a 200 with an empty metric object is a soft block, not a real zero.
Rotate per request by default and reserve sticky sessions for browser flows that genuinely need device/session stability across a render.
Log raw response length and status with every parse so schema drift and soft blocks surface as alerts, not as quietly wrong dashboards weeks later.

Common mistakes that burn proxy budget

Hand-forging TikTok's X-Bogus/signature and treating it as a stable primary path - it works for days after a deploy, then returns malformed or empty payloads that poison the dataset.
Ignoring exit geography and reading 'trending' from whatever country the IP happened to land in, then reporting personalized-but-wrong numbers as insights.
Persisting HTTP 200 responses with empty or truncated bodies because the status code looked fine, silently flat-lining trend charts.
Backing off and retrying on the same throttled IP instead of rotating to a fresh exit first, wasting the delay and staying blocked.
Refetching the Spotify anonymous token on every call - the token endpoint is more bot-gated than the data endpoints, so it gets you blocked faster than the data ever would.
Sending a bare User-Agent with no Accept-Language or full header set to Cloudflare-fronted Twitch, which is an obvious automation tell that draws challenges.