How to Scale Python Web Scrapers

What goes wrong when this scraper fails

The prototype works: a for-loop over a few hundred URLs with requests. Then the target list grows to millions, and everything that was fine becomes a bottleneck — the synchronous loop would take days, bumping the thread count gets you blocked, and the first run that buffers all results in a list dies with a MemoryError at 80%. Scaling is not 'the same script but bigger'; it is a different architecture built around concurrency, failure isolation, and backpressure.

Why this failure mode happens

A synchronous request spends almost all its time waiting on the network, so a single-threaded loop leaves the CPU idle 95% of the time. The naive fix — many threads or processes hammering through one IP — trips per-IP velocity limits and gets you blocked. And without bounded concurrency, retries, and streaming, error rates and memory both climb until the job collapses at exactly the volume you built it for.

Challenges that make this hard to automate

• I/O-bound waiting that leaves a synchronous scraper mostly idle
• Per-IP velocity limits that break naive thread/process fan-out
• Unbounded concurrency exhausting file descriptors, sockets, and RAM
• Transient failures (timeouts, 5xx, 429) that cascade without retries
• asyncio.gather aborting an entire batch when one task raises

Approaches that usually fail

• A synchronous requests loop — rock-solid and far too slow at volume
• Threaded loops on one IP — faster, but a direct path to bans
• Ad-hoc multiprocessing — heavy, memory-hungry, and awkward to coordinate
• No retry or backpressure strategy — small error rates snowball into failed runs

When residential proxies fix this — and when they cannot

Concurrency only helps if the target tolerates it, and it tolerates it when requests are spread thin across many IPs. Routing a high-concurrency async client through a large rotating residential pool keeps each exit IP's velocity low, so you can raise in-flight requests for throughput without raising block rates. Per-request rotation pairs naturally with async — every coroutine effectively gets its own fresh IP.

How Aethyn residential proxies help here

High-volume crawling needs a pool that is large, cheap per byte, and rotates automatically so your concurrency model stays simple. That is the Premium tier's sweet spot.

• A large Premium pool sized for high-volume, cost-efficient crawls
• Automatic per-request rotation that pairs cleanly with async clients
• Sticky sessions for the workers that genuinely need IP continuity
• Country targeting per job from the same endpoint, no re-plumbing
• Per-byte metering so a millions-of-pages crawl stays cost-predictable

How to implement this with residential proxies

1. 1

   Move to async I/O and reuse one client

   Async lets a single process keep hundreds of requests in flight while they wait on the network. Create one AsyncClient (so connections are pooled and reused) and route it through the residential endpoint — spinning up a client per request throws away connection reuse and tanks throughput.

   Python (httpx async)Copy

   import asyncio, httpx
   
   PROXY = "http://aethyn-XXXXX:PASSWORD@proxy.aethyn.io:2099"
   LIMITS = httpx.Limits(max_connections=200, max_keepalive_connections=50)
   
   async def fetch(client, url):
       r = await client.get(url, timeout=30)
       return url, r.status_code
   
   async def main(urls):
       async with httpx.AsyncClient(proxy=PROXY, limits=LIMITS) as client:
           return await asyncio.gather(*(fetch(client, u) for u in urls),
                                       return_exceptions=True)
   
   print(asyncio.run(main(["https://example.com"] * 5)))

   Field note: return_exceptions=True is not optional at scale. A bare asyncio.gather cancels every sibling task the instant one raises — so a single bad URL can wipe out the other 199 in-flight requests. Collect exceptions and handle them per item instead.

2. 2

   Bound concurrency with a semaphore

   Never let concurrency run unbounded — that is how you exhaust file descriptors, sockets, and memory, and how you overwhelm the target. Gate every request behind a semaphore and tune the limit empirically while watching error and block rates.

   Python (semaphore)Copy

   sem = asyncio.Semaphore(50)   # tune to the target + your resources
   
   async def guarded(client, url):
       async with sem:
           return await fetch(client, url)

   Field note: There are two separate limits in play: how many requests your machine can hold open (the semaphore) and how fast the target tolerates them (velocity). Residential rotation relaxes the second, so usually it's local resources, not the target, that cap your semaphore.

3. 3

   Retry transient failures with backoff and isolation

   Wrap each request so timeouts, 5xx, and 429s retry with exponential backoff and jitter — and rotate IP between attempts (per-request rotation does this for free). Keep retries scoped to the single item so one stubborn URL never stalls the batch.

   Python (retry)Copy

   import random, asyncio
   
   async def with_retry(coro_factory, attempts=4):
       for i in range(attempts):
           try:
               return await coro_factory()
           except (httpx.TransportError, httpx.HTTPStatusError):
               if i == attempts - 1:
                   raise
               await asyncio.sleep(min(30, 2 ** i) + random.random())

   Field note: Make the unit of work idempotent and checkpoint completed URLs (a set in Redis, a 'done' column). When a long run dies at 70% — and eventually one will — you resume the missing 30% instead of re-scraping everything and re-paying for the bandwidth.

4. 4

   Use a producer/consumer queue for backpressure

   For very large jobs, decouple URL generation from fetching with an asyncio.Queue and a fixed set of workers. This gives you natural backpressure (producers block when the queue is full), steady memory use, and a clean place to plug rotation, retries, and metrics.

   Python (queue + workers)Copy

   async def worker(name, queue, client, results):
       while True:
           url = await queue.get()
           try:
               results.append(await with_retry(lambda: fetch(client, url)))
           finally:
               queue.task_done()
   
   async def run(urls, n_workers=50):
       queue = asyncio.Queue(maxsize=1000)
       results = []
       async with httpx.AsyncClient(proxy=PROXY, limits=LIMITS) as client:
           workers = [asyncio.create_task(worker(i, queue, client, results))
                      for i in range(n_workers)]
           for u in urls:
               await queue.put(u)        # blocks when full -> backpressure
           await queue.join()
           for w in workers:
               w.cancel()
       return results

   Field note: Stream each result straight to storage from inside the worker (DB insert, file append, or a message bus) instead of appending to a list. Buffering millions of parsed records in a Python list is the most common way these jobs OOM.

5. 5

   Instrument throughput, errors, and block rate

   At scale you cannot eyeball a run. Emit counters for requests/sec, error rate, retry rate, and block rate, and watch them live. They tell you when to widen the pool, when a target tightened defenses, and when concurrency is too high — long before the data quality suffers.

Best practices that keep scrapers reliable

• Use async I/O and reuse one pooled client across requests
• Always bound concurrency with a semaphore; tune it to local resources and the target
• Use return_exceptions / per-task try-except so one failure can't cancel the batch
• Checkpoint completed URLs so a dead run resumes instead of restarting
• Stream results to storage from the worker; never buffer millions in RAM
• Scale by widening the pool and worker count, not by hammering IPs
• Instrument requests/sec, error rate, and block rate and watch them live

Common mistakes that burn proxy budget

• A bare asyncio.gather that cancels every task when one raises
• Creating a new HTTP client per request and losing connection reuse
• Unbounded concurrency that exhausts sockets, file descriptors, or memory
• Raising worker counts on a single IP until the target blocks it
• Accumulating all results in a list until the process OOMs
• No checkpointing, so a crash at 80% means re-scraping (and re-paying for) everything

Common questions about this scraper problem