How to Build a Proxy Rotation System

What goes wrong when this scraper fails

When a scraper outgrows a single IP, teams reach for a rotation system. The common version — a list of proxies cycled round-robin — fails quietly: dead and banned IPs stay in rotation, multi-step flows break because the IP changes mid-session, and there's no visibility into which IPs are healthy. A real rotation system needs health checking, ban detection with cooldowns, success-weighted selection, and session affinity. The deeper question is whether to build it at all, since managed rotating endpoints handle rotation server-side.

Why this failure mode happens

Naive round-robin assumes every proxy is equally healthy and every request is independent — neither is true. IPs get temporarily blocked, vary in quality, and some workflows (login, cart, pagination) require IP continuity. Without state — health, cooldowns, affinity — rotation degrades into routing traffic through dead or burned IPs. The plumbing to do it well is non-trivial, which is exactly why provider-side rotation exists.

Challenges that make this hard to automate

• Removing dead/banned IPs instead of cycling through them
• Detecting bans reliably from heterogeneous response signals
• Maintaining session affinity for multi-step flows
• Weighting selection toward consistently healthy IPs
• Observability: knowing per-IP success rates and pool health

Approaches that usually fail

• Round-robin over a static list — routes through dead and banned IPs
• Random selection — same problem, no health awareness
• Manual list curation — unsustainable as IPs churn
• No affinity handling — multi-step flows break on rotation

When residential proxies fix this — and when they cannot

A managed residential endpoint performs rotation server-side: each request can exit from a fresh IP from a large, continuously-maintained pool, and a sticky session keeps one IP for a flow — all selected via the username, with no client-side pool to health-check. That removes the hardest parts (liveness, churn, scale). If you still build your own orchestration (e.g. across multiple sources), the same endpoint becomes one healthy, high-quality upstream behind your logic.

How Aethyn residential proxies help here

Most rotation plumbing disappears when rotation is handled at the endpoint. Aethyn exposes both rotation and affinity through the username.

• Server-side per-request rotation across a large residential pool
• Sticky sessions via -session-<id>-lifetime-<min> for flow affinity
• Country/city targeting so rotation stays within the right geography
• Premium pricing that's efficient for high request volumes
• Per-byte billing so a busy rotation workload stays predictable

How to implement this with residential proxies

1. 1

   Decide whether to build at all

   If you need stateless rotation and sticky sessions over residential IPs, a managed rotating endpoint already does it — point your client at it and move on. Build your own orchestration only for genuine multi-source needs (combining providers, custom routing policies). Don't reimplement pool maintenance you can get for free.

   Python (managed rotation — no client pool)Copy

   import requests
   
   # Rotation handled server-side: each call may exit a fresh residential IP.
   ROTATE = "http://aethyn-XXXXX-country-us:PASSWORD@proxy.aethyn.io:2099"
   # Sticky: same IP for a multi-step flow.
   STICKY = "http://aethyn-XXXXX-country-us-session-cart42-lifetime-10:PASSWORD@proxy.aethyn.io:2099"
   
   requests.get("https://example.com", proxies={"https": ROTATE}, timeout=30)

   Field note: The cheapest rotation system is the one you don't maintain. Reserve a custom layer for when you truly orchestrate multiple upstreams — otherwise endpoint-side rotation is less code and fewer failure modes.

2. 2

   Model the pool with health state

   If you do build orchestration, represent each upstream with state: status (healthy/cooldown), success/failure counters, and a cooldown-until timestamp. Selection should consider health, not just order. This is what separates a real rotator from round-robin.

   Python (pool state)Copy

   import time, random
   from dataclasses import dataclass, field
   
   @dataclass
   class Upstream:
       url: str
       ok: int = 0
       fail: int = 0
       cooldown_until: float = 0.0
       def healthy(self): return time.time() >= self.cooldown_until
       def score(self):  # success rate, smoothed
           n = self.ok + self.fail
           return (self.ok + 1) / (n + 2)

   Field note: Smooth the success rate (add-one/Laplace) so a single early failure doesn't permanently sink a good upstream. Raw ratios are too jumpy when sample counts are small.

3. 3

   Detect bans and cool down, don't discard

   Treat 403/429 and challenge markers as ban signals: increment failure, put the upstream in cooldown for a backoff interval, and keep it for later — most blocks are temporary. Discarding IPs permanently shrinks your pool over a long run.

   Python (ban handling)Copy

   def record(up: Upstream, resp):
       banned = resp.status_code in (403, 429) or "captcha" in resp.text.lower()
       if banned:
           up.fail += 1
           up.cooldown_until = time.time() + min(600, 30 * (up.fail))
       else:
           up.ok += 1
       return not banned

   Field note: Use escalating cooldowns per consecutive failure rather than a fixed one. An IP that fails repeatedly should rest longer, while a one-off failure shouldn't sideline a healthy IP for long.

4. 4

   Select by weight, keep affinity for sessions

   For stateless work, pick among healthy upstreams weighted by score so traffic favors reliable IPs. For multi-step flows, pin a session to one upstream (or one sticky endpoint session) for its lifetime so cookies and login state survive — never rotate mid-flow.

   Python (weighted choice + affinity)Copy

   def pick(pool):
       healthy = [u for u in pool if u.healthy()]
       if not healthy:
           return min(pool, key=lambda u: u.cooldown_until)  # least-bad
       return random.choices(healthy, weights=[u.score() for u in healthy])[0]
   
   sessions = {}  # session_id -> Upstream (affinity)
   def pick_for_session(pool, sid):
       if sid not in sessions or not sessions[sid].healthy():
           sessions[sid] = pick(pool)
       return sessions[sid]

   Field note: Keep stateless rotation and session affinity as two distinct code paths. Conflating them is how flows end up rotating mid-session — the single most common rotation bug.

5. 5

   Make the pool observable

   Expose per-upstream success rate, cooldown count, and pool-wide healthy ratio. Alert when the healthy ratio drops, which usually means your global velocity is too high rather than the IPs being bad. Without metrics you're flying blind on a degrading pool.

Best practices that keep scrapers reliable

• Prefer endpoint-side rotation; build custom only for multi-source needs
• Model upstream health, not just rotation order
• Cool down banned IPs with escalating backoff instead of discarding
• Weight selection by smoothed success rate
• Keep session affinity separate from stateless rotation
• Expose pool health metrics and alert on drops

Common mistakes that burn proxy budget

• Round-robin over a list with no health awareness
• Permanently discarding IPs on a single block
• Rotating IPs mid-session and breaking logged-in flows
• Reimplementing pool maintenance a managed endpoint already provides
• Using raw success ratios that overreact to one failure
• No observability into which IPs are healthy

Common questions about this scraper problem