How to Rotate Residential IPs Automatically

What goes wrong when this scraper fails

It's tempting to write a 'rotation manager': fetch a list of proxies, health-check them, track which ones got banned, round-robin the survivors. With a real residential network this is wasted effort — the pool is millions of IPs the provider already health-checks and cycles. The genuine decisions are narrower and more important: per-request rotation versus a sticky session, how long to pin a session, and how to make concurrent workers rotate independently without trampling each other.

Why this failure mode happens

Hand-rolled rotation is fragile because IP lists go stale, banned IPs linger, and coordinating 'who uses which IP' across workers is a distributed-systems problem you don't want. At the same time, blind always-rotate logic breaks the workflows that genuinely need a stable identity — logins, multi-page pagination, carts — because the session's cookies suddenly arrive from a new IP and the site invalidates them.

Challenges that make this hard to automate

• Choosing fresh-per-request vs a stable IP for each kind of job
• Avoiding stale or banned IPs that manual lists inevitably accumulate
• Preserving session continuity for logins and multi-step flows
• Coordinating independent rotation across many concurrent workers
• Remembering that rotation alone doesn't defeat fingerprint-based detection

Approaches that usually fail

• Maintaining your own IP list — endless upkeep, and bans you discover the hard way
• Round-robin over a few proxies — a small pool that gets flagged together fast
• Custom ban-tracking and health-check logic — complex, stateful, and error-prone
• A network with no stickiness option — quietly breaks every login and cart flow

When residential proxies fix this — and when they cannot

A managed residential network rotates across a huge, continuously-refreshed pool for you. You stop maintaining anything and simply declare intent: the base username for a fresh IP per request, or a session id plus lifetime when a flow needs continuity. The network handles pool health, geography, and assignment.

How Aethyn residential proxies help here

Aethyn expresses rotation entirely through the username, so one endpoint gives you either a fresh IP per request or a pinned session — no second gateway, no client-side IP bookkeeping.

• Per-request rotation by default — a fresh residential IP on every call
• Sticky sessions via -session-<id> with -lifetime-<minutes> (up to 30)
• Independent sessions: different session ids map to different stable IPs
• Country (and Elite city) targeting composes with either mode in one string
• Identical syntax across HTTP/HTTPS and SOCKS5, Premium and Elite

How to implement this with residential proxies

1. 1

   Rotate on every request (the default)

   For stateless scraping — independent page fetches with no shared login — use the base username. Every request is assigned a fresh residential IP automatically, which keeps per-IP velocity low without you tracking anything. Verify it with two calls to an IP-echo endpoint.

   cURL (rotating)Copy

   curl -x "http://aethyn-XXXXX:PASSWORD@proxy.aethyn.io:2099" https://httpbin.org/ip
   curl -x "http://aethyn-XXXXX:PASSWORD@proxy.aethyn.io:2099" https://httpbin.org/ip
   # Two different exit IPs

   Field note: This is the right default for ~80% of scraping. Reach for stickiness only when you can name the specific reason the same IP must persist — if you can't, you don't need it.

2. 2

   Pin one IP with a sticky session

   When a flow must look like one continuous user — log in, then load several authenticated pages — add a session id and a lifetime in minutes. Every request carrying the same session string lands on the same exit IP until the lifetime expires.

   cURL (sticky)Copy

   curl -x "http://aethyn-XXXXX-session-job7-lifetime-10:PASSWORD@proxy.aethyn.io:2099" https://httpbin.org/ip
   curl -x "http://aethyn-XXXXX-session-job7-lifetime-10:PASSWORD@proxy.aethyn.io:2099" https://httpbin.org/ip
   # Same exit IP for ~10 minutes

   Field note: Choose the shortest lifetime that covers the flow. A 30-minute pin for a 90-second login is just a slow way to over-use one IP — exactly the velocity concentration rotation exists to avoid.

3. 3

   Give each concurrent worker its own session

   The classic parallelism bug is many workers sharing one sticky session — they all pile onto a single IP and get it banned. Derive a unique session id per worker (or per logical task) so each gets an independent, stable exit. Compose country targeting into the same string when you need it.

   Python (per-worker session)Copy

   def proxy_for_worker(worker_id, country="us"):
       user = f"aethyn-XXXXX-country-{country}-session-w{worker_id}-lifetime-15"
       return f"http://{user}:PASSWORD@proxy.aethyn.io:2099"
   
   # Worker 1 and worker 2 get different stable IPs:
   proxy_for_worker(1)   # ...-session-w1-...
   proxy_for_worker(2)   # ...-session-w2-...

   Field note: Make session ids deterministic from the unit of work (session-w{id} or session-{account_id}), not random per request. Deterministic ids make retries reuse the same identity and make runs reproducible when you're debugging a ban.

4. 4

   Use rotation per request in an async pool

   With per-request rotation you don't pre-allocate IPs — you just point your async client at the rotating endpoint and let every coroutine pick up a fresh exit. This is why rotation pairs so naturally with high concurrency: there's no shared IP state to coordinate.

   Python (httpx async, rotating)Copy

   import asyncio, httpx
   
   ROTATING = "http://aethyn-XXXXX:PASSWORD@proxy.aethyn.io:2099"
   
   async def fetch(client, url):
       r = await client.get(url, timeout=30)   # fresh IP per request
       return r.status_code
   
   async def main(urls):
       async with httpx.AsyncClient(proxy=ROTATING) as client:
           return await asyncio.gather(*(fetch(client, u) for u in urls),
                                       return_exceptions=True)

   Field note: Rotation fixes reputation and velocity, not fingerprint. If you're still blocked after rotating cleanly, the problem is upstream of the IP — check headers and TLS fingerprint (see the IP-blocking guide).

Best practices that keep scrapers reliable

• Let the network manage the pool — don't build IP lists or health checks
• Default to per-request rotation; justify every sticky session
• Set the shortest lifetime that covers the flow
• Derive a unique, deterministic session id per worker or task
• Compose -country/-city into the same username instead of separate gateways
• Remember rotation is one layer — keep headers and fingerprint clean too

Common mistakes that burn proxy budget

• Re-implementing rotation the managed network already does
• Sharing one sticky session across unrelated parallel workers, banning the IP
• Pinning a long lifetime for a short flow and concentrating velocity
• Using random per-request session ids, so retries can't reuse an identity
• Rotating mid-login and invalidating the session's cookies
• Assuming rotation alone defeats fingerprint-based detection

Common questions about this scraper problem