Master the art of proxy infrastructure

A Scrapy proxy middleware sets the proxy on every request. This one builds the Aethyn username, holds a sticky session per walk, and gives retried requests a fresh exit.

A scraping API returns the page and bills credits per request. A proxy gives you the exit and bills bytes. See who runs each part of the job and three questions that pick the tool.

aiohttp handles HTTP proxies natively and needs aiohttp-socks for SOCKS5. Working code for both, remote DNS with rdns=True, and one sticky session per login.

A rotating residential proxy sends each request out through a different household IP. Here is how the gateway picks the exit, when rotation is per request or timed, and why pool size is not the number to watch.

A sticky proxy holds one residential IP for a set time instead of changing it on every request. Here is the exact session and lifetime username, how long it lasts, and when to use it.

Transparent proxies reveal your IP, anonymous proxies hide it but announce themselves, and elite proxies do neither. Here is how to tell them apart with one header test.