Scrapy Proxy Middleware for Rotating and Sticky Residential IPs (2026)

A Scrapy proxy middleware is a downloader middleware that sets request.meta["proxy"] on every outgoing request, so a spider's traffic leaves through a proxy without each Request carrying the setting. With Aethyn it also builds the proxy username, which is where the country, the session id and the lifetime live. One small class can run a crawl on rotating exits, hold a single exit for a multi-page walk, and give a retried request a fresh exit. The Scrapy integration page shows the minimal meta['proxy'] setup. This guide covers the middleware that wraps it and the two ordering mistakes that quietly break it.

The code on this page was run against a local stand-in proxy on Scrapy 2.20 to check the client side: the login, the username format, the session handling and the retry behaviour. It was not run against live targets, so treat the exits it prints as yours to verify. The spider uses start(), which needs Scrapy 2.13 or newer. Older releases use start_requests().

The middleware

The class reads the account, password, host, port, country and lifetime from settings, so no credential sits in a spider. If a request carries meta["session"], the username gets session and lifetime suffixes and the request holds one exit. With no session, the username carries only the country and each request can draw a new exit. The password is percent-encoded before it goes into the proxy URL. Scrapy decodes the login in that URL when it builds the Proxy-Authorization header, so a password containing @, : or / arrives intact. A session id cannot contain a dash, because the dash separates username fields, so the middleware raises on one.

Code Snippet
from urllib.parse import quote


class AethynProxyMiddleware:
    """Put an Aethyn proxy URL on every request.

    meta["session"] -> sticky: the same id keeps the same exit for the lifetime.
    No session      -> rotating: each request can draw a new exit.
    """

    def __init__(self, user, password, host, port, country, lifetime):
        self.user, self.password = user, password
        self.host, self.port = host, port
        self.country, self.lifetime = country, lifetime

    @classmethod
    def from_crawler(cls, crawler):
        s = crawler.settings
        return cls(
            user=s["AETHYN_USER"],
            password=s["AETHYN_PASSWORD"],
            host=s.get("AETHYN_HOST", "proxy.aethyn.io"),
            port=s.getint("AETHYN_PORT", 2099),
            country=s.get("AETHYN_COUNTRY"),
            lifetime=s.getint("AETHYN_LIFETIME", 30),
        )

    def process_request(self, request, spider):
        # Leave a proxy the spider set itself alone, but re-sign our own on retries.
        if request.meta.get("proxy") and not request.meta.get("_aethyn"):
            return
        parts = [self.user]
        if self.country:
            parts += ["country", self.country]
        session = request.meta.get("session")
        if session:
            if "-" in session:
                raise ValueError("session id must not contain '-'")
            retries = request.meta.get("retry_times", 0)
            if retries:
                session = f"{session}r{retries}"  # a retry gets a fresh exit
            parts += ["session", session, "lifetime", str(self.lifetime)]
        user = "-".join(parts)
        request.meta["proxy"] = (
            f"http://{quote(user, safe='')}:{quote(self.password, safe='')}"
            f"@{self.host}:{self.port}"
        )
        request.meta["_aethyn"] = True

Settings, and why the number is 350

Code Snippet
# settings.py
import os

AETHYN_USER = os.environ["AETHYN_USER"]  # aethyn-XXXXX, from the dashboard
AETHYN_PASSWORD = os.environ["AETHYN_PASSWORD"]
AETHYN_COUNTRY = "de"
AETHYN_LIFETIME = 30

DOWNLOADER_MIDDLEWARES = {
    "myproject.middlewares.AethynProxyMiddleware": 350,  # must be below 750
}

RETRY_ENABLED = True
RETRY_TIMES = 3
RETRY_HTTP_CODES = [403, 429, 500, 502, 503, 504]
CONCURRENT_REQUESTS = 16
DOWNLOAD_TIMEOUT = 30

Keep the middleware below 750. Scrapy's own HttpProxyMiddleware sits at 750 and turns the login inside meta["proxy"] into a Proxy-Authorization header. If your middleware runs after it, nothing signs the login. In a local test at priority 800, Scrapy raised ValueError: invalid hostname, because the user:password@ text was still sitting in the proxy URL when the downloader read it. At 350 the middleware runs before both the retry middleware (550) and the proxy handling (750). Keep the credentials in environment variables, as above, and out of the repository. The concurrency and timeout values match the Scrapy integration page. Raise concurrency only while the share of 2xx responses holds.

Rotating, sticky, and one session per walk

Set the session on the first request of a walk and copy it onto the follow-ups with response.follow(href, meta={"session": response.meta["session"]}). Use one session id per walk, not one per spider. Every request that shares an id shares an exit, so a high CONCURRENT_REQUESTS on a single id puts the whole crawl on one IP. The longer explanation of the two modes is in how rotating residential proxies work and sticky proxies explained.

Retries that change the exit

Scrapy's RetryMiddleware re-queues a 403, 429 or 5xx with the request's meta copied across. If a proxy middleware skips any request that already has meta["proxy"], the retry reuses the exact login that just failed. On a sticky session that is the same blocked exit, and the retry fails the same way.

The middleware above marks its own proxy with _aethyn and signs the request again on every pass. On a retry it appends r1, r2 to the session id, using retry_times, so the retry opens a new session and draws a new exit. In a local test where the stand-in proxy answered 429 for session page1, the guarded version finished on page1r1. The version that skipped requests with a proxy never produced a result. A spider that sets its own proxy on a request is left alone, because the flag is only on proxies this class set. For what a 403 or 429 means before you retry, see fixing 403 and 429 when scraping, and for a 407 see the proxy authentication guide.

Check the exit before the crawl

Code Snippet
import json

import scrapy


class ExitCheckSpider(scrapy.Spider):
    name = "exit_check"

    async def start(self):
        for _ in range(3):
            yield scrapy.Request(
                "https://api.ipify.org?format=json",
                meta={"session": "check1"},  # remove this line to rotate
                dont_filter=True,
                callback=self.parse,
            )

    def parse(self, response):
        self.logger.info("exit %s", json.loads(response.text)["ip"])

Run scrapy crawl exit_check. Three log lines with the same address mean the session holds. Remove the session line and the three addresses can differ. Confirm the country with an IP lookup through the same proxy URL before you point the crawl at a target, because a status code of 200 does not prove the exit landed where you asked. Targeting reference is in the docs. To run it on real traffic, see rotating residential proxies on Premium HTTP port 2099, the Elite pool on port 5499 when you need city or ISP, and pricing.