SC
Scraping Frameworks
Scrapy proxy integration
Scrapy reads the proxy from each request's meta, so you can route an entire spider through Aethyn by setting request.meta['proxy']. For large crawls, set it once in a middleware or in start_requests so every request inherits the residential exit.
Setup steps
- 1Set meta['proxy'] on requests (or in a downloader middleware).
- 2Use the full http://user:pass@host:port form.
- 3Add targeting suffixes to the username as needed.
- 4Tune CONCURRENT_REQUESTS and DOWNLOAD_TIMEOUT for residential latency.
Code
Per-request proxy in a spider
import scrapy
class IpSpider(scrapy.Spider):
name = "ip"
proxy = "http://aethyn-XXXXX-country-us:PASSWORD@proxy.aethyn.io:2099"
def start_requests(self):
yield scrapy.Request(
"https://httpbin.org/ip",
meta={"proxy": self.proxy},
callback=self.parse,
)
def parse(self, response):
self.logger.info(response.text)settings.py tuning for residential
# settings.py
CONCURRENT_REQUESTS = 16
DOWNLOAD_TIMEOUT = 30
RETRY_ENABLED = True
RETRY_TIMES = 3
RETRY_HTTP_CODES = [403, 429, 500, 502, 503, 504]Replace aethyn-XXXXX and PASSWORD with your dashboard credentials. See the full targeting reference and protocol/port list.
Tips
- Rotation is automatic — each request gets a fresh IP unless you use a session.
- Enable retries on 403/429 to ride out occasional blocks gracefully.
- Keep concurrency reasonable; residential routes reward steady throughput.
Scrapy proxy FAQ
Do I need a third-party Scrapy proxy plugin?
No. Setting meta['proxy'] uses Scrapy's built-in HttpProxyMiddleware. A custom middleware is only useful if you want to set the proxy globally for every request.
Docs, solutions & blog
Related solution guideEnd-to-end workflow that pairs with this integration.Python SDKproxy-builder-sdk on PyPI — requests/httpx helpers and validated targeting.Premium residential proxiesCost-efficient rotating residential pool from €2.00/GB for bulk scraping and monitoring.Elite residential proxiesCurated high-trust pool with city and ISP targeting for hard anti-bot targets.Quickstart docsCredentials, endpoints, and your first proxied request.Related blog articleDeeper context on proxies, errors, and anti-bot for this stack.