How to Scale Selenium Automation

What goes wrong when this scraper fails

Selenium works for one browser, but scaling it surfaces specific pain. Its native proxy option doesn't cleanly support authenticated proxies (username/password), so credentialed residential endpoints need a different approach. Running many drivers on one machine exhausts memory, and leaked driver processes pile up when jobs crash. And full-page loads waste bandwidth. Scaling Selenium is about per-session authenticated proxies, real parallelism via a Grid, disciplined driver lifecycle, and resource trimming.

Why this failure mode happens

Selenium drives real browsers, each of which is heavy, and its proxy capability was designed around host/port without a built-in way to send proxy credentials — so authenticated residential proxies need Selenium Wire or an on-the-fly extension. Parallelism beyond a handful of drivers needs a Grid or containers, and because each driver is a separate browser+driver process, failing to quit() them leaks processes until the host runs out of memory or handles.

Challenges that make this hard to automate

• Authenticated proxies that Selenium's native option doesn't support
• Parallelizing beyond a few drivers without exhausting one host
• Leaked driver processes from crashes and missing quit()
• Bandwidth and memory from full-page loads at scale
• Per-session proxy assignment for rotation and affinity

Approaches that usually fail

• Native proxy capability — no clean username/password support
• Many drivers on one machine — memory exhaustion and instability
• No explicit driver cleanup — leaked processes accumulate
• Loading full pages — heavy bandwidth on assets you don't need

When residential proxies fix this — and when they cannot

Assigning a residential proxy per Selenium session gives each parallel browser its own identity, so work spreads across many IPs and no single one accumulates blocking velocity. Per-session rotation (or a sticky session for a multi-step flow) lets you push real parallelism through a Grid while staying under per-IP limits — the same identity-isolation model that makes any browser automation scale.

How Aethyn residential proxies help here

Scaling Selenium needs authenticated, per-session proxies and the volume to feed many nodes. Aethyn provides both through the username.

• Per-session rotation so each Selenium driver gets a fresh IP
• Sticky sessions (up to 30 min) for multi-step authenticated flows
• Premium pool that's cost-efficient for high-volume browser traffic
• Country/city targeting per session for geo-specific runs
• Per-byte billing that rewards disabling images and assets

How to implement this with residential proxies

1. 1

   Use Selenium Wire for authenticated proxies

   Selenium's native proxy option can't send username/password, so use Selenium Wire (or inject an auth extension) to route a driver through a credentialed residential endpoint. Pass the full authenticated URL in seleniumwire_options so the proxy auth just works.

   Python (Selenium Wire + auth proxy)Copy

   from seleniumwire import webdriver
   from selenium.webdriver.chrome.options import Options
   
   def make_driver(session_id):
       user = f"aethyn-XXXXX-country-us-session-{session_id}-lifetime-10"
       proxy = f"http://{user}:PASSWORD@proxy.aethyn.io:2099"
       sw = {"proxy": {"http": proxy, "https": proxy}}
       opts = Options()
       opts.add_argument("--headless=new")
       opts.add_argument("--no-sandbox")
       opts.add_argument("--disable-dev-shm-usage")
       return webdriver.Chrome(seleniumwire_options=sw, options=opts)

   Field note: --disable-dev-shm-usage matters in containers: Chrome's default /dev/shm is tiny in Docker, and omitting this flag causes mysterious renderer crashes under parallel load.

2. 2

   Parallelize with a Grid or containers

   Don't pile drivers onto one host. Run Selenium Grid (a hub with multiple nodes) or one browser per container, so parallelism scales horizontally and a crash is isolated to a node. Point your jobs at the hub and let it distribute sessions.

   Python (remote driver to Grid)Copy

   from seleniumwire import webdriver
   
   def grid_driver(session_id, hub="http://grid-hub:4444/wd/hub"):
       user = f"aethyn-XXXXX-country-us-session-{session_id}-lifetime-10"
       proxy = f"http://{user}:PASSWORD@proxy.aethyn.io:2099"
       sw = {"proxy": {"http": proxy, "https": proxy}}
       from selenium.webdriver.chrome.options import Options
       opts = Options(); opts.add_argument("--headless=new")
       return webdriver.Remote(command_executor=hub,
                               seleniumwire_options=sw, options=opts)

   Field note: Size node count to memory like any browser fleet: measure peak RAM per session and cap concurrency accordingly. Over-subscribing a Grid node degrades every session on it, not just the extra ones.

3. 3

   One driver per job, always quit()

   Create a driver for a job and guarantee quit() in a finally block so the browser and driver processes are released even on failure. Leaked drivers are the number-one cause of Selenium fleets slowly exhausting memory and file handles.

   Python (lifecycle hygiene)Copy

   def run_job(session_id, url):
       driver = make_driver(session_id)
       try:
           driver.set_page_load_timeout(30)
           driver.get(url)
           return driver.title
       finally:
           driver.quit()   # release browser + driver, even on error

   Field note: Set page-load and script timeouts on every driver. A hung navigation without a timeout pins a session forever, and a few of those will starve the whole Grid of free slots.

4. 4

   Trim resources: kill images, tune Chrome

   Disable image loading and unneeded features so pages are lighter and faster, cutting both memory and proxy bandwidth. Combined with headless mode, this is the biggest stability and cost win for a Selenium fleet.

   Python (disable images)Copy

   from selenium.webdriver.chrome.options import Options
   
   def lean_options():
       o = Options()
       o.add_argument("--headless=new")
       o.add_experimental_option("prefs",
           {"profile.managed_default_content_settings.images": 2})
       return o

   Field note: Blocking images can cut page bytes dramatically, which with per-byte proxy billing is a direct cost reduction — just confirm the content you scrape isn't itself rendered as an image on the targets you care about.

5. 5

   Match rotation to the task and make jobs idempotent

   Use per-session rotation for independent jobs and a sticky session for multi-step authenticated flows, never rotating mid-flow. Persist results per job so a node crash resumes rather than restarts, and log per-job duration and outcome to tune concurrency.

Best practices that keep scrapers reliable

• Use Selenium Wire (or an auth extension) for credentialed proxies
• Parallelize with a Grid or containers, not one host
• Create one driver per job and quit() it in finally
• Set page-load and script timeouts on every driver
• Disable images and run headless to cut memory and bandwidth
• Use sticky sessions for flows; rotate per session otherwise

Common mistakes that burn proxy budget

• Trying to use authenticated proxies via the native proxy option
• Running too many drivers on one machine
• Forgetting quit() and leaking driver processes
• No page-load timeouts, so hung sessions starve the Grid
• Loading full pages and wasting bandwidth on assets
• Rotating IPs mid-session and breaking authenticated flows

Common questions about this scraper problem