Aggregating Public Flight Fares and Travel Reviews from Skyscanner and Tripadvisor

What goes wrong when this scraper fails

Travel price intelligence teams need two public data streams: live flight fares (who is cheapest on a route, how prices move by booking window and currency) and accommodation/attraction sentiment (review counts, average ratings, and ranking position). Skyscanner is the canonical fare aggregator and Tripadvisor the canonical review corpus, but neither exposes this to the public through a free API - Skyscanner's partner API is gated and rate-limited, and Tripadvisor's Content API caps you at a small allowance of locations. Practitioners therefore collect the public web pages directly, which means handling two very different anti-automation postures on the same pipeline.

Why this failure mode happens

Fare data is expensive to compute and commercially sensitive, so Skyscanner protects its fare-polling endpoints with a PerimeterX-class behavioral challenge, session tokens, and aggressive per-IP velocity limits - a burst of searches from one address gets soft-blocked within seconds. Tripadvisor's problem is different: its pages are cheap to serve but heavily scraped, so it fronts everything with Cloudflare (JA3/TLS fingerprinting, managed challenges) and defers review bodies to lazy-loaded XHR calls that only fire under a real browser. On top of that, both sites personalize output by geography, so a technically successful request from the wrong country returns correct-looking but commercially wrong data.

Challenges that make this hard to automate

• Skyscanner never returns fares synchronously - it issues a session/poll handshake and streams results over 3-10 seconds, so a naive single-GET scraper captures an empty or partial fare set.
• Skyscanner's PerimeterX-class layer scores mouse/timing behavior and per-IP velocity; a handful of fare searches from one exit triggers a soft block that still returns 200 with truncated results.
• Tripadvisor's Cloudflare fingerprints your TLS/JA3 handshake and header ordering, so a plain requests.get is challenged even from a clean residential IP.
• Tripadvisor lazy-loads review bodies and ratings via secondary XHR/GraphQL calls, so the initial HTML contains only skeletons - you must render or replay those calls.
• Both sites localize output: fares vary by currency and departure-market IP, and reviews/rankings vary by country and language, so a geo mismatch silently corrupts your dataset while every request still returns 200.

Approaches that usually fail

• Official partner APIs (Skyscanner Travel APIs, Tripadvisor Content API) - clean and legal but heavily gated, low rate limits, limited location coverage, and often commercial-partner-only, so they cannot back a broad market-research sweep.
• A single datacenter IP with a scripted browser - works for a dozen requests, then Skyscanner's velocity limiter and Tripadvisor's Cloudflare challenge lock it out; datacenter ASNs are pre-flagged.
• Headless browser farms without proxy rotation - solves the JavaScript/lazy-load problem but every session shares one IP reputation, so you scale straight into blocks.
• Third-party 'travel data' resellers - offload the anti-bot fight but give you stale snapshots, no control over currency/locale, and no way to re-price a route on demand.

When residential proxies fix this — and when they cannot

Rotating residential proxies give every fare search and every review page its own clean, geographically-correct exit IP, so Skyscanner's per-IP velocity limiter never sees a burst and Tripadvisor's Cloudflare reputation checks see ordinary consumer traffic. Because Aethyn lets you pin the exit country (and city) per request, you can price JFK-LHR from a real GB IP in GBP and simultaneously from a US IP in USD, getting the market-correct number each time. For Skyscanner's multi-second poll cycle you attach a short sticky session so the whole handshake stays on one exit, then rotate away for the next route.

How Aethyn residential proxies help here

Aethyn is a residential proxy network built for exactly this kind of geo-sensitive, high-velocity public-data collection. You get per-request rotation by default, precise country/city targeting so your fares and reviews come from the right market, and short-lived sticky sessions for Skyscanner's poll handshake - all through one endpoint. For Skyscanner specifically you can start on Premium and escalate individual routes to Elite when the anti-bot layer gets aggressive.

• Per-request IP rotation by default, so consecutive Skyscanner fare searches never stack on one exit and trip the velocity limiter.
• Country- and city-level targeting (e.g. -country-gb, -country-us-city-chicago) so fares are priced in the correct departure market and Tripadvisor serves the right locale.
• Short sticky sessions (up to 30 minutes) that keep Skyscanner's session-open plus poll cycle pinned to a single exit IP, then release it.
• Clean residential ASNs that clear Tripadvisor's Cloudflare reputation checks where datacenter ranges are pre-flagged.
• One endpoint, two tiers: Premium (port 2099) for the bulk of the work, with a one-line switch to Elite (port 5499) for Skyscanner routes under heavy challenge.
• Large exit pool so you can spread a full route matrix or a city's hotel list across many IPs and stay under any per-IP threshold.

How to implement this with residential proxies

1. 1

   Route every request through geo-correct rotating residential proxies

   Both sites personalize by geography, so the exit country is part of your query, not an afterthought. Use per-request rotation as the default - a fresh IP for each fare search and each review page - and pin the country to the market you are measuring. For Skyscanner, the departure-market IP plus the currency query param must agree; for Tripadvisor, the country IP plus the locale param must agree. This snippet builds a Premium proxy on port 2099 and confirms the exit geography before you trust any data.

   pythonCopy

   import requests
   
   # Premium tier -> port 2099. Rotate on every request (no session token).
   ACCOUNT = "aethyn-XXXXX"
   PASSWORD = "PASSWORD"
   
   def proxy_for(country):
       user = f"{ACCOUNT}-country-{country}"
       url = f"http://{user}:{PASSWORD}@proxy.aethyn.io:2099"
       return {"http": url, "https": url}
   
   # Verify the exit really is in the market you think it is
   for cc in ("gb", "us"):
       r = requests.get("https://ipinfo.io/json", proxies=proxy_for(cc), timeout=20)
       j = r.json()
       print(cc, "->", j.get("country"), j.get("city"), j.get("org"))
       assert j.get("country", "").lower() == cc, "exit country mismatch - do not collect on this IP"

   Field note: Never let the currency/locale query param and the proxy country drift apart. A GBP fare pulled through a US IP is not the London market price - it is a corrupted row that will quietly skew every average you compute downstream.

2. 2

   Replay Skyscanner's session-open then poll cycle on one sticky exit

   Skyscanner does not hand you fares in a single response. Its front end opens a search session, gets back a session/poll token, then long-polls until the results stop updating. You must reproduce that two-phase flow, and the whole cycle has to stay on one IP - so use a short sticky session, not per-request rotation, for the duration of a single route. Poll until the status reports complete or you stop seeing new itineraries, then rotate away for the next route. Keep the session lifetime tight (a couple of minutes) because a fare quote is only valid for minutes anyway.

   pythonCopy

   import requests, time
   
   ACCOUNT = "aethyn-XXXXX"
   PASSWORD = "PASSWORD"
   
   def sticky_proxy(country, token, minutes=3):
       # Pin the whole open+poll handshake to ONE exit. Lifetime max 30.
       user = f"{ACCOUNT}-country-{country}-session-{token}-lifetime-{minutes}"
       url = f"http://{user}:{PASSWORD}@proxy.aethyn.io:2099"
       return {"http": url, "https": url}
   
   def poll_fares(session, poll_url, max_polls=8):
       itineraries = []
       for attempt in range(max_polls):
           resp = session.get(poll_url, timeout=25)
           resp.raise_for_status()
           data = resp.json()
           itineraries = data.get("itineraries", itineraries)
           status = data.get("status", "")
           if status == "RESULT_STATUS_COMPLETE":
               break
           time.sleep(1.5)  # human-plausible cadence between polls
       # A complete poll with an empty list is a soft block, not an empty route
       assert itineraries, "no itineraries after polling - treat as blocked, retry on a new exit"
       return itineraries
   
   s = requests.Session()
   s.proxies.update(sticky_proxy("gb", token="jfklhr01", minutes=3))
   # open the search first (returns a poll_url / session id), then:
   # itineraries = poll_fares(s, poll_url)

   Field note: Space your polls like a browser would - roughly one every 1-2 seconds, not a tight loop. Skyscanner's PerimeterX-class layer scores request cadence, and machine-gun polling from one session is one of the fastest ways to get your results silently truncated.

3. 3

   Escalate stubborn Skyscanner routes from Premium to Elite

   Premium (port 2099) clears the bulk of Skyscanner traffic. But hot routes, high-frequency re-pricing, or a sudden anti-bot tightening can start returning truncated polls or challenge pages even with clean rotation. Rather than throttling your whole pipeline, escalate just the affected routes to Elite (port 5499) - a cleaner, lower-contention exit pool - and keep everything else on Premium. The only change is the port; username format and rotation logic are identical.

   pythonCopy

   ACCOUNT = "aethyn-XXXXX"
   PASSWORD = "PASSWORD"
   
   PREMIUM_PORT = 2099   # default for this pipeline
   ELITE_PORT   = 5499   # escalate here when Skyscanner challenges hard
   
   def proxy(country, session=None, minutes=3, port=PREMIUM_PORT):
       user = f"{ACCOUNT}-country-{country}"
       if session:
           user += f"-session-{session}-lifetime-{minutes}"
       url = f"http://{user}:{PASSWORD}@proxy.aethyn.io:{port}"
       return {"http": url, "https": url}
   
   def fares_with_escalation(open_and_poll, country, token):
       try:
           return open_and_poll(proxy(country, token, port=PREMIUM_PORT))
       except AssertionError:
           # Premium got truncated/blocked - retry the same route on Elite
           return open_and_poll(proxy(country, token, port=ELITE_PORT))

   Field note: Escalate per-route, not globally. Elite is your scarce, clean pool - burning it on routes Premium already handles just raises your contention on the ones that actually need it. Track block rate per route and promote only the offenders.

4. 4

   Render Tripadvisor with a real browser engine to clear Cloudflare and lazy-loaded reviews

   Tripadvisor fronts its pages with Cloudflare, which fingerprints your TLS/JA3 handshake and header order - a plain requests.get gets a managed challenge even from a clean residential IP. On top of that, review bodies, ratings, and dates lazy-load via secondary XHR/GraphQL calls that only fire in a real engine. Drive Playwright through an Aethyn residential exit whose country matches the locale you want, wait for the review nodes to actually appear, then parse the rendered DOM. Match the proxy country to the Tripadvisor domain/locale (e.g. a GB IP for the en-GB experience).

   pythonCopy

   from playwright.sync_api import sync_playwright
   
   ACCOUNT = "aethyn-XXXXX"
   PASSWORD = "PASSWORD"
   
   def run(url, country="gb"):
       server = "http://proxy.aethyn.io:2099"   # Premium
       username = f"{ACCOUNT}-country-{country}"  # rotate per browser context
       with sync_playwright() as p:
           browser = p.chromium.launch(
               headless=True,
               proxy={"server": server, "username": username, "password": PASSWORD},
           )
           ctx = browser.new_context(
               locale="en-GB",
               user_agent=("Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
                           "AppleWebKit/537.36 (KHTML, like Gecko) "
                           "Chrome/125.0.0.0 Safari/537.36"),
           )
           page = ctx.new_page()
           page.goto(url, wait_until="domcontentloaded", timeout=45000)
           # Wait for lazy-loaded reviews, not just the skeleton HTML
           page.wait_for_selector("div[data-reviewid]", timeout=20000)
           cards = page.query_selector_all("div[data-reviewid]")
           assert cards, "no review nodes rendered - Cloudflare challenge or block"
           html = page.content()
           browser.close()
           return html

   Field note: Wait on a concrete review selector (data-reviewid), not a fixed sleep. If that selector never appears you have hit a Cloudflare challenge, not an empty page - retry on a fresh exit instead of parsing the skeleton and storing zeros.

5. 5

   Parse defensively and reject soft blocks before persisting

   Both sites will hand you HTTP 200 responses that are actually failures: a Skyscanner poll that never completed, or a Tripadvisor challenge page dressed up as content. Your parser must treat structure as untrusted - use optional lookups, validate types, and above all assert that you actually extracted the fields that matter (a fare number, a review count, a rating) before writing a row. Store the exit country and currency/locale alongside every record so a geo-mismatch is auditable later.

   pythonCopy

   from bs4 import BeautifulSoup
   import re
   
   def parse_tripadvisor(html, exit_country, locale):
       soup = BeautifulSoup(html, "html.parser")
       cards = soup.select("div[data-reviewid]")
       if not cards:
           raise ValueError("zero review cards - soft block, do not persist")
       rows = []
       for c in cards:
           rating_el = c.select_one("svg[aria-label*='bubbles'], span.ui_bubble_rating")
           rating = None
           if rating_el is not None:
               label = rating_el.get("aria-label") or rating_el.get("class", [""])[-1]
               m = re.search(r"([0-5](?:\.[05])?)", str(label))
               rating = float(m.group(1)) if m else None
           title_el = c.select_one("a[href*='ShowUserReviews'] span, div[data-test-target='review-title']")
           rows.append({
               "review_id": c.get("data-reviewid"),
               "rating": rating,
               "title": title_el.get_text(strip=True) if title_el else None,
               "exit_country": exit_country,   # provenance for audit
               "locale": locale,
           })
       # Guard: a page of cards with no usable ratings is suspect
       if not any(r["rating"] is not None for r in rows):
           raise ValueError("cards present but no ratings parsed - layout drift or block")
       return rows

   Field note: Assert a non-empty, well-typed result set before every database write. A 200 with zero fares or zero parsed ratings is almost never a real empty result - it is a soft block or a layout change, and silently storing it is how a price-intelligence dashboard goes quietly wrong for weeks.

6. 6

   Spread the route/date matrix across exits and pace to the market's freshness

   Fares are perishable, so you want the whole matrix collected in a tight window - but not so fast that one IP or one session lights up the velocity limiter. Distribute routes across many rotating exits, cap concurrency per exit country, and add small jitter so traffic looks human. For a full re-price, one fresh exit per (route, date, currency) cell keeps every request under Skyscanner's per-IP radar while still finishing quickly.

   pythonCopy

   import concurrent.futures as cf
   import random, time
   
   ACCOUNT = "aethyn-XXXXX"
   PASSWORD = "PASSWORD"
   
   def proxy(country):
       user = f"{ACCOUNT}-country-{country}"        # per-request rotation
       url = f"http://{user}:{PASSWORD}@proxy.aethyn.io:2099"
       return {"http": url, "https": url}
   
   def price_cell(cell, fetch):
       route, date, currency, country = cell
       time.sleep(random.uniform(0.2, 1.1))          # jitter, not a fixed rate
       return fetch(route, date, currency, proxy(country))
   
   def reprice_matrix(cells, fetch, max_workers=12):
       results = []
       with cf.ThreadPoolExecutor(max_workers=max_workers) as pool:
           futures = {pool.submit(price_cell, c, fetch): c for c in cells}
           for fut in cf.as_completed(futures):
               cell = futures[fut]
               try:
                   results.append((cell, fut.result()))
               except Exception as e:
                   print("cell failed, requeue on new exit:", cell, str(e))
       return results

   Field note: Tune concurrency to your block rate, not to your CPU. If failures climb past a few percent, lower max_workers and widen the jitter before you touch anything else - contention on the exit pool, not throughput, is what gets you soft-blocked.

Best practices that keep scrapers reliable

• Collect only public, non-personal data - published fares, aggregate ratings, review counts and rankings. Do not scrape reviewer personal details, do not automate bookings, and never generate or post fake reviews. Respect each site's Terms of Service and consult counsel for your jurisdiction.
• Match proxy country to currency/locale on every single request - a GB IP for GBP/en-GB, a US IP for USD/en-US - and store that provenance on each row so mismatches are auditable.
• Use per-request rotation as the default; reserve sticky sessions for exactly the flows that need IP continuity, like Skyscanner's open+poll handshake.
• Keep Skyscanner session lifetimes short (2-3 minutes) - fares expire fast and long-lived sessions accumulate anti-bot suspicion for no data-quality benefit.
• Assert non-empty, well-typed results before persisting; treat any 200 with zero fares or zero parsed reviews as a soft block and retry on a fresh exit.
• Escalate to Elite (port 5499) per-route on demand rather than globally, so your clean pool is spent only where Premium is actually failing.
• Cache and de-duplicate: reviews change slowly, so re-fetch Tripadvisor pages on a sane cadence (hours/days) and spend your request budget on the perishable fares.

Common mistakes that burn proxy budget

• Treating Skyscanner like a static page - firing one GET and storing whatever comes back, which captures an empty or partial fare set before the poll cycle completes.
• Rotating the IP mid-session on Skyscanner, so the poll token opened on one exit is polled from another and the session is rejected.
• Hitting Tripadvisor with plain requests/httpx and no browser engine, then parsing the Cloudflare challenge or the un-hydrated review skeleton as if it were real content.
• Ignoring geography - pulling fares or reviews through whatever exit is handy, producing correct-looking rows in the wrong currency or locale that silently corrupt aggregates.
• Persisting empty results on a 200 OK - no assertion that a fare number or rating was actually extracted, so soft blocks masquerade as 'this route has no flights'.
• Running maximum concurrency from a small IP set, which stacks requests per exit and trips both Skyscanner's velocity limiter and Cloudflare's reputation checks.

Common questions about this scraper problem