How to Scrape Booking.com

What goes wrong when this scraper fails

Hotels, OTAs, and travel-intelligence teams need competitive room rates and availability across markets and dates. Booking.com makes this genuinely hard: the price depends entirely on the search (dates, guests, rooms, currency, length of stay), so a snapshot without those parameters is meaningless; rates and even which deals show can vary by currency and apparent geography; and the site runs strong bot detection that blocks datacenter IPs and abnormal patterns. Collected carelessly, the data is a mix of currencies and date scenarios that no analyst can trust.

Why this failure mode happens

Booking.com computes rates in real time from the property's inventory and your exact search parameters, and it localizes currency and some offers to the visitor. It defends its pages with layered bot management (challenges and reCAPTCHA) that scores IP reputation, fingerprint, and behavior. So an automated request from a datacenter IP is both likely to be challenged and, if it succeeds, likely to reflect the wrong currency or a non-comparable date scenario.

Challenges that make this hard to automate

• Rates that only exist relative to dates, occupancy, rooms, and length of stay
• Currency and offers that shift with apparent geography if not pinned
• Heavy bot detection and challenges on a high-value OTA
• Offset-based pagination and per-search result limits
• Reviews and reviewer profiles that are personal data

Approaches that usually fail

• Datacenter proxies — challenged quickly on a defended OTA
• Collecting 'a price' without fixing dates/occupancy — non-comparable noise
• Letting currency follow the IP — silently mixed price series
• Manual rate shopping — accurate but unscalable across markets and dates

When residential proxies fix this — and when they cannot

Residential IPs in the target market read as real travelers, clear the bot detection that blocks datacenter ranges, and (with currency pinned) return the rates a local guest would be quoted. A large rotating pool lets you run the many date/market combinations a rate-shopping study needs without any single IP tripping velocity limits.

How Aethyn residential proxies help here

Booking.com is a high-security, geo- and currency-sensitive target. Aethyn supplies the reputation and IP diversity it demands, controlled through the username.

• Elite high-trust residential IPs to get past Booking's bot management
• Country targeting so availability and localized offers match the market
• A large rotating pool for many date × market rate lookups
• Sticky sessions to keep a search → property-detail flow coherent
• Per-byte metering so recurring rate shopping stays cost-predictable

How to implement this with residential proxies

1. 1

   Define the search: dates, occupancy, rooms, currency

   A Booking rate is only meaningful with the full query. Set check-in/check-out, group/adults and rooms, and crucially pin the currency with selected_currency so it doesn't follow the IP. Standardize a scenario (e.g. 2 adults, 1 room, 2 nights, 30 days out, EUR) and apply it consistently.

   cURLCopy

   curl -x "http://aethyn-XXXXX-country-fr:PASSWORD@proxy.aethyn.io:5499" \
     "https://www.booking.com/searchresults.html?ss=Paris&checkin=2026-08-10&checkout=2026-08-12&group_adults=2&no_rooms=1&selected_currency=EUR"

   Field note: Always set selected_currency explicitly. If you let currency default to the exit IP's country, you'll silently build a price series that mixes EUR, USD, and GBP — and no amount of later cleaning fully untangles it.

2. 2

   Render and read the structured rate data

   Drive a headless browser through the residential proxy and wait for the results to render, since rates load dynamically. Extract per-property: hotel ID, name, room type, the rate for your scenario, board/cancellation terms, and the aggregate review score. Prefer embedded data attributes over volatile class names.

   Python (Playwright)Copy

   from playwright.sync_api import sync_playwright
   
   def search(url):
       with sync_playwright() as p:
           b = p.chromium.launch(proxy={
               "server": "http://proxy.aethyn.io:5499",
               "username": "aethyn-XXXXX-country-fr",
               "password": "PASSWORD",
           })
           page = b.new_page(locale="fr-FR")
           page.goto(url, wait_until="networkidle")
           cards = page.query_selector_all("[data-testid='property-card']")
           rows = []
           for c in cards:
               name = c.query_selector("[data-testid='title']")
               price = c.query_selector("[data-testid='price-and-discounted-price']")
               rows.append({"name": name.inner_text() if name else None,
                            "price": price.inner_text() if price else None})
           b.close()
           return rows

   Field note: data-testid attributes are far more stable than Booking's hashed class names. Anchor parsing on them and guard each field — when a testid disappears across the board, that's your signal Booking changed the markup, not that hotels lost prices.

3. 3

   Paginate through the full result set

   Booking paginates with an offset parameter (in steps matching the page size). Increment offset and keep collecting until a page returns no new properties, deduping on hotel ID. Respect that a single search only surfaces so many results — narrow by area or filters for dense markets.

   Python (offset pagination)Copy

   def all_results(base_url, fetch, step=25, max_offset=1000):
       seen = {}
       for offset in range(0, max_offset, step):
           rows = fetch(f"{base_url}&offset={offset}")   # one residential request each
           if not rows:
               break
           for r in rows:
               seen[r["hotel_id"]] = r
       return list(seen.values())

   Field note: For big destinations, slice the search (by district, star rating, or property type) instead of paginating endlessly. Narrower searches return more complete, less truncated result sets than one broad query.

4. 4

   Capture rate terms, not just the number

   Two rates at the same price aren't equal if one is non-refundable and room-only and the other is free-cancellation with breakfast. Store board type, cancellation policy, and room type with every rate so comparisons are apples-to-apples and rate-parity analysis is valid.

   Field note: For rate-parity work, also record whether a 'Genius'/member discount applied. A price gap between two sites is often a loyalty discount, not a true parity violation.

5. 5

   Schedule, rotate, and trend responsibly

   Run your scenarios on a cadence through rotating residential IPs, store (hotel_id, scenario, currency, rate, terms, timestamp), and trend rates and availability. Collect property-level review scores rather than individual reviews, and back off on any challenge.

Best practices that keep scrapers reliable

• Fix dates, occupancy, rooms, and currency before collecting any rate
• Pin selected_currency explicitly so series don't mix currencies
• Anchor parsing on data-testid, not hashed class names
• Paginate by offset until empty, deduping on hotel ID
• Store board/cancellation/room-type terms with every rate
• Collect aggregate review scores, not individual reviewer data

Common mistakes that burn proxy budget

• Collecting a rate without dates/occupancy, producing non-comparable data
• Letting currency follow the IP and mixing currencies in one series
• Comparing a non-refundable room-only rate to a flexible breakfast rate
• Stopping pagination early and truncating dense markets
• Scraping individual reviews/reviewer profiles with no lawful basis
• Running a defended OTA from datacenter IPs and getting challenged

Common questions about this scraper problem