What goes wrong when this scraper fails
Teams that track only Google miss a meaningful slice of search visibility — Bing has real market share in some regions and audiences, and it underpins other surfaces. But scraping Bing has its own quirks: results localize by an explicit market parameter, pagination uses an offset rather than page numbers, and while Bing is less hostile than Google, careless collection from datacenter IPs still gets blocked. The task is to collect Bing SERPs with correct market targeting and clean pagination, reliably and at scale.
Why this failure mode happens
Bing keys localization off a market code (mkt, e.g. en-US, en-GB) and a language setting, and it paginates with a first= result offset. If you don't set the market, results don't reflect the locale you intend; if you treat pagination like page numbers, you collect the wrong slices. And like any search engine, Bing scores IP reputation and velocity, so high-volume collection needs residential IPs and rotation even though its defenses are lighter than Google's.
Challenges that make this hard to automate
- Setting market (mkt) and language so results localize correctly
- Paginating with the first= offset instead of page numbers
- Keeping geo of the IP consistent with the requested market
- Separating organic results from ads and answer boxes
- Avoiding blocks from datacenter IPs and high velocity
Approaches that usually fail
- Scraping bing.com with no mkt — results don't match the target locale
- Treating pagination as page=N — collects wrong result windows
- Datacenter proxies — cheaper but still blocked under load
- Skipping Bing entirely — leaving real search visibility untracked
When residential proxies fix this — and when they cannot
Routing Bing collection through residential IPs in the target market keeps results consistent with the mkt you request and avoids the reputation flags that block datacenter ranges. Rotation spreads volume so you can collect many queries and deep result windows without tripping Bing's lighter-but-real velocity limits.
How Aethyn residential proxies help here
Bing collection is geo-sensitive but lighter than Google — a good Premium-tier fit. Aethyn handles market geography and rotation through the username.
- Country targeting so the IP geography matches the requested mkt
- Premium pool — cost-efficient for high-volume SERP collection
- Per-request rotation to spread query volume under velocity limits
- Sticky sessions for any multi-step collection flow
- Per-byte billing so large query sets stay cost-predictable
How to implement this with residential proxies
- 1
Set market and language explicitly
Bing localizes by the mkt parameter (e.g. en-US, de-DE) and setlang. Set both so results reflect the locale you intend, and route through a residential IP in the same country so geography and market agree.
cURLcurl -x "http://aethyn-XXXXX-country-gb:PASSWORD@proxy.aethyn.io:2099" \ "https://www.bing.com/search?q=residential+proxies&mkt=en-GB&setlang=en"Field note: Keep mkt and IP country aligned. Requesting mkt=en-GB from a US IP can produce blended signals; matching them is what makes Bing results reflect a real UK searcher.
- 2
Paginate with the first= offset
Bing pages with first=, the index of the first result on the page, advancing in steps of about 10. Increment first by the page size to walk deep results; don't use a page number, which Bing doesn't honor the same way.
Python (offset pagination)import requests PROXY = "http://aethyn-XXXXX-country-gb:PASSWORD@proxy.aethyn.io:2099" proxies = {"http": PROXY, "https": PROXY} def bing_pages(query, pages=5, step=10): for p in range(pages): params = {"q": query, "mkt": "en-GB", "first": p * step + 1} yield requests.get("https://www.bing.com/search", params=params, proxies=proxies, timeout=30).textField note: Bing's results-per-page can vary slightly, so derive the next first= from how many results you actually parsed rather than assuming exactly 10 — otherwise you can skip or repeat results across pages.
- 3
Parse organic results apart from ads and answers
Separate organic listings (typically under the main results list) from ads and answer/entity boxes so positions are clean. Capture title, URL, and snippet, and track answer boxes as features if you care about visibility, just like a Google SERP.
Python (parse)from bs4 import BeautifulSoup def parse_bing(html): soup = BeautifulSoup(html, "html.parser") out = [] for i, li in enumerate(soup.select("ol#b_results > li.b_algo"), start=1): a = li.select_one("h2 a") if a: out.append({"position": i, "title": a.get_text(strip=True), "url": a.get("href")}) return outField note: Anchor on li.b_algo for organic results — it excludes the b_ad blocks, so your organic positions stay clean without manually filtering ads out afterward.
- 4
Rotate, pace, and store consistently
Spread queries across rotating residential IPs, keep velocity modest, and store results as a time series keyed on (query, mkt, run) so you can track Bing rankings over time alongside Google. Detect blocks (challenge pages, 429s) and back off rather than retrying in place.
Best practices that keep scrapers reliable
- Always set mkt and setlang, and match IP country to the market
- Paginate with first=, derived from results actually parsed
- Anchor parsing on b_algo to keep organic positions clean
- Rotate residential IPs and keep velocity modest
- Store results as a time series keyed on query + market
- Detect blocks and back off instead of retrying in place
Common mistakes that burn proxy budget
- Omitting mkt so results don't match the intended locale
- Using page=N instead of the first= offset
- Assuming exactly 10 results per page and skipping/repeating
- Mixing ads into organic positions
- Collecting Bing from datacenter IPs at volume and getting blocked
- Requesting a market from a mismatched IP country