Real-world proxy solutions, explained.

LinkedIn is both one of the hardest technical targets on the web and one of the most legally sensitive, because almost everything on it is personal data. This guide is deliberately conservative: how to collect genuinely public information responsibly, where the legal lines actually sit, and why logged-in automation is a trap.

Two people searching the same keyword in different cities see different rankings — so 'where do we rank?' has no single answer until you pin a location. This guide covers building rank tracking that measures the position a real local searcher sees, consistently, across countries and cities, without poisoning the data with personalization or blocks.

A price intelligence system is the difference between 'we scraped some prices' and 'pricing can trust this number at 9am every day.' That gap is architecture: decoupled stages, durable storage of raw and parsed data, product matching, and data-quality monitoring. This guide covers the whole pipeline, not just the scraper.

Pulling a whole catalog — every title, price, variant, image, and stock status across thousands of products — is a different job from grabbing one price. You have to traverse category trees, survive three kinds of pagination, model variants correctly, and not get throttled halfway through. This guide covers doing it reliably and completely.

An LLM or ML model inherits the biases, gaps, and noise of its data — and increasingly, the legal exposure of how that data was sourced. This guide covers building large, diverse web corpora that hold up on three fronts: technical quality (dedup, cleaning, filtering), representativeness (geography and language), and responsible, documented provenance.

Walmart shows a different price and 'available near you' to a shopper in Dallas than to one in Newark, gates its pages behind HUMAN/PerimeterX, and renders everything from a JSON blob in the page. This guide covers collecting accurate, store-level Walmart pricing without living on the 'Robot or human?' wall.

Most people scrape Shopify stores the hard way — rendering pages and parsing HTML — when almost every Shopify store exposes a clean, paginated JSON feed of its entire catalog. This guide shows how to use it (and what to do when a store locks it down), all through residential proxies so recurring monitoring doesn't get throttled.

Airbnb's price isn't a property of a listing — it's a function of dates, guests, and length of stay — and a single map search only ever returns a capped slice of an area. This guide covers collecting short-term rental data correctly: pricing for real date ranges, full-area coverage via map tiling, and responsible handling of host data.

On Booking.com a hotel has no fixed price — it has a rate for specific dates, occupancy, currency, and length of stay. This guide covers collecting hotel availability and rates accurately, with the date/occupancy parameters that define a real quote, while getting past one of travel's tougher anti-bot setups.

A useful local-business dataset isn't one source scraped once — it's many sources reconciled into one clean record per business, with full geographic coverage. This guide covers gridding your area to beat per-search caps, deduping records across maps and directories on NAP, and doing it compliantly.

A lead pipeline that just dumps scraped rows into a CRM produces dirty, duplicate, non-deliverable lists that burn sender reputation. This guide designs a staged pipeline — source, normalize, dedupe, enrich, validate — that yields clean, compliant B2B records, with residential proxies underneath the collection stage.

Playwright is easy for one page and brutal at a thousand — memory balloons, IPs get burned, and bandwidth costs explode on images you never needed. This guide covers scaling Playwright the right way: pooling browsers, isolating contexts with a proxy each, blocking heavy resources, and capping concurrency to what your machine can actually sustain.

A CAPTCHA is a symptom, not the disease. By the time one appears, a site has already decided you look automated. This guide focuses on prevention — fixing the IP reputation, fingerprint, behavior, and velocity that trigger CAPTCHAs — so you see far fewer of them, rather than paying solver services to grind through walls you keep building.

Tracking one keyword is a script; tracking thousands across countries, devices, and SERP features is a system. This guide is the architecture for a SERP tracking tool — the keyword × geo × device matrix, residential collection, feature-aware parsing, time-series storage, and change detection that turn raw SERPs into reliable rank history.

Bing is the second search engine most rank trackers forget — and it powers more than its own results (it feeds other engines and assistants). This guide covers collecting Bing SERPs correctly: setting the market so results localize, paginating with the first= offset, and using residential IPs to keep collection clean.

Plenty of teams build a proxy rotation layer when they shouldn't, and skip the parts that matter when they do. This guide covers what a real rotation system needs — health checks, ban detection, weighting, and session affinity — and when a managed rotating endpoint makes most of it unnecessary.

Selenium scales fine — until you need authenticated proxies per session and dozens of parallel browsers without melting the host. This guide covers the parts that actually bite: per-session proxy auth, a Grid for parallelism, driver lifecycle hygiene, and trimming resource usage so runs stay stable.

The fast way to get a batch of social accounts banned is to run them from one IP and one browser. Platforms link accounts by shared signals. This guide covers managing many accounts safely: one stable identity per account — sticky residential IP, isolated browser profile, consistent fingerprint — plus human-like behavior.

Brand mentions don't live in one place — they're scattered across search, news, forums, review sites, and marketplaces, and what shows up depends on where you look from. This guide covers building brand-mention monitoring that's geo-accurate, deduplicated across sources, and tuned to alert on what matters instead of drowning you in noise.

Localized sites look perfect from headquarters and break in the field — the wrong currency, an untranslated string, a redirect loop that only fires from one country. This guide covers running localization (l10n) testing from real in-country IPs so you verify what users in each market actually experience, and automating those checks in CI.

Raw web text is not a training set — it's mostly boilerplate, duplicates, and noise, and shipping it to a model wastes compute and inflates benchmarks. This guide covers turning broad web collection into an LLM-ready corpus: quality filtering, aggressive dedup, benchmark decontamination, PII handling, and sharded storage — with residential proxies under the collection stage.

StockX exposes no public pricing API, so resale-market analysts have to read bid/ask and last-sale data off the same JSON the product pages consume - and StockX, GOAT, Flight Club, and the major sneaker retailers all sit behind Akamai/PerimeterX-class bot management that fingerprints your TLS stack before your first byte of HTML lands. Release calendars and price data are also geo-gated, so a Frankfurt datacenter IP sees a different (or empty) picture than a US residential one. This guide covers collecting accurate, localized resale-price and drop-date data at volume - including the traffic spikes that hit the moment a hyped pair drops - without your fleet getting reputation-flagged into oblivion.

Ticket pricing is one of the fastest-moving public datasets on the web: face-value inventory on Ticketmaster and resale listings on StubHub and SeatGeek reprice by the minute as an event approaches, and every price is geo- and event-specific. There is no unified public price API across these marketplaces, so building a price-intelligence or consumer-comparison view means reading each site's public listing pages and internal JSON endpoints directly. This guide covers how to collect accurate public price and availability data at volume - defensively, respectfully, and strictly for comparison and analytics, never for buying.

The big listing portals have no open, general-purpose feed for their public inventory, so anyone building an AVM, a comps engine, or a property-search product ends up querying the public pages directly. That means dealing with three very different anti-bot postures at once: Zillow leans hard on PerimeterX (HUMAN) and is the toughest of the three, Realtor.com runs Akamai Bot Manager, and Rightmove is lighter but strictly UK-geofenced. This guide covers how to pull list price, beds/baths, square footage, location, days-on-market, and price history at volume - accurately and from the correct country exit - while staying on public, non-personal data.

None of the big social platforms sell a clean public-analytics feed, so anyone doing influencer sizing, trend research, or media analysis has to read the public surfaces directly - and each surface fights back differently. TikTok signs its requests with X-Bogus and fingerprints mobile clients, Twitch's public GraphQL sits behind Cloudflare with per-IP rate limits, and Spotify gates its anonymous web token behind an anti-bot layer. All three geo-personalize what counts as "trending," so an exit in the wrong country returns the wrong numbers. Here is how to collect accurate, market-correct, public aggregate metrics at volume without tripping any of them.

Neither Quora nor Pinterest offers a general content API for research, so anyone building trend datasets has to read the public pages directly. Both are hostile to automation in different ways: Quora fronts nearly everything with Cloudflare, pushes logged-out visitors toward a login wall, and throttles hard on velocity, while Pinterest serves almost nothing in initial HTML and lazy-loads pins through internal JSON resource endpoints that are fingerprinted aggressively. This guide shows how to collect public, non-personal Q&A and pin data at volume - respecting robots.txt and Terms of Service, and never touching authenticated or personal profile data.

Best Buy and Target both price and stock the same SKU differently depending on the store and ZIP you are looking from, so any serious price-intelligence pipeline has to query them from many geographic vantage points. Both sit behind commercial anti-bot stacks - Akamai Bot Manager at Best Buy, an Imperva-class system in front of Target's RedSky JSON APIs - that fingerprint your TLS handshake and browser as aggressively as they count your requests. Here is how to collect accurate, store-localized public pricing and availability at volume without tripping challenge pages or poisoning your dataset with soft blocks.

Craigslist and Mercari expose the same category of data - public listings with item, price, location, and category - but defend it in opposite ways. Craigslist is a thin-HTML site partitioned into hundreds of per-metro city subdomains that blocks on IP reputation and request velocity, firing fast 403/429s once a source IP gets loud. Mercari is a JavaScript app backed by an internal GraphQL-style JSON API that fingerprints and rate-limits its search, item, and price endpoints and expects a client-generated DPoP signing token. Here is how to aggregate public listing data across both at volume - for price benchmarking and trend analysis - while collecting only non-personal fields and respecting each site's Terms of Service.

Skyscanner and Tripadvisor both publish enormous amounts of public data - live fares by route/date/currency, and hotel/restaurant rankings by locale - but neither offers an open, unauthenticated API for it, so you have to query the public surfaces directly. Skyscanner runs session-based fare polling behind a PerimeterX-class challenge on its search endpoints; Tripadvisor sits behind Cloudflare with lazy-loaded reviews and content that changes by country and language. Here is how to collect accurate, geo-correct travel data at volume without tripping either defense - using only public, non-personal data and no booking automation.

Neither Glassdoor nor Indeed ships a general-purpose public data API, so anyone building hiring intelligence has to read the same public pages a browser sees - and both sites sit behind aggressive Cloudflare protection with fingerprint-based bot detection layered on top. Glassdoor adds soft login walls and geo-gated salary data; Indeed throws interstitial challenges and locale-specific result sets. Here is how to collect accurate, region-specific labor-market data at volume - job descriptions, aggregate salary bands, and public company ratings - without harvesting a single person's private information.

Yandex owns roughly 60% of Russian search and has no public organic-results API, so rank trackers have to query the public SERP pages directly - and Yandex runs one of the most aggressive anti-automation stacks outside Google, from its showcaptcha 'are you a robot' interstitial to IP-reputation and request-velocity scoring. On top of that, results are region-personalized through the lr= region code, so an exit in the wrong country returns rankings nobody in your target market actually sees. Here is how to collect accurate, region-correct public SERP data at reasonable volume without drowning in robot checks.

A 403 or 429 mid-crawl is rarely 'the site hates proxies' — it is usually one of four predictable failure modes with different fixes. This troubleshooting guide maps each status code to the signal it carries, the root cause behind it, and the rotation or credential change that actually resolves it.

Flipkart prices, offers, and delivery promises are India-specific — a US exit shows different availability and often blocks outright. This pipeline guide walks from SKU discovery through INR-normalized alerts, routed through Indian residential IPs on the Premium pool.

Infinite scroll looks like one page but is dozens of paginated API calls triggered by viewport geometry. Stopping after the first screen misses most of the catalog. This guide covers Playwright scroll loops, Intersection Observer instrumentation, and proxy rotation so you load every batch without tripping velocity defenses.

A 403, 429, or empty 200 is not a diagnosis — it is a symptom shared by half a dozen vendors. This hub maps the observable signals (cookies, response headers, status codes, body markers) to the system that produced them, then states honestly whether rotating a residential IP can change that layer.

Akamai Access Denied is often a fingerprint decision that happens before your HTTP semantics matter — and the Reference # string is a resolvable receipt, not random noise. This guide shows how to read Reference #18 vs #9, which cookies prove Bot Manager is present (not that you passed), and when rotating residential IPs actually changes the outcome.

DataDome’s block page looks the same whether you hit a device check, a CAPTCHA, or a hard ban — HTTP 403, little branding, similar copy. The useful work is parsing the challenge payload and headers, then deciding whether a cleaner residential IP can help or whether your Python client fingerprint is the real problem.

Your monitor says green because the status line says 200 — meanwhile the body is Access Denied, “page unavailable,” or an empty catalog shell. Soft blocks poison datasets without tripping raise_for_status(). This page shows how to catch them with title, length, and selector asserts, using verified public examples.

The anti-bot page loaded. Your framework shrugged. Playwright’s page.goto hands back a Response with .status 403 and does not throw; Selenium’s driver.get likewise raises nothing while driver.title becomes Access Denied. Timeouts on #product-price are a late, misleading signal — this pipeline makes silence impossible.

Most stacks treat HTTP 429 as “slow down and retry.” Kasada inverts that reflex: on many HTML routes the challenge itself arrives as 429 with a tiny HTML bootstrap, no Retry-After, and almost no vendor chrome — so exponential backoff becomes an infinite burn. Identify via x-kpsdk-* / KPSDK / fixed UUID script paths; a plain 403 without those artifacts is not Kasada.

HUMAN (PerimeterX) Bot Defender is easy to misread: the SDK library defaults to MONITOR with blockingScore 100, the durable client cookie is _px3, and the portable challenge body marker is meta name=description content=px-captcha — not a mythical always-on response beacon. This page maps those layers so you stop treating every clean 200 as “no PX” and every press-and-hold as something a proxy can click through.

Imperva’s brand history (Incapsula Cloud WAF + Distil-origin ABP + Thales ownership) creates messy search results. On the wire, attribute positively: visid_incap_*, incap_ses_*, nlbi_*, X-Iinfo, X-CDN: Imperva, /_Incapsula_Resource, or “Incapsula incident ID”. Treat the incident-ID string as opaque — do not invent digit-length meanings. Treat reese84 / ___utmvc as Distil-lineage cookie names from third-party tooling, not standalone proof.

Sibling vendors stamp cookies and ray IDs on the way out. F5 Shape Security (Bot Defense) expects your client to send telemetry — HTTP headers or POST body fields minted after JS runs — and it attaches policy to endpoints and methods, not “the whole hostname.” Retail symptom: homepage 200s; one XHR or login POST returns transformed junk. Do not “detect Shape” via TS*, BIGipServer*, or reese84.

Sneaker drops reuse the same vendor fingerprints as the rest of the cluster — with harsher vocabulary and noisier unverified attributions. This page sticks to verified edge attributions, states the US/UK legal line clearly, and notes the practical buying pattern: demand here is usually per-IP ISP quality, not bulk GB residential.