How to Build a SERP Tracking Tool

What goes wrong when this scraper fails

A team needs day-over-day ranking for many keywords across markets — and the naive approach (loop keywords, grab the HTML, find your domain) breaks down fast. Results are personalized and localized, so position depends on where and on what device you searched. The page is dominated by SERP features that change what 'position 1' even means visually. And at volume, collection gets blocked. Building a real SERP tracker is a systems problem: define the tracking matrix, collect consistently from the right place, parse features, and store comparable history.

Why this failure mode happens

Search engines tailor results to location, device, and language, and they fill the page with features (ads, People Also Ask, local packs, featured snippets, AI overviews) that vary by query. So an organic position number alone is incomplete and is only comparable if every run holds geo, device, and language constant. At scale, consistent collection also requires geo-accurate IPs and controlled velocity, which is why a durable tool wraps collection, parsing, and storage into one disciplined pipeline.

Challenges that make this hard to automate

• Defining and scheduling a keyword × geo × device × language matrix
• Collecting localized, device-correct results consistently
• Parsing organic results separately from SERP features
• Reconciling organic position with visual/pixel position
• Storing comparable time series and detecting meaningful changes

Approaches that usually fail

• A loop that greps the HTML for your domain — ignores geo, device, and features
• Third-party rank APIs — convenient but opaque and costly at scale
• Datacenter proxies for collection — blocked and geo-inaccurate
• Spreadsheets of manual checks — not reproducible or schedulable

When residential proxies fix this — and when they cannot

A SERP tracker is only as trustworthy as its collection. Residential IPs in each target city return the localized results a real searcher there sees, and a rotating pool lets you run the full matrix on a schedule without tripping velocity limits. Holding geo and language constant per series is exactly what makes day-over-day positions comparable.

How Aethyn residential proxies help here

A tracker needs geo precision, device control, and the volume to run a large matrix. Aethyn provides all three through the username.

• City-level targeting so each series reflects the exact local SERP
• Elite high-trust IPs that reliably return clean, un-challenged SERPs
• A large rotating pool to run thousands of matrix cells on schedule
• Sticky sessions when a multi-step collection needs IP continuity
• Per-byte billing so a daily full-matrix run stays cost-predictable

How to implement this with residential proxies

1. 1

   Define the tracking matrix

   Every tracked unit is a (keyword, location, device, language) tuple. Store these as the tool's core schema and schedule each as its own job, so a keyword tracked in 5 cities on 2 devices becomes 10 independent series. This makes coverage explicit and runs reproducible.

   Python (matrix expansion)Copy

   from itertools import product
   
   def build_jobs(keywords, locations, devices, lang="en"):
       for kw, loc, dev in product(keywords, locations, devices):
           yield {"keyword": kw, "location": loc, "device": dev,
                  "language": lang}
   
   jobs = list(build_jobs(
       keywords=["residential proxies", "proxy api"],
       locations=[("us", "new york"), ("gb", "london")],
       devices=["desktop", "mobile"]))

   Field note: Treat each matrix cell as immutable configuration. If you later change a keyword's tracked city, start a new series rather than mutating the old one — otherwise your history compares apples to oranges across the config change.

2. 2

   Collect each cell from the right place and device

   Use a residential exit in the cell's location and present the matching device profile (mobile vs desktop). Request 100 results so you capture deep rankings, and keep personalization off where possible. Geo and device must be real, not just URL parameters.

   Python (geo + device collection)Copy

   import requests
   
   UAS = {"desktop": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) ... Chrome/124.0",
          "mobile":  "Mozilla/5.0 (iPhone; CPU iPhone OS 17_0 like Mac OS X) ... Mobile/15E148"}
   
   def fetch_serp(job):
       country, city = job["location"]
       user = f"aethyn-XXXXX-country-{country}-city-{city.replace(' ','')}"
       proxy = f"http://{user}:PASSWORD@proxy.aethyn.io:5499"
       params = {"q": job["keyword"], "num": 100, "hl": job["language"], "pws": 0}
       return requests.get("https://www.google.com/search", params=params,
                           headers={"User-Agent": UAS[job["device"]]},
                           proxies={"http": proxy, "https": proxy}, timeout=30)

   Field note: Keep the device profile consistent for the life of a series. A keyword's mobile and desktop rankings genuinely differ, so mixing devices within one series produces phantom volatility that isn't real movement.

3. 3

   Parse organic results and SERP features separately

   Split the page into organic results (with their order) and SERP features (ads, People Also Ask, local pack, featured snippet, knowledge panel, AI overview). Tracking features is half the value — a competitor 'losing' rank often just got pushed below a new feature block.

   Python (parse organic + features)Copy

   from bs4 import BeautifulSoup
   
   def parse(html):
       soup = BeautifulSoup(html, "html.parser")
       organic = []
       for i, r in enumerate(soup.select("div.g a:has(h3)"), start=1):
           organic.append({"position": i, "url": r.get("href")})
       features = {
           "ads": len(soup.select("[data-text-ad]")),
           "paa": bool(soup.select_one("div[jsname][data-initq]")),
           "featured_snippet": bool(soup.select_one(".xpdopen .kp-blk")),
       }
       return {"organic": organic, "features": features}

   Field note: Selectors drift as engines tweak markup. Centralize them in one module and add a daily canary check on a known query — when organic count drops to zero, your parser broke, not the rankings.

4. 4

   Record organic AND visual position

   Position 1 organic can sit below ads, a featured snippet, and a PAA block — physically halfway down the page. Store the organic index and an estimate of visual position (features above it) so reports reflect real visibility, not just the organic ordinal.

   Field note: Visibility, not just rank, is what stakeholders feel. A 'rank 3' above the fold outperforms a 'rank 1' buried under three feature blocks — surfacing visual position prevents misleading wins and losses.

5. 5

   Store an append-only time series and diff for alerts

   Write one immutable row per (series, run timestamp) with positions and features. Diff each run against the previous to compute deltas, detect new/lost rankings and feature changes, and fire alerts only on meaningful movement to avoid noise. Schedule the full matrix on a cadence through rotating IPs.

   Python (diff for alerts)Copy

   def diff_runs(prev, curr, domain):
       def pos(run):
           for r in run["organic"]:
               if domain in (r["url"] or ""):
                   return r["position"]
           return None
       p, c = pos(prev), pos(curr)
       if p != c:
           return {"domain": domain, "from": p, "to": c}
       return None

   Field note: Alert on sustained moves, not single-run blips. Require a change to persist across two consecutive runs (or exceed a threshold) before notifying — SERPs are noisy and same-day fluctuations are usually not real ranking changes.

Best practices that keep scrapers reliable

• Model tracking as an immutable keyword × geo × device × language matrix
• Collect each cell from a real residential IP and matching device
• Parse organic and SERP features as separate first-class data
• Record both organic ordinal and visual/pixel position
• Store append-only history and diff runs for change detection
• Alert on sustained moves to keep the signal clean

Common mistakes that burn proxy budget

• Ignoring geo/device and comparing non-comparable results
• Grepping for your domain and missing SERP-feature context
• Mixing devices within one series and inventing volatility
• Reporting organic rank while ignoring features pushing it down
• Overwriting rows instead of keeping an append-only series
• Alerting on every single-run blip and drowning in noise

Common questions about this scraper problem