How to Build a Price Intelligence System

What goes wrong when this scraper fails

A company that competes on price wants a system that, every morning, tells pricing exactly where they stand against competitors across marketplaces and regions — with history, alerts, and enough trust to drive real decisions. A single scraper script cannot deliver that. It gets blocked, breaks on a layout change with no way to recover the lost days, mixes currencies, keeps no history, and silently degrades until someone notices the dashboard is wrong. The actual problem is designing a durable pipeline where each concern is isolated and observable.

Why this failure mode happens

One-off scrapers don't survive production because they couple everything: fetching, parsing, currency handling, and presentation all live in one fragile script with no buffer between stages. When a target blocks the crawler, parsing stops; when a layout changes, you lose data permanently because the raw HTML was never kept; when one currency is misparsed, it pollutes the dashboard directly. Without decoupling, durable raw storage, and quality monitoring, the data is incomplete, stale, and untrustworthy precisely when pricing needs to rely on it.

Challenges that make this hard to automate

• Reliable, geo-aware collection across many heterogeneous sources
• Recovering history when a parser or layout changes after the fact
• Normalizing currency and units and matching equivalent products across sites
• Storing time-series history that supports trends and audits
• Detecting meaningful changes and surfacing data-quality regressions without alert fatigue

Approaches that usually fail

• Spreadsheets and manual checks — stale the moment they're saved
• A single monolithic scraper — fragile, unrecoverable, and blocked at scale
• Bought price feeds — limited coverage, unknown freshness, and not your exact SKUs
• A BI dashboard with no reliable, monitored ingestion underneath it

When residential proxies fix this — and when they cannot

Every stage downstream depends on the collection layer actually returning correct, geo-accurate pages on schedule — so that layer's reliability is the ceiling for the whole system. Residential proxies with rotation and country targeting keep ingestion working across many sources and regions, giving normalization, matching, and serving trustworthy input instead of a stream of blocks and wrong-region prices.

How Aethyn residential proxies help here

In a price intelligence system the proxy network is the foundation of the collection stage. Aethyn is built to be that backbone: one endpoint, per-source tier selection, rotation, and per-byte cost control for continuous crawling.

• Elite reputation for the high-security marketplaces usually in the source mix
• Premium for general retail sources, switchable per source by port
• Country targeting so regional pricing is captured accurately at ingestion
• Per-request rotation for durable, scheduled, high-volume collection
• Per-byte metering so continuous crawling stays cost-predictable

How to implement this with residential proxies

1. 1

   Design decoupled stages around a queue

   Lay out five stages connected by a message queue (SQS, Kafka, or Redis Streams): collect → store raw → parse → normalize/match → serve. Decoupling means a blocked crawler doesn't stop parsing of already-fetched pages, a parser bug doesn't lose data, and you can scale or redeploy any stage independently. Add a dead-letter queue so poison records are quarantined instead of stalling the line.

   Field note: The single most valuable design decision is persisting raw HTML (e.g. to S3) before parsing. When a site changes layout — or you find a parser bug — you re-run parsing over stored raw pages and recover history, instead of staring at a permanent gap in your charts.

2. 2

   Build the proxied collection layer

   Crawl each source from the correct country through rotating residential IPs, selecting the tier per source by port (Elite 5499 for marketplaces, Premium 2099 for general retail). Emit the raw payload to storage/queue with enough metadata to trace and re-parse it later — never parse inline here.

   Python (collector)Copy

   import requests, time, hashlib
   
   def collect(url, country, tier_port=5499):
       proxy = f"http://aethyn-XXXXX-country-{country}:PASSWORD@proxy.aethyn.io:{tier_port}"
       proxies = {"http": proxy, "https": proxy}
       r = requests.get(url, proxies=proxies, timeout=30)
       return {
           "url": url,
           "country": country,
           "tier_port": tier_port,
           "status": r.status_code,
           "fetched_at": time.time(),
           "html_key": store_raw(r.text),          # persist raw -> object store
           "content_hash": hashlib.sha256(r.content).hexdigest(),
       }

   Field note: Store the content_hash and skip re-emitting downstream when a page is byte-identical to last crawl. Most product pages don't change between crawls; deduping on hash cuts parsing load and proxy bandwidth dramatically.

3. 3

   Parse and normalize as separate concerns

   Parse raw HTML into typed fields, then normalize in its own step: parse the localized price to a (amount, currency) pair and convert to a base currency with a pinned daily FX snapshot. Keeping parse and normalize separate means an FX or rounding change never requires re-fetching pages.

   Python (normalize)Copy

   import re
   
   def parse_price(text, decimal=","):
       num = re.sub(r"[^\d.,]", "", text)
       num = num.replace(".", "").replace(",", ".") if decimal == "," else num.replace(",", "")
       return round(float(num), 2)
   
   def to_base(amount, currency, rates):
       # rates = pinned daily snapshot, relative to base currency
       return round(amount / rates[currency], 2)

   Field note: Pin FX to a daily snapshot and store both local and base price. Converting with live rates makes every product look like it changed price on every crawl — currency drift masquerading as competitor moves.

4. 4

   Match products as a dedicated service

   Treat product matching as its own component, not a line in the parser. Match on stable identifiers (GTIN/UPC/MPN) first, fall back to attribute + fuzzy-title matching, and attach a confidence score. Route low-confidence matches to human review so one bad mapping can't quietly drive a pricing decision.

   Python (matching)Copy

   def match(record, catalog):
       if record.get("gtin") and record["gtin"] in catalog.by_gtin:
           return catalog.by_gtin[record["gtin"]], 1.0          # exact
       cand, score = catalog.fuzzy(record["title"], record.get("brand"))
       return cand, score                                       # score in [0,1]
   
   # downstream: auto-accept >= 0.9, queue 0.6-0.9 for review, drop < 0.6

   Field note: Persist the match decision and its score, not just the final link. When pricing disputes a comparison, being able to show 'matched on GTIN, confidence 1.0' versus 'fuzzy title, 0.7' is the difference between trust and a credibility problem.

5. 5

   Store time-series history and monitor data quality

   Append every observation to a time-series store (TimescaleDB/Postgres) keyed by (product, source, region, timestamp). Then monitor the pipeline like a product: track coverage (% of SKUs successfully collected), freshness (age of the newest data per source), and parse-success rate. A drop in any of these is a data-quality incident, even when nothing has 'errored'.

   Field note: Alert on coverage and freshness, not just exceptions. The dangerous failure mode isn't a crash — it's a source quietly slipping from 98% to 60% coverage while every individual job reports success.

6. 6

   Serve dashboards and threshold alerts

   Expose the normalized, matched, historical data to BI dashboards and push change alerts — using the same threshold-plus-confirmation logic that suppresses false positives — to where pricing teams actually work. The serving layer reads from storage only; it never touches the scrapers, so a collection hiccup never takes the dashboard down.

   Python (change detection)Copy

   def is_real_change(history, new, threshold=0.01):
       # history: recent base prices, oldest -> newest
       if not history:
           return False                      # first observation: record, don't alert
       last = history[-1]
       delta = abs(new - last) / last
       if delta < threshold:
           return False                      # within noise band
       # confirm: the previous crawl must already agree with 'new'
       return len(history) >= 2 and abs(history[-1] - new) / new < threshold

Best practices that keep scrapers reliable

• Decouple collection, parsing, normalization, matching, and serving with a queue
• Persist raw HTML before parsing so you can re-parse and recover history
• Make product matching a service with confidence scores and human review
• Pin FX daily and store both local and base price
• Monitor coverage, freshness, and parse-success as core SLOs
• Geo-target collection per source and select tier by port
• Capture promo badges, availability, and seller alongside price, not price in isolation
• Alert with hysteresis — a threshold plus confirm-on-next-crawl — and record the first observation silently so a new SKU never fires a phantom change

Common mistakes that burn proxy budget

• Wiring scrapers directly to dashboards with no buffer between stages
• Parsing inline and never keeping raw HTML, so layout changes lose history
• Burying product matching in the parser with no confidence or review
• Converting with live FX, turning currency drift into phantom changes
• Storing only the latest price with no time-series history
• Watching only for crashes while coverage silently decays
• Setting the exit country but leaving Accept-Language mismatched, so a German exit sending en-US quietly returns the English/EUR variant instead of the local price

Common questions about this scraper problem