How to Monitor Brand Mentions

What goes wrong when this scraper fails

Brand, PR, and trust teams need to know when and where their brand is mentioned — for reputation, crisis response, and competitive intelligence. The hard parts aren't fetching one page: mentions are spread across many source types, results are localized (a complaint trending in one country may be invisible elsewhere), the same story gets syndicated into dozens of near-identical copies, and high-frequency polling across sources gets throttled. Effective monitoring is geo-aware multi-source collection plus disciplined dedup, classification, and alerting.

Why this failure mode happens

Brand conversation is inherently distributed and localized: search and news personalize and regionalize results, forums and marketplaces vary by locale, and syndication multiplies a single story across many domains. So a single-source, single-region monitor misses most mentions and double-counts the rest. And because timely monitoring means frequent polling across many sources, it needs geo-accurate IPs and rotation to stay reliable rather than getting blocked.

Challenges that make this hard to automate

• Covering many source types, not just one search engine
• Localized results that hide region-specific mentions
• Syndicated near-duplicates flooding the feed
• Classifying sentiment and severity to prioritize
• Frequent multi-source polling getting throttled

Approaches that usually fail

• Watching one search engine — misses forums, reviews, marketplaces
• Single-region collection — blind to geo-specific mentions
• No dedup — the same syndicated story counted dozens of times
• Manual searching — not timely or comprehensive

When residential proxies fix this — and when they cannot

Residential IPs across the regions you care about return the localized search, news, and forum results a real user there sees, so geo-specific mentions surface. A rotating pool lets you poll many sources frequently — for timely detection — without any single IP hitting velocity limits, which is what makes near-real-time multi-source monitoring practical.

How Aethyn residential proxies help here

Brand monitoring is geo-sensitive, multi-source, and high-frequency. Aethyn supplies geography and volume through the username.

• Country/city targeting to catch region-specific mentions
• Elite high-trust IPs for search/social sources that defend aggressively
• A large rotating pool for frequent polling across many sources
• Sticky sessions for multi-step source flows when needed
• Per-byte billing so continuous monitoring stays cost-predictable

How to implement this with residential proxies

1. 1

   Define your sources and query set

   List the source types that matter for your brand — search engines, news, key forums, review sites, marketplaces — and the queries (brand names, products, misspellings, executives, competitors). This explicit map is what makes coverage measurable instead of accidental.

   Python (source/query map)Copy

   SOURCES = ["google_news", "google_web", "reddit", "review_site", "marketplace"]
   QUERIES = ["aethyn", "aethyn proxies", "aethyn.io", "aethy proxies"]  # incl. misspellings
   REGIONS = [("us","new york"), ("gb","london"), ("de","berlin")]
   
   def jobs():
       for src in SOURCES:
           for q in QUERIES:
               for region in REGIONS:
                   yield {"source": src, "query": q, "region": region}

   Field note: Include common misspellings, handles, and product names — a lot of real conversation never uses your exact brand string, and competitors are mentioned in the same breath, which is valuable competitive signal.

2. 2

   Collect each source from the right region

   Run each job through a residential IP in its region so localized results surface. Geo-accurate collection is what lets you catch, say, a complaint trending on a German forum that a US-only monitor would never see.

   Python (geo collection)Copy

   import requests
   
   def fetch(job):
       country, city = job["region"]
       user = f"aethyn-XXXXX-country-{country}-city-{city.replace(' ','')}"
       proxy = f"http://{user}:PASSWORD@proxy.aethyn.io:5499"
       # source-specific URL/params built per job["source"]
       return requests.get(job["url"], proxies={"https": proxy}, timeout=30).text

   Field note: Keep a consistent region set so mention counts are comparable region-over-region and week-over-week. Ad-hoc geography makes trend lines meaningless.

3. 3

   Normalize and dedupe across sources

   Extract a normalized record per mention (source, url, title, snippet, date) and dedupe aggressively — syndicated stories appear as dozens of near-identical copies. Hash normalized title+snippet for exact dupes and use near-duplicate detection (e.g. MinHash) to collapse reworded syndications into one event.

   Python (near-dup collapse)Copy

   import re, hashlib
   
   def norm(text):
       return re.sub(r"\s+", " ", re.sub(r"[^a-z0-9 ]", " ", text.lower())).strip()
   
   def exact_key(m):
       return hashlib.sha256(norm(m["title"] + " " + m["snippet"]).encode()).hexdigest()
   
   # Group exact keys first; then run MinHash/LSH on the survivors to
   # collapse reworded syndications of the same story into one event.

   Field note: Cluster by story, not by URL. Leadership wants 'one negative story picked up by 40 outlets', not 40 separate alerts — collapsing syndication into a single event with a reach count is far more actionable.

4. 4

   Classify sentiment and severity

   Tag each mention cluster with sentiment and a severity estimate (reach of the source, negativity, whether it's spreading). This is what turns a firehose into prioritized signal so the team responds to a brewing crisis, not to routine positive chatter.

   Field note: Weight severity by source reach and velocity, not just sentiment. A mildly negative post going viral matters more than a scathing one on a dead forum — velocity is the early-warning signal for crises.

5. 5

   Alert on what matters and trend over time

   Fire alerts on high-severity or rapidly-spreading clusters, route the rest to a dashboard, and store everything as a time series so you can trend mention volume, sentiment, and share of voice. Poll on a cadence through rotating IPs and back off on any source that challenges.

Best practices that keep scrapers reliable

• Define an explicit source and query set, including misspellings
• Collect each source from its relevant region for localized mentions
• Dedupe exact, then collapse near-duplicate syndications by story
• Weight severity by source reach and velocity, not just sentiment
• Alert on high-severity/spreading clusters; dashboard the rest
• Store a time series for sentiment and share-of-voice trends

Common mistakes that burn proxy budget

• Monitoring one search engine and missing forums/reviews/marketplaces
• Collecting from one region and missing geo-specific mentions
• No dedup, so syndicated stories flood the feed
• Alerting on volume instead of severity and velocity
• Counting URLs instead of clustering by story
• High-frequency single-IP polling that gets throttled

Common questions about this scraper problem