How to Build a B2B Lead Generation Pipeline

What goes wrong when this scraper fails

Sales and growth teams want a steady flow of qualified B2B leads from public sources — company directories, maps, public professional profiles — enriched with firmographics and reachable contacts. The failure mode is treating it as one big scrape: the output is full of duplicates, malformed fields, dead domains, and undeliverable addresses, and it ignores privacy obligations. Sending to that list damages deliverability and creates legal exposure. The real work is the pipeline around collection — reconciliation, enrichment, validation, and compliance.

Why this failure mode happens

Lead data comes from many sources with different schemas, overlapping coverage, and varying freshness, so raw collection is inherently messy and duplicated. Email and firmographic accuracy decay constantly as people change roles and companies. And B2B outreach is regulated, so a pipeline that doesn't bake in lawful basis, suppression, and validation produces lists that are simultaneously low-quality and non-compliant.

Challenges that make this hard to automate

• Reconciling overlapping records from multiple sources
• Deduping before wasting enrichment spend on duplicates
• Verifying email deliverability without harming sender reputation
• Keeping firmographics fresh as companies and roles change
• Meeting GDPR/CAN-SPAM obligations (basis, suppression, opt-out)

Approaches that usually fail

• One monolithic scraper dumping rows straight into the CRM — dirty and duplicated
• Buying static lead lists — stale, generic, and often non-compliant
• Manual prospecting in spreadsheets — accurate but unscalable
• Skipping validation — high bounce rates that wreck deliverability

When residential proxies fix this — and when they cannot

The sourcing stage queries many public sources at volume; routing it through rotating residential IPs keeps those collectors reliable instead of getting blocked partway through a run, and geo-targeting returns the correct regional business results. With collection stable, the rest of the pipeline (dedupe, enrich, validate) can run on a dependable input stream.

How Aethyn residential proxies help here

Lead sourcing is high-volume, lightweight collection across many directories and maps — a strong fit for Premium. Aethyn keeps the collection stage reliable through the username.

• Premium residential pool — cost-efficient for high-volume source collection
• Per-request rotation so multi-source sweeps don't get blocked mid-run
• Country/city targeting for region-specific company and directory data
• Sticky sessions for multi-step listing or profile flows
• Per-byte billing so a continuous sourcing pipeline stays predictable

How to implement this with residential proxies

1. 1

   Stage 1 — Source from public data through proxies

   Collect candidate companies and public business contacts from directories, maps, and public professional pages, routed through rotating residential IPs. Write raw, unmodified records to a staging store with source and timestamp — never transform in the collector, so you can reprocess later.

   Python (collector)Copy

   import requests, time
   
   PROXY = "http://aethyn-XXXXX-country-us:PASSWORD@proxy.aethyn.io:2099"
   proxies = {"http": PROXY, "https": PROXY}
   
   def collect(source_urls, sink):
       for url in source_urls:
           r = requests.get(url, proxies=proxies, timeout=30)   # rotates per call
           if r.ok:
               sink.write_raw({"source": url, "html": r.text, "ts": time.time()})

   Field note: Keep collection and parsing as separate stages writing through a queue. When a source changes its markup, you re-run only the parser over stored raw payloads instead of re-scraping the whole web.

2. 2

   Stage 2 — Normalize to one schema

   Parse raw records into a single canonical schema: company name, domain, address, phone, industry, size, and any public contact. Standardize domains (strip www, lowercase), phones (E.164), and names so downstream matching works.

   Python (normalize)Copy

   import re
   from urllib.parse import urlparse
   
   def domain_of(url_or_email):
       if "@" in (url_or_email or ""):
           return url_or_email.split("@")[-1].lower()
       host = urlparse(url_or_email).netloc.lower()
       return host[4:] if host.startswith("www.") else host
   
   def normalize(raw):
       return {
           "company": re.sub(r"\s+", " ", raw["name"]).strip(),
           "domain": domain_of(raw.get("website", "")),
           "phone": "+" + re.sub(r"\D", "", raw.get("phone", "")),
           "industry": raw.get("industry"),
       }

   Field note: Company domain is the backbone identifier of a B2B record — far more stable than a name. Resolve it early (from the website or email) and carry it through every later stage.

3. 3

   Stage 3 — Dedupe before you spend on enrichment

   Collapse duplicates on a composite key (domain + normalized company name) before enrichment, because enrichment is the expensive step and you don't want to pay for it twice. Merge duplicates, keeping the richest non-null fields and source provenance.

   Python (dedupe)Copy

   def dedupe(records):
       by_key = {}
       for r in records:
           key = (r["domain"] or r["company"].lower())
           if key in by_key:
               for k, v in r.items():          # fill gaps from duplicates
                   by_key[key].setdefault(k, None)
                   by_key[key][k] = by_key[key][k] or v
           else:
               by_key[key] = dict(r)
       return list(by_key.values())

   Field note: Dedup on domain first; companies appear under many name variants ('Acme', 'Acme Inc', 'Acme Corporation') but share one domain. Deduping on name alone leaves obvious duplicates in the list.

4. 4

   Stage 4 — Enrich firmographics

   Augment each deduped company with firmographic signals from its own public web presence (industry, size hints, tech stack, locations) so reps can segment and prioritize. Route enrichment fetches through proxies too, and cache aggressively — firmographics change slowly.

   Field note: Score and prioritize during enrichment: fit signals (industry, size, region) decide which leads are worth a rep's time. A ranked list of 200 good-fit accounts beats 5,000 unscored rows every time.

5. 5

   Stage 5 — Validate deliverability, then export

   Before anything reaches the CRM, validate contact emails: syntax, then domain MX records, and flag catch-all domains. Drop or quarantine invalid and risky addresses so you don't bounce. Only then upsert clean, scored, compliant records to the CRM.

   Python (email validation)Copy

   import re, dns.resolver
   
   EMAIL_RE = re.compile(r"^[^@\s]+@[^@\s]+\.[^@\s]+$")
   
   def deliverable(email):
       if not EMAIL_RE.match(email or ""):
           return False
       domain = email.split("@")[-1]
       try:
           return len(dns.resolver.resolve(domain, "MX")) > 0
       except Exception:
           return False

   Field note: MX validation catches dead domains cheaply and safely. Avoid aggressive SMTP 'ping' verification at scale — it can get your verifying IPs blacklisted and damage the very deliverability you're trying to protect.

6. 6

   Stage 6 — Bake in compliance

   Record a lawful basis and source for every contact, maintain a suppression list, honor opt-outs and do-not-contact requests automatically, and keep collection to public B2B data. Make compliance a pipeline stage, not an afterthought — it protects deliverability and reduces legal risk.

Best practices that keep scrapers reliable

• Build decoupled stages connected by queues, not one script
• Resolve and key on company domain throughout
• Dedupe before enrichment to avoid paying twice
• Validate syntax + MX (not aggressive SMTP) before the CRM
• Score for fit so reps work the best leads first
• Make lawful basis, suppression, and opt-out a built-in stage

Common mistakes that burn proxy budget

• Dumping raw scrapes straight into the CRM
• Deduping on name only and leaving domain duplicates
• Enriching before deduping and wasting budget on duplicates
• Skipping validation and bouncing into a reputation hole
• Aggressive SMTP verification that blacklists your IPs
• Ignoring GDPR/CAN-SPAM basis, suppression, and opt-outs

Common questions about this scraper problem