How to Collect Web Data for AI Training Datasets

What goes wrong when this scraper fails

An ML team needs a large, varied corpus — multilingual text, listings, structured content — from across the public web to train or fine-tune a model. Three things make this hard. Scale plus diversity stresses any single-IP crawler into rate limits and geo-locked dead-ends, yielding a narrow sample. Raw web data is full of near-duplicates, boilerplate, and personal information that degrade training and create risk. And the legal landscape has shifted: provenance, licensing, and AI-specific opt-out signals now matter, so a dataset you can't account for is a liability even if every URL was reachable.

Why this failure mode happens

Volume and diversity together break naive crawlers: a single vantage point hits per-IP throttling and can't see geo-locked or region-specific content, so the corpus skews toward whatever one network and locale could reach. Meanwhile the open web is inherently messy — the same article appears on dozens of mirrors, pages are mostly nav and ads, and personal data is scattered throughout — so without deliberate dedup, cleaning, filtering, and provenance tracking, you train on a biased, noisy, unauditable pile.

Challenges that make this hard to automate

• Achieving scale and genuine geographic/language diversity at once
• Reaching geo-locked or region-specific content from the right vantage points
• Rate limits and blocks fragmenting very large crawls
• Near-duplicates and boilerplate that inflate size and harm training
• PII, licensing, and AI opt-out signals creating real compliance obligations

Approaches that usually fail

• Single-region crawlers — fast, but narrow and biased samples
• Datacenter proxies — blocked at scale and geographically uniform
• Purchased datasets — opaque provenance and licensing you inherit
• Manual curation — high quality but impossible at corpus scale

When residential proxies fix this — and when they cannot

Residential proxies give a dataset the geographic and network diversity that makes it representative, and rotation across a large pool keeps massive crawls running without per-IP throttling. The result is a corpus that's broad and multilingual by design rather than a single network's narrow, blocked view of the web.

How Aethyn residential proxies help here

Building a representative corpus needs reach (many countries), reliability at volume, and predictable cost. Aethyn provides diverse rotating residential capacity from one endpoint with per-byte billing.

• 195+ country coverage for genuinely diverse, multilingual corpora
• Per-request rotation for reliable large-scale crawling without throttling
• Country targeting to deliberately sample region- and language-specific data
• Premium tier for cost-efficient high-volume collection
• Per-byte metering so large dataset builds stay predictable

How to implement this with residential proxies

1. 1

   Define scope, sources, and a compliance policy first

   Specify domains, content types, languages, and regions — and, before any collection, set the rules: check each source's robots.txt and Terms, decide how you'll honor AI opt-out signals, and document how you'll handle licensing and personal data. For a training corpus this policy is part of the dataset; a model trained on undocumented data is hard to defend later.

   Field note: Beyond robots.txt, watch for AI-specific opt-outs: noai/noimageai directives, TDM (text-and-data-mining) reservations, and blocked AI crawler user-agents. Honoring them is both increasingly expected and a cheap way to reduce legal and reputational risk.

2. 2

   Crawl diverse regions through rotation

   Sweep target countries through rotating residential IPs so the corpus reflects multiple locales and languages rather than one network's view. Deliberate geographic sampling is how you fight the bias that creeps in when all data comes from a single vantage point.

   Python (multi-region crawl)Copy

   import requests
   
   REGIONS = ["us", "gb", "de", "br", "in", "jp"]
   
   def fetch(url, country):
       proxy = f"http://aethyn-XXXXX-country-{country}:PASSWORD@proxy.aethyn.io:2099"
       proxies = {"http": proxy, "https": proxy}
       return requests.get(url, proxies=proxies, timeout=30)
   
   for c in REGIONS:
       r = fetch("https://example.com", c)
       print(c, r.status_code)

   Field note: Track per-region document counts as you go and rebalance the crawl toward under-represented languages. Left unmanaged, English and a few large markets dominate and the model's weaker languages stay weak.

3. 3

   Deduplicate — exact and near-duplicate

   Deduplication is one of the highest-leverage steps in dataset building: near-duplicate documents inflate size, waste training compute, and increase memorization of repeated text. Use a content hash to drop exact duplicates, then a similarity technique (MinHash/LSH or SimHash) to catch near-duplicates like mirrored articles and templated pages.

   Python (dedupe)Copy

   import hashlib
   from datasketch import MinHash, MinHashLSH
   
   def exact_id(text):
       return hashlib.sha256(text.encode("utf-8")).hexdigest()
   
   def minhash(text, num_perm=128):
       m = MinHash(num_perm=num_perm)
       for token in set(text.lower().split()):
           m.update(token.encode("utf-8"))
       return m
   
   lsh = MinHashLSH(threshold=0.8, num_perm=128)   # ~80% similar = near-dup
   def is_near_dup(doc_key, text):
       m = minhash(text)
       if lsh.query(m):
           return True
       lsh.insert(doc_key, m)
       return False

   Field note: Run cheap exact-hash dedup first to shrink the set, then near-dup detection on what remains. Running MinHash/LSH over the full raw crawl is far more expensive than necessary.

4. 4

   Filter PII and low-quality content

   Strip boilerplate (nav, ads, cookie banners), detect and balance languages, drop low-quality pages, and remove or redact personal data such as emails and phone numbers. Doing this before training is cheaper than retrofitting and materially reduces the risk of the model regurgitating personal information.

   Python (PII redaction)Copy

   import re
   
   EMAIL = re.compile(r"[\w.+-]+@[\w-]+\.[\w.-]+")
   PHONE = re.compile(r"\+?\d[\d\s().-]{7,}\d")
   
   def redact_pii(text):
       text = EMAIL.sub("[EMAIL]", text)
       text = PHONE.sub("[PHONE]", text)
       return text   # extend with NER for names/addresses where needed

   Field note: Also guard against benchmark contamination: filter out documents that contain known evaluation sets. If test data leaks into training, your eval scores are inflated and meaningless.

5. 5

   Record provenance and publish a dataset card

   Store source URL, fetch timestamp, region, content hash, and any license signal with every document so the corpus is auditable and reproducible. Summarize the collection method, filtering, language distribution, and known limitations in a dataset card (datasheet) — it's what lets others (and future you) trust and reuse the data responsibly.

Best practices that keep scrapers reliable

• Set a compliance policy (robots, Terms, AI opt-outs) before collecting
• Sample geography and language deliberately, and rebalance under-represented locales
• Dedup exact first, then near-duplicates with MinHash/SimHash
• Redact PII and strip boilerplate before training, not after
• Filter known benchmarks to avoid evaluation contamination
• Record per-document provenance and publish a dataset card

Common mistakes that burn proxy budget

• Collecting from one region and baking sampling bias into the corpus
• Skipping near-duplicate detection, inflating size and memorization
• Training on raw boilerplate and unfiltered personal data
• Ignoring robots.txt, licensing, and AI opt-out signals
• Letting benchmark data leak into training and inflating eval scores
• Storing no provenance, leaving the dataset unauditable and hard to defend

Common questions about this scraper problem