How to Build LLM Training Datasets From the Web

What goes wrong when this scraper fails

Teams building pretraining or domain corpora often equate 'more web text' with 'better data'. In practice raw crawls are dominated by navigation boilerplate, near-duplicate copies of the same content, machine-generated spam, and personal data — and they frequently contain the very benchmarks you'll evaluate on. Train on that and you waste compute, encourage memorization, leak PII, and report inflated scores. Building an LLM dataset is a data-engineering pipeline: collect broadly, then filter, dedup, decontaminate, redact, and format into shards.

Why this failure mode happens

The web is built for humans and SEO, not for training: pages repeat headers/footers, syndicate the same articles across domains, and include large volumes of low-information or auto-generated text. Benchmarks are themselves published on the web, so crawls ingest them. And personal data is everywhere. None of this is removed by simply downloading more pages — it requires explicit processing stages, which is where dataset quality is actually won or lost.

Challenges that make this hard to automate

• Separating substantive text from boilerplate and spam
• Removing exact and near-duplicate documents at corpus scale
• Decontaminating against evaluation benchmarks
• Redacting PII and respecting licensing/AI opt-out signals
• Storing huge corpora in an efficient, shardable format

Approaches that usually fail

• Dumping raw crawl text into training — boilerplate, dupes, and PII included
• Exact-dedup only — misses reworded near-duplicates
• No decontamination — benchmark leakage inflates eval scores
• Flat single-file storage — unwieldy and slow to stream at scale

When residential proxies fix this — and when they cannot

Building a representative corpus means broad, geographically- and linguistically-diverse collection — exactly what a large residential pool with country targeting enables, while rotation keeps the crawl reliable at volume. Residential IPs also reach the regionally-varied content that datacenter ranges often can't, so the corpus isn't skewed toward a single locale. The heavy lifting after collection is pure data engineering.

How Aethyn residential proxies help here

The collection stage of a dataset pipeline needs scale and geographic/linguistic breadth. Aethyn provides both through the username.

• Country/city targeting to deliberately sample many regions and languages
• A large rotating pool for broad, high-volume crawling
• Premium pricing that's cost-efficient for big collection workloads
• Sticky sessions for multi-step source flows when needed
• Per-byte billing so a large crawl's cost stays transparent

How to implement this with residential proxies

1. 1

   Collect broadly and diversely, store raw

   Crawl your target sources through rotating residential IPs, deliberately sampling regions and languages so the corpus isn't skewed. Write raw documents with provenance (source URL, fetch time, region, content hash) to immutable storage — never filter in the collector, so you can reprocess as your filters improve.

   Python (collect with provenance)Copy

   import time, hashlib, requests
   
   PROXY = "http://aethyn-XXXXX-country-de:PASSWORD@proxy.aethyn.io:2099"
   
   def fetch(url, sink):
       r = requests.get(url, proxies={"https": PROXY}, timeout=30)
       if r.ok:
           sink.write({"url": url, "ts": time.time(), "region": "de",
                       "hash": hashlib.sha256(r.content).hexdigest(),
                       "html": r.text})

   Field note: Record per-document provenance and licensing/robots/AI-opt-out signals at collection time. You cannot reconstruct lawful basis or honor opt-outs after the fact — capture it when you fetch or you'll have an unauditable corpus.

2. 2

   Extract main text and quality-filter

   Strip boilerplate (nav, footers, ads) to main content, then filter hard: drop documents below a length threshold, with low text-to-markup ratios, garbled encoding, or signals of machine-generated spam. A smaller, cleaner corpus trains better than a huge noisy one.

   Python (quality filters)Copy

   def quality_ok(text):
       if len(text) < 200:                  # too short to be useful
           return False
       words = text.split()
       if len(words) < 50:
           return False
       # crude symbol-ratio guard against menus/garbled pages
       alpha = sum(c.isalpha() or c.isspace() for c in text) / max(1, len(text))
       return alpha > 0.7

   Field note: Use a lightweight quality classifier (e.g. trained to distinguish reference text from web spam) for the borderline middle. Hard heuristics catch the obvious junk; a classifier recovers the gray-area documents that simple rules wrongly drop.

3. 3

   Dedup exact, then near-duplicate

   Duplication causes memorization and wastes training tokens. Remove exact duplicates by content hash, then collapse near-duplicates (reworded syndications, templated pages) with MinHash + LSH. At corpus scale this is one of the highest-impact quality steps.

   Python (MinHash near-dedup)Copy

   from datasketch import MinHash, MinHashLSH
   
   def minhash(text, num_perm=128):
       m = MinHash(num_perm=num_perm)
       for tok in set(text.lower().split()):
           m.update(tok.encode())
       return m
   
   def dedup(docs, threshold=0.8):
       lsh = MinHashLSH(threshold=threshold, num_perm=128)
       keep = []
       for i, d in enumerate(docs):
           mh = minhash(d["text"])
           if not lsh.query(mh):          # no near-duplicate already kept
               lsh.insert(str(i), mh)
               keep.append(d)
       return keep

   Field note: Dedup across the whole corpus, not per-source. The same article syndicated across many domains is the dominant duplicate pattern, and per-source dedup leaves all the cross-domain copies in.

4. 4

   Decontaminate against benchmarks

   If evaluation data leaks into training, your scores are inflated and meaningless. Build a set of known benchmark items and remove training documents that contain them (exact and n-gram overlap). Treat decontamination as mandatory, not optional, for any model you'll evaluate.

   Field note: Decontaminate against the specific benchmarks you report on, using n-gram overlap rather than only exact match — paraphrased or reformatted test items still leak capability signal if left in the training mix.

5. 5

   Redact PII, then shard the output

   Redact personal data (emails, phones, and named entities where required) before training, not after. Then write the cleaned corpus as compressed, sharded JSONL or Parquet so it streams efficiently during training, and publish a dataset card documenting sources, filtering, language mix, and known limitations.

   Python (redact + shard)Copy

   import re, json, gzip
   
   EMAIL = re.compile(r"[\w.+-]+@[\w-]+\.[\w.-]+")
   PHONE = re.compile(r"\+?\d[\d\s().-]{7,}\d")
   
   def redact(t):
       return PHONE.sub("[PHONE]", EMAIL.sub("[EMAIL]", t))
   
   def write_shards(docs, shard_size=100_000, prefix="corpus"):
       for s in range(0, len(docs), shard_size):
           with gzip.open(f"{prefix}-{s//shard_size:05d}.jsonl.gz", "wt") as f:
               for d in docs[s:s+shard_size]:
                   f.write(json.dumps({"text": redact(d["text"]),
                                       "url": d["url"]}) + "\n")

   Field note: Shard by a fixed document count and keep shards independently streamable. Training pipelines parallelize over shards, so uniform, self-contained shards make data loading simple and let you mix data sources by sampling shards.

Best practices that keep scrapers reliable

• Collect broadly across regions/languages and store raw with provenance
• Quality-filter hard, using a classifier for borderline documents
• Dedup exact then near-duplicate across the whole corpus
• Decontaminate against the benchmarks you evaluate on
• Redact PII before training and honor AI opt-out/licensing
• Store as compressed, sharded JSONL/Parquet with a dataset card

Common mistakes that burn proxy budget

• Equating more raw web text with a better dataset
• Training on boilerplate, spam, and unfiltered low-quality pages
• Exact-dedup only, leaving reworded near-duplicates in
• Skipping decontamination and inflating eval scores
• Per-source dedup that misses cross-domain syndication
• Ignoring PII, licensing, and AI opt-out signals

Common questions about this scraper problem