Data Cleaning and Deduplication for LLM Pretraining Corpora

Why Corpus Quality Matters More Than Corpus Size

The Chinchilla scaling paper (Hoffmann et al., 2022, DeepMind) demonstrated that for a given compute budget, models trained on more tokens at smaller model size consistently outperform models trained on fewer tokens at larger model size. The practical implication is widely understood: train on more data. The less-discussed implication is that quality matters: their analysis assumed high-quality deduplicated corpora. When the training corpus contains 40–60% near-duplicate documents — common in raw Common Crawl snapshots — effective token count is far lower than raw token count, and the quality gap narrows the Chinchilla benefit.

Lee et al. (2022, Google Research) studied deduplication directly and found that training on a deduplicated C4 corpus produced models with 5–20 lower perplexity points on held-out evaluation sets compared to models trained on the full duplicated corpus at matched training compute. Memorisation of training examples — a privacy and safety risk — dropped significantly: the probability of verbatim extraction of a training document fell by 50–70% after exact deduplication. These findings held across model sizes from 1B to 137B parameters.

The implication for teams building domain-specific LLMs or instruction-tuned models on proprietary corpora is direct: budget for corpus preparation before training begins. The cost of deduplication and quality filtering at training time is fixed and predictable. The cost of retraining a model on a corrected corpus after discovering that your raw corpus contained 45% near-duplicates is not.

Exact Deduplication: SHA-256 Hashing at Scale

Exact deduplication removes documents that are character-for-character identical. It is implemented using cryptographic hashing — SHA-256 or MD5 — computed over the normalised document text (lowercased, whitespace-normalised, stripped of leading and trailing punctuation). Documents with matching hashes are exact duplicates; one copy is retained and the rest discarded.

Exact deduplication is fast and parallelises trivially: hash computation is O(n) in document length and the global dedup operation is a sort or hash-table lookup over the hash set. For a 1 trillion token web corpus, exact deduplication typically removes 8–18% of documents. The removal rate is higher for curated sources that are widely redistributed: Creative Commons books, Wikipedia snapshots, GitHub repositories, and news wire content appear as exact copies across hundreds of web pages.

Normalisation before hashing is critical. Two documents that differ only in whitespace, capitalisation, or HTML entity encoding (& vs &) are functionally identical. A deduplication pipeline that does not normalise before hashing will miss a significant fraction of true exact duplicates. Standard normalisation includes: Unicode NFC normalisation, whitespace collapsing, lowercasing, and HTML entity decoding.

Near-Duplicate Removal: MinHash LSH at Trillion-Token Scale

Near-duplicate removal identifies documents that are sufficiently similar to be informationally redundant without being character-for-character identical. Near-duplicates in web corpora include: the same news article scraped from the syndication feed and the original publisher page, book chapters that appear across multiple e-book distribution sites, forum posts that are copied with minor modifications, and templated product descriptions that differ only in product name. These near-duplicates inflate raw corpus size without contributing proportional unique signal.

MinHash locality-sensitive hashing (LSH) is the standard near-duplicate detection algorithm for billion-to-trillion-token corpora. The algorithm works as follows: each document is decomposed into n-gram shingles (typically 5-grams or 13-grams at the character or word level), a MinHash signature of k hashes (typically 128 or 256) is computed over the shingle set, and documents whose MinHash signatures share sufficient hash bands — indicating likely Jaccard similarity above the dedup threshold — are identified as near-duplicate candidates. Candidate pairs are then verified by exact Jaccard computation on their full shingle sets.

Jaccard similarity threshold selection is the critical parameter decision. A threshold of 0.70 removes documents that share 70% or more of their shingles — appropriate for syntactic near-duplicates (the same text with minor edits). A threshold of 0.85 removes only very close near-duplicates. A threshold of 0.50 is aggressive and may remove thematically similar but genuinely distinct documents. Most production LLM deduplication pipelines use 0.75–0.80 for web text and 0.85–0.90 for domain-specific corpora where syntactic variation is higher.

Quality Filtering: Removing Low-Signal Documents

Deduplication removes redundancy; quality filtering removes low-signal documents that are unique but contribute poor training signal. The five standard quality filters for LLM pretraining corpora are perplexity filtering, length filtering, repetition filtering, boilerplate filtering, and language identification.

Perplexity Filtering

Perplexity filtering uses a reference language model — typically a KenLM 5-gram language model trained on a high-quality curated corpus — to score each document. Documents with high perplexity (above the 90th or 95th percentile of the corpus perplexity distribution) contain unusual token sequences relative to the reference model: garbled OCR output, non-natural-language text (encoded data, serialised JSON), machine-translated low-quality text, or heavily templated content with low lexical diversity.

Perplexity filtering must be calibrated to the target domain. A KenLM model trained on English Wikipedia will assign high perplexity to legitimate domain-specific text (medical case reports, legal contracts, code comments) that uses specialised vocabulary. Domain-specific LLM pretraining pipelines typically use domain-matched KenLM reference models or replace perplexity filtering with content-type classification for domain-specific corpus segments.

Repetition and Length Filtering

Repetition filtering removes documents where a short n-gram appears an abnormally high number of times — indicating templated, machine-generated, or spam content. A document where any 20-gram appears more than 4 times, or where the top-1 character n-gram accounts for more than 30% of all n-grams, is flagged for removal. This catches navigation menus repeated across scraped pages, keyword-stuffed SEO content, and forum spam.

Length filtering removes documents that are too short (under 150–200 characters, typically navigation fragments or cookie consent banners) or pathologically long (over 100,000 characters, typically entire e-commerce catalogue dumps or repeatedly-appended log files). Both extremes contribute disproportionate noise to pretraining relative to their unique informational content.

PII Redaction and Toxic Content Filtering

PII redaction is a legal requirement for organisations operating under the Australian Privacy Act 1988 and a best-practice requirement for LLM training data. Patterns to redact include: Australian phone numbers (04xx xxx xxx), email addresses, Tax File Number patterns (DDD DDD DDD), Medicare number patterns, physical addresses with postcode patterns, and names in contexts that indicate personal rather than public reference.

Regex-based PII redaction is fast and sufficient for high-confidence patterns. Named entity recognition (NER) models fine-tuned on PII detection — such as Presidio or GLiNER — handle ambiguous cases (a name that might be a public figure or a private individual). Toxic content filtering uses classifier models (fastText or BERT-based) trained on hate speech, CSAM-adjacent content, and harmful instructions to flag or remove documents that would degrade model safety characteristics.

Our data QA and validation service includes annotator-in-the-loop review for PII edge cases that regex and NER models cannot reliably resolve — such as names that appear in medical records context versus public biographical context.

Case Study: Legal AI Startup — 19-Point Perplexity Improvement from Structured Corpus Cleaning

A legal AI startup developing a contract analysis model for Australian corporate law assembled a 28 billion token pretraining corpus from six sources: Common Crawl legal domain segments, Australian court decision databases, legal publisher APIs, law school open access repositories, parliamentary Hansard transcripts, and a proprietary corpus of anonymised client contracts. The team planned to begin pretraining immediately without a deduplication or quality filtering pass, having prioritised corpus size over corpus quality.

Before: A corpus audit conducted before training found that the raw 28 billion token corpus contained significant quality problems. Exact deduplication identified 4.1 billion tokens (14.7%) as exact duplicates — primarily court decisions and statutory instruments that were present across both official court databases and law school repositories. Near-duplicate removal at 0.80 Jaccard threshold identified a further 8.9 billion tokens (31.8%) as near-duplicates — the same court decisions appearing in multiple scrape of the same sources and legal commentary that paraphrased judgments with minor variation. After deduplication, the effective unique corpus was 15.0 billion tokens — 53.6% of the raw corpus size.

Quality filtering on the deduplicated corpus identified an additional 2.1 billion tokens (14%) for removal: 0.8 billion tokens of boilerplate (navigation, cookie notices, advertisement text from legal publisher sites), 0.7 billion tokens of high-perplexity text (garbled OCR from older court documents, serialised XML metadata), and 0.6 billion tokens of repetitive content (templated contract clauses appearing more than 10 times in sequence). PII redaction modified 340 million tokens across 1.2 million documents. The cleaned corpus was 12.9 billion unique, high-quality tokens.

After: The model trained on the cleaned 12.9 billion token corpus achieved a held-out legal text perplexity of 18.3 — compared to a preliminary model trained on 5 billion raw tokens that had achieved perplexity of 37.1. On Australian contract clause classification benchmarks (NDA, limitation of liability, IP assignment, governing law), the cleaned-corpus model achieved 91.4% accuracy versus 82.7% for the raw-corpus baseline. Downstream contract analysis recall on the team's internal evaluation set improved from 78.3% to 93.1%. The corpus cleaning process cost AUD $42,000 and took three weeks; the team estimated the cleaning prevented two retraining cycles that would have cost approximately AUD $180,000 in compute.

Domain-Specific Corpus Cleaning Considerations

Web-scale LLM deduplication pipelines (C4, RefinedWeb, Dolma, RedPajama) are designed for general-purpose English web text. Domain-specific LLM pretraining corpora require adjusted parameters at every stage of the cleaning pipeline.

Medical corpora (clinical notes, radiology reports, pathology narratives) contain legitimate repetition in structured fields (patient history sections, template language in discharge summaries) that repetition filters will incorrectly remove. Length filters calibrated for web text will remove short but high-value clinical notes. Perplexity filters using web-text KenLM models will flag medical jargon as low-quality. Medical corpus cleaning requires domain-matched KenLM reference models and relaxed repetition filter thresholds, with annotator review for filtered documents in clinical domains.

Code corpora (GitHub repositories, Stack Overflow, documentation) have different deduplication semantics: the same function copied across forks is a near-duplicate in some senses but a signal that the function represents common practice in others. Most code LLM teams use file-level exact deduplication but apply near-duplicate removal only at the repository level rather than the file level. Language identification filters must be disabled for code or replaced with programming language identification.

For annotator-in-the-loop quality review of domain-specific filtered documents, see our data QA and validation services and our post on annotation QA process design.

Deduplication Across Training Stages

A common oversight in multi-stage LLM training pipelines — pretraining, then continued pretraining on domain-specific data, then instruction tuning — is applying deduplication only within each stage rather than across stages. Documents that appear in both the pretraining corpus and the instruction tuning dataset are effectively upweighted during training: the model sees the content during pretraining and again during instruction tuning. This produces memorisation effects and can cause the instruction-tuned model to reproduce verbatim passages from pretraining in response to instruction prompts — a privacy risk if the pretraining corpus contained PII.

Cross-stage deduplication requires maintaining document-level hashes across all corpus stages and running deduplication against the combined hash set when adding each new stage. For very large corpora where full cross-stage near-duplicate deduplication is computationally prohibitive, fuzzy near-dedup across stages can be approximated by blocking on MinHash band signatures.

Related reading: our post on RLHF vs DPO preference data collection and our guide to instruction tuning dataset design cover the downstream training stages where corpus quality decisions propagate.

Related Reading

• Annotation QA Process Guide
• RLHF vs DPO: Which Preference Data Should You Collect?
• Instruction Tuning Dataset Design
• Data QA & Validation Services

Frequently Asked Questions