Data at Frontier Scale

Every frontier model is shaped, at a fundamental level, by the data it was trained on. The model's knowledge of chemistry, its facility with Python, its understanding of historical events, its stylistic tendencies, and its embedded biases all originate in decisions made during data collection and curation — decisions that often receive far less public attention than model architecture choices. Data is not a solved problem at frontier scale. It is one of the most contested and consequential engineering challenges in the field.

How Much Data Does a Frontier Model Need?

Early large language models were trained on data measured in gigabytes. GPT-2 (2019) trained on roughly 40 gigabytes of text from web pages. Within a few years, frontier models were consuming data measured in terabytes, and the datasets underpinning models like GPT-4 and Claude 3 are estimated to contain several trillion tokens — roughly equivalent to millions of books. A token is the basic unit models process: roughly 0.75 words, or about 4 characters of English text. Three trillion tokens corresponds to approximately 15 trillion characters, or enough text to fill 15 million average-length books. The practical challenge is that the open internet, while enormous, does not contain infinite high-quality text. Labs training on trillions of tokens eventually must make careful decisions about what counts as high-quality data and how to weight different sources.

Data Sources: Where Does the Data Come From?

Frontier training corpora draw from multiple source categories, each with different characteristics. Web crawls are the largest single source. Organizations like Common Crawl regularly scrape the publicly accessible internet and release snapshots. A single Common Crawl snapshot contains petabytes of HTML. The challenge is that most of the raw internet is low-quality: spam, duplicated content, machine-generated junk, and incoherent text. Turning a raw web crawl into useful training data requires aggressive filtering. Books and long-form text provide high-quality, coherent language that differs structurally from the short, fragmented writing common on the web. Projects like Books3 and various digitized library collections have been widely used, though their legal status under copyright law is actively litigated. Code repositories, primarily GitHub, provide an enormous corpus of programming language. Models trained on substantial code demonstrate significantly improved logical reasoning, even on non-coding tasks — a finding that has shaped how frontier labs blend data sources. Scientific papers, Wikipedia, and curated encyclopedic sources provide factual density. Wikipedia alone, despite covering fewer total tokens than a large web crawl, is given high weight in many training mixes because of its relatively high factual accuracy and consistent structure. Synthetic data — text generated by a previous AI model — is increasingly used to fill gaps in coverage or to provide training examples in specialized domains. This practice raises questions about data provenance and the risk of 'model collapse' if models are trained too heavily on their own outputs.

Cleaning, Filtering, and Deduplication

Raw data from any source is not training-ready. A typical data pipeline applies multiple filtering stages. Language identification removes text not in the target language(s). Quality filtering uses heuristics or classifier models to score text for coherence, appropriate length, and absence of spam signals. A common heuristic is to check whether the text could plausibly have been written by a human producing genuine content. Pages with very high or very low word-per-sentence ratios, excessive repetition, or high proportions of special characters are often excluded. Deduplication removes near-duplicate documents. This matters for two reasons. First, memorizing duplicated content is not learning structure — it is rote storage. Second, heavily duplicated content distorts what the model learns to predict, causing it to over-represent whatever content happens to appear many times. Exact-match deduplication catches verbatim copies. MinHash or SimHash algorithms catch near-duplicates — documents that share most but not all of their text. Content filtering removes material the lab judges harmful or inappropriate for training. Exact hash matching against known databases of child sexual abuse material (CSAM) is standard. Additional filters attempt to reduce the presence of violent, hateful, or privacy-violating content, though these are imperfect. Data mixing assigns weights to different sources so that the final training corpus reflects a deliberate balance. A lab might decide that one token of high-quality curated text is worth ten tokens of raw web crawl, expressing this as a sampling probability.