Data Collection and Sourcing

'More data beats a better algorithm' is a phrase you will hear often in ML circles, and it contains real truth. Models learn by generalizing from examples, and examples are data. But the phrase is also incomplete and sometimes dangerously misleading. The quantity of data matters enormously — and so does its quality, its representativeness, and the process by which it was collected. A large dataset that is biased, stale, or mislabeled will train a model that is biased, stale, or wrong. This lesson examines where training data comes from and how to think critically about what it can and cannot represent.

Sources of Training Data

Training data comes from many sources, and each source carries its own characteristics and risks. Organically collected data is gathered as a byproduct of normal operations. A bank accumulates years of loan records with outcomes. An e-commerce site logs every click and purchase. This data is often large, cheap, and highly relevant — but it reflects the decisions made under the old system, which can encode historical biases into the new model. Manually labeled data is created specifically for training. Workers — sometimes professional annotators, sometimes crowd workers on platforms like Amazon Mechanical Turk — are shown inputs and asked to provide labels. This is expensive but gives you control over label quality. Medical imaging datasets, for example, typically require expert radiologists to label, costing thousands of dollars per case. Synthetic data is generated algorithmically rather than collected from the world. A self-driving car team might simulate thousands of rare accident scenarios that real-world collection would take years to encounter. Synthetic data avoids privacy concerns and can cover edge cases deliberately — but only if the simulation is realistic enough.

Consider building a hate-speech classifier for a social platform. The data source is user posts. The problem: only a small fraction of posts are hate speech. If you sample randomly, your dataset is 98% benign and 2% harmful. You need to deliberately oversample the minority class — collect more harmful examples — to build a useful classifier. This is called stratified sampling: you sample each class at a rate that gives your model enough examples of each to learn the distinction. Another example: a facial recognition system trained on a dataset that is 80% photographs of lighter-skinned adults will perform worse on darker-skinned faces and children — not because the algorithm is inherently biased, but because the data was not representative of all the people the system will encounter in production. The sampling strategy determines whom the model learns about.

Quantity Versus Quality

The tension between data quantity and quality is real and important. More data, holding quality constant, almost always helps. But quality cannot be held constant — every expansion of a dataset introduces new risks. Label noise means some training labels are simply wrong. A crowd worker may mislabel a photograph. A sensor may malfunction. A historical record may have been entered incorrectly. Studies have shown that state-of-the-art models can tolerate surprisingly high label noise rates (sometimes up to 20-30%) and still perform well, because the correct labels in the majority overwhelm the incorrect ones during training. But above a threshold — and for certain task types — label noise degrades performance severely. Data staleness occurs when the world changes after the data was collected. A model trained on user behavior from 2019 may not represent 2024 behavior at all. The more your domain changes over time — social media trends, market conditions, language use — the more quickly your training data expires. The practical guidance: treat data quality issues as debts. Small debts are manageable; large ones compound. Document every known quality problem in your dataset, estimate its likely impact, and build monitoring to catch new problems after deployment.