Data Quality for Machine Learning: The Unglamorous Work That Decides Results

When a machine learning model underperforms, the instinct is to reach for a bigger model or a new algorithm. More often, the problem sits upstream in the data: labels that disagree with each other, a column that secretly encodes the answer, or training records that look nothing like what the model will face in production. Model choice matters, but data quality usually sets the ceiling.

This guide covers the data problems that most often derail projects, how to detect each one, and the habits that keep them from coming back.

Why data quality matters more in ML

In traditional software, bad data usually causes visible failures: an error, a crash, a report with an obviously wrong total. In machine learning, bad data rarely breaks anything. The pipeline runs, the model trains, and a metric comes out. The damage is silent. The model simply learns the wrong thing, or learns the right thing from a distorted picture of the world.

A model is a compressed summary of its training data. It will faithfully reproduce missing segments, systematic labeling errors, and historical quirks, and then apply them at scale.

The dimensions of data quality

"Clean data" is vague. It helps to break quality into specific dimensions you can check.

Accuracy

Do values reflect reality? A customer's recorded age of 212, a delivery timestamp before the order timestamp, or a price entered in cents instead of dollars are accuracy problems.

Completeness

Are required values present, and are all relevant records included? Missing values are the obvious case. Missing rows are sneakier: a logging bug that dropped mobile events for two weeks, or an export that silently capped at a fixed row limit.

Consistency

Do the same things look the same everywhere? "US," "USA," and "United States" in one column. Revenue recorded gross in one system and net in another. Timestamps in local time from one source and UTC from another.

Validity

Do values follow expected formats and ranges? Emails without an "@," negative quantities, category codes that do not exist in the reference list.

Timeliness

Is the data current enough for the decision? A model predicting inventory needs based on data refreshed monthly may be stale for a fast-moving product line.

Representativeness

Does the dataset cover the situations the model will meet? This dimension is specific to ML and often the most important. A perfectly accurate dataset of only desktop users is still wrong for a mobile-heavy product.

Uniqueness

Are records duplicated? Duplicates distort class balance, inflate the importance of certain patterns, and can leak between training and test sets.

Labels: the part everyone underestimates

For supervised learning, labels are the definition of success. Label problems do more damage than feature problems, because the model treats labels as ground truth.

Where label noise comes from

• Ambiguous guidelines. If the instructions for "urgent" support tickets are vague, different annotators will draw the line in different places.
• Genuinely hard cases. Some examples could reasonably get either label. Forcing a single answer hides that uncertainty.
• Fatigue and speed pressure. Annotators working through large queues make more mistakes later in a session.
• Proxy labels. Many "labels" are really stand-ins. "Customer clicked" is used for "customer was interested." "Ticket was closed" is used for "problem was solved." The gap between proxy and intent is a quiet source of error.
• Delayed outcomes. Labels like "defaulted on loan" or "churned" only become known later. Recent records may be labeled negative simply because the outcome has not happened yet.

How to improve labels

1. Write guidelines with examples. Include clear positives, clear negatives, and borderline cases with explanations. Update the document when new edge cases appear.
2. Measure agreement. Have multiple people label the same sample and check how often they agree. Low agreement means the task or the guidelines need work, not that annotators need scolding.
3. Review disagreements. The cases where labelers disagree reveal where definitions are unclear.
4. Audit confident model errors. After a first training run, inspect examples where the model is confident but disagrees with the label. A meaningful share are often labeling mistakes.
5. Allow an "unsure" option where appropriate, so ambiguous cases can be handled separately rather than labeled arbitrarily.
6. Respect outcome windows. Only label records old enough for the outcome to be known.

Data leakage: when the answer sneaks in

Leakage happens when training data contains information that would not be available at the moment of prediction. The model looks excellent in evaluation and fails in production, because in production the leaked information does not exist.

Common forms

• Target leakage. A feature is a consequence of the outcome. Imagine a churn model that uses "account closure reason." That field is only filled in after a customer leaves, so it perfectly predicts churn in historical data and is always empty for current customers.
• Temporal leakage. Features computed with data from after the prediction time, such as a "total purchases" field calculated over all time, including purchases after the date you are trying to predict from.
• Train-test contamination. The same customer, patient, or near-duplicate document appears in both training and test sets. The model recognizes individuals instead of learning patterns.
• Preprocessing leakage. Scaling, imputation, or feature selection is fitted on the full dataset before splitting, so statistics from the test set influence training.

How to catch it

• For every feature, ask: would I know this value at the exact moment I need to make the prediction?
• Build features using an explicit "as of" timestamp and only use data before it.
• Split data by time or by entity before any preprocessing, and fit transformations on the training portion only.
• Treat suspiciously strong features and suspiciously high scores as bugs until proven otherwise.

Missing data, outliers, and duplicates

Missing values

Before filling gaps, understand why they exist. Missingness is often informative. A blank "company size" field might mean the lead came from an individual rather than a business. Replacing it with the average hides that signal.

Practical options:

• Add an indicator feature that records whether the value was missing, alongside any imputed value.
• Use models that handle missing values natively, such as many tree-based methods.
• Drop features that are mostly empty and not informative.
• Investigate sudden changes in the missing rate, which usually signal an upstream pipeline issue.

Outliers

Separate errors from rare but real events. A transaction for an unusually large amount might be a typo, or it might be exactly the kind of case a fraud model needs to learn. Remove values that are impossible; keep values that are unusual but plausible, and consider transformations or robust models to limit their influence.

Duplicates

Exact duplicates are easy to find. Near-duplicates, like the same product with slightly different titles or the same person with two email addresses, need fuzzy matching or similarity search. Remove duplicates before splitting data, or group them so all copies land on the same side of the split.

Representativeness and skew

A dataset can be accurate, complete, and consistent and still lead to a poor model if it does not reflect the population the model will serve.

Sampling bias

Data collected through one channel reflects that channel. Imagine a loan model trained only on applicants who were previously approved. It never saw outcomes for rejected applicants, so it has no evidence about a large part of the population it will score. This is sometimes called selection bias, and it is common whenever past decisions determined which outcomes got recorded.

Underrepresented segments

If one region, language, or customer type makes up a tiny share of the data, the model's performance there may be much worse than the overall number suggests. Check per-segment counts before training, and evaluate per segment afterward.

Training-serving skew

Sometimes the data is fine, but the features are computed differently in training and in production. A batch job computes "average order value" over 90 days for training, while the live service computes it over 30. Or text is lowercased in the training pipeline but not in the live one. The model receives inputs that differ subtly from what it learned on.

The strongest defense is using the same code path to compute features in both places, and logging live feature values so you can compare their distributions with training data.

Drift: quality is not a one-time check

Data changes after deployment.

• Data drift means input distributions shift. New customer segments arrive, a marketing campaign changes traffic sources, or a product redesign changes user behavior.
• Concept drift means the relationship between inputs and outcomes changes. Fraudsters adapt their tactics, or economic conditions change what predicts late payment.
• Upstream changes are the most mundane cause: a renamed field, a new unit of measurement, a default value that changed from blank to zero.

Monitor feature distributions, missing rates, and prediction distributions continuously. When labels arrive, track live performance too. Alerts on sudden shifts catch pipeline breaks early, which are far more common than dramatic changes in the world.

A practical data quality checklist

Use this before training, and automate as much of it as possible for ongoing pipelines.

Before training

1. Document where each field comes from and what it means.
2. Check row counts against expectations for each source and time period.
3. Profile every column: type, range, distinct values, missing rate.
4. Validate formats, allowed categories, and logical relationships between fields, such as end dates after start dates.
5. Find exact and near duplicates.
6. Review label guidelines, measure annotator agreement, and inspect a random sample of labels by hand.
7. Confirm every feature is available at prediction time.
8. Split by time or entity, then fit preprocessing on training data only.
9. Check segment coverage for important groups.

After training

1. Inspect confident errors for label mistakes.
2. Look at the strongest features for signs of leakage.
3. Evaluate performance per segment.

In production

1. Validate incoming data against a schema with explicit expectations.
2. Compare live feature distributions to training distributions.
3. Alert on changes in volume, missing rates, and prediction patterns.
4. Version datasets so any model can be traced to the exact data it learned from.

Making it a habit

Data quality work tends to get skipped because it lacks visible milestones. A few practices help it stick:

• Assign ownership. Each important dataset should have a named team responsible for its definitions and fixes.
• Write data contracts. Agree with upstream teams on schemas, meanings, and change notifications so fields do not silently change.
• Put checks in the pipeline. Automated validation that blocks a training run on bad data is more reliable than a checklist people remember to consult.
• Budget time for it. Plan explicit time for data investigation early in every project, before model experiments begin.

Key takeaways

• Bad data rarely causes visible errors in ML; it silently teaches the model the wrong lessons, so quality has to be checked deliberately.
• Labels define success, so invest in clear guidelines, agreement checks, and audits of confident model errors.
• Leakage produces models that look excellent and fail in production; for every feature, confirm it is known at prediction time.
• Accurate data can still be unrepresentative; check segment coverage, selection effects, and training-serving skew.
• Data and relationships drift after launch, so monitor inputs, predictions, and outcomes continuously.
• Automate validation, version datasets, and assign clear ownership so quality work becomes routine rather than heroic.