Looking at the Data Before You Model It
45 min readreadingIntroduction to AI and Machine Learning
2 of 40Ground Truth: AI from First Principles
Up next · Measuring a Model, and the Two Linear Ones
A model can only be as good as what it was fed. This lesson covers the part of the work that consumes most of a real project and decides most of its outcome: looking at the data before modelling it, preparing it without poisoning it, and setting up an evaluation protocol whose verdict you can actually trust.
The basic rule of data analysis is that you must not train a model on data you have not examined. EDA (exploratory data analysis) is the systematic inspection of the data before any model is built: sizes and types, how the values are distributed, whether there are missing values and outliers, how the features relate to one another, whether the classes are balanced. Skipping this step is like starting treatment without reading the test results.
The clearest illustration of this rule is Anscombe's quartet: four different datasets in which almost every familiar statistic agrees — the same means of and , the same variances, the same correlation (0.816), even the same regression line. Numerically these datasets are identical. Their pictures are nothing alike.
Set I is a linear relationship. Set II is really a parabola, for which a linear model is unfit. Set III is a straight line distorted by a single outlier. In set IV the entire relationship rests on one point. Four datasets with identical numerical summaries, and completely different structure. Descriptive statistics without visualisation is therefore not enough: start the analysis with a picture, and only then move to the numbers.
The difference between the mean and the median is easiest to show with a neutral example. Add one person with an extraordinarily high income to a group of people who all earn roughly the same, and the mean jumps sharply while nobody else's actual income has changed. The median — the value in the middle of the ordered list — barely moves. For skewed distributions such as incomes, prices, and durations, the median is therefore the more representative summary. Outliers are formally identified by the 1.5·IQR rule, which treats values outside the interval
as suspicious, where are the first and third quartiles. But calling a value suspicious is not the same as removing it. What to do with an outlier depends on the meaning of the problem, not on the formula. A typing error such as "age 250" should be corrected or dropped. A genuine rare event — a fraudulent transaction for a large sum — may be the single most valuable row in the dataset.
The strength of a linear relationship between two features is measured by the Pearson correlation coefficient, which lies between and :
In meaning, describes the direction in which two features move together. A positive value means they rise together, a negative one means they move in opposite directions, and a value near zero indicates no linear relationship. A correlation heat map lets you take in every pairwise relationship at once.
Two typical misreadings follow from this. The first is forgetting that only rules out a linear relationship. In set II of Anscombe's quartet the relationship is perfect but non-linear, and Pearson's coefficient does not see it. The second is more serious: taking correlation for causation. Ice-cream sales and shark attacks correlate almost perfectly over a year, and there is no causal link between them whatsoever. Both are driven by a hidden third factor.
Whenever a strong correlation turns up, it is worth testing the hypothesis of a hidden common cause. Both features may be consequences of a third phenomenon, in which case steering one of them through the other makes no sense at all.
The engineering principle "garbage in, garbage out" has the force of a law in machine learning. The most sophisticated algorithm is helpless against poor data. This is why preparing data takes 60–80 % of the time on a real project: it is the stage where the quality of the result is actually decided.
Real data has gaps in it, caused by unfilled fields, sensor failures, or processing errors. The strategies for handling them differ in sophistication. The simplest is deletion of rows or columns, which is reasonable when there are few gaps or when a column is beyond saving. More widely used is imputation — substituting a value, which may be a constant, the median for numeric features, or the mode, the most frequent value, for categorical ones. A missing value can also be predicted by a separate model. Beyond that, it is effective to add a binary feature recording the fact that the value was missing. The absence itself is often informative: a customer who declined to state their income may behave differently from one who stated it, and the model deserves to be told so.
Models work only with numbers, so categories have to be encoded. For nominal features the standard is one-hot encoding, in which every category gets its own binary column. Encoding categories into a single numeric column — Kyiv = 1, Lviv = 2, Odesa = 3 — is wrong. Under that encoding a linear model takes the numbers literally and learns the false relations "Odesa = 3 Kyivs" and "Lviv is halfway between Kyiv and Odesa". One-hot encoding removes that invented ordering.
| city | → | city_Kyiv | city_Lviv | city_Odesa |
|---|---|---|---|---|
| Lviv | 0 | 1 | 0 | |
| Kyiv | 1 | 0 | 0 |
Consider comparing objects by two features whose ranges differ wildly — height in tens of centimetres and income in tens of thousands. Without a preliminary transformation the distance between objects is decided almost entirely by income, and the contribution of height is lost. Algorithms built on distances or gradients — kNN, SVM, linear models, neural networks — are sensitive to feature scale. Two transformations are in common use:
Standardisation brings a feature to zero mean and unit deviation, which suits data with outliers and a roughly bell-shaped distribution. The min-max transformation maps values into the interval . With no scaling at all, the "income" feature completely dominates "height", and kNN effectively classifies on one feature, producing a vertical boundary.
Data leakage is information from the test data, or from the future, seeping into the training process. Compare it with someone who gets hold of the answer key in advance and then badly misjudges how well prepared they are. The most common form of leakage is computing the mean and deviation for standardisation over the whole sample before splitting it into train and test. The model then uses the test data indirectly. The reliable defence is to express the entire preparation as a pipeline, which is fitted on train alone and only then applied to test.
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
clf = Pipeline([("scaler", StandardScaler()),
("model", LogisticRegression(max_iter=1000))])
clf.fit(X_train, y_train) # scaling statistics come from train ONLY
score = clf.score(X_test, y_test)The goal of any model is to work well on data it did not see during training. It therefore has to be evaluated on new data, otherwise learning the regularities cannot be told apart from memorising the examples. Hence the standard three-way split of the sample.
The test set should be used once. Repeatedly tuning a model against its test results gradually overfits it to that test and destroys the one objective estimate you had. All experiments therefore run against a separate validation set, and test is opened a single time at the end.
When data is scarce, K-fold cross-validation is used. The sample is divided into equal parts and training is repeated times, each time with a different part held out for testing and the rest used for training. Every example then takes part in both training and checking, and instead of one estimate you get of them — enough to judge not only the mean but the spread . A stable model is preferable to one with an accidentally high single result. For classification the class proportions must be preserved in each fold (stratification), and for time series the split is made strictly by time, since future data must never be used to train on the past.