Learning the Rules Instead of Writing Them
40 min readreadingIntroduction to AI and Machine Learning
1 of 40Ground Truth: AI from First Principles
Up next · Looking at the Data Before You Model It
This chapter is the foundation the whole course rests on. In one dense sitting it covers material that classical textbooks spread across three separate chapters: what machine learning actually is and why engineering practice has moved away from writing rules by hand; how to look at data without drawing false conclusions; the full arsenal of classical algorithms, from linear regression to gradient boosting; and how to measure their quality honestly. Nearly every formula comes with a picture or a plain analogy meant to explain the idea — the algebra you can always derive yourself afterwards. This first lesson takes the opening four sections: what learning from data means, the three paradigms, the two main problem types, and the notation used from here to the end of the course.
It helps to start with a concrete example. Suppose you have to write a program that tells a picture of a cat from a picture of a dog. The classical approach is to state the rules explicitly: "if the ears are triangular and the whiskers are long, it is a cat." In practice that leads to thousands of lines of rules with separate branches for the awkward cases, and the program still makes crude mistakes whenever the viewing conditions get difficult. The reason is not that the developer is unskilled. It is that some relationships simply cannot be written out by a human at all. You recognise a cat in a fraction of a second, and you cannot describe in words how you did it.
This is where the approach reverses. Machine learning (ML) turns the usual scheme around. In classical programming you supply the rules, add data, and get answers. In machine learning you hand the computer the data together with the correct answers, and it derives the rules itself. Compare a cook who writes a recipe with a cook who reconstructs a recipe from the taste of the finished dish. You no longer dictate the method to the machine; instead you give it the result and let it recover the method that produces it.
The classical engineering definition was stated by Tom Mitchell in 1997:
A program learns from experience with respect to a class of tasks and a performance measure if its performance at tasks in , as measured by , improves with experience .
Each of the three letters names a separate strand of engineering work. Take a spam filter. The task is labelling messages as spam or not-spam. The experience is thousands of messages sorted by hand beforehand. The measure defines what counts as success, and choosing it is the hardest part. It could be the fraction of correct decisions, or the amount of spam that gets through, or the number of important messages wrongly filed as spam. Which model is judged good depends on the choice of , which is why a large part of this chapter is given over to choosing it well. In machine learning, a badly chosen quality measure distorts every measurement made with it.
The terms artificial intelligence, machine learning and deep learning are not synonyms; they nest inside one another. Artificial intelligence is the widest of the three and denotes intelligent machine behaviour in general. Machine learning is its main working instrument today — learning from data. Deep learning is the subset of ML built on multilayer neural networks, taken up from chapter 2 onward. This chapter covers the classical methods that everything after it is built on.
Before the algorithms themselves, it is worth sketching the shape of the field. Machine learning divides into three broad paradigms according to the kind of data the model is given and how much of the right answer comes with it.
In supervised learning every example carries a label, and the model learns a mapping from input to correct output. This is by far the most used paradigm — roughly 80–90 % of industrial ML applications — and it is what this chapter concentrates on. In unsupervised learning the labels are absent and the model looks for structure in the data itself: natural clusters, compressed representations, anomalies. In reinforcement learning an agent acts in an environment and receives only a reward signal describing how good the action was. Compare it with learning to ride a bicycle: nobody gives you the exact parameters of the movement, and you arrive at them through a sequence of attempts, failures carrying negative reward, and successes carrying positive reward. This is the approach used to learn Go and to control robots, and chapter 7 is devoted to it.
| Paradigm | What is given | Typical problems | Example |
|---|---|---|---|
| Supervised | (object, label) pairs | classification, regression | spam filter, price forecast |
| Unsupervised | objects only | clustering, dimensionality reduction, anomalies | customer segmentation |
| Reinforcement | an environment and a reward | sequential decisions | games, robot control |
Within supervised learning the type of problem is set by the nature of the quantity being predicted. If the target variable is discrete and names membership of a class, the problem is classification: you are looking for a function that assigns a label to every object. If the target variable is continuous — a number on a scale — the problem is regression, that is . With no labels at all, when you only group similar objects together, the problem is clustering, and belongs to unsupervised learning.
The type of problem is easiest to read off the form of the question. A question about a quantity — a cost, a duration, a weight — is regression. A question about which class an object belongs to is classification. A question about splitting a set of objects into groups is clustering. The same phenomenon can raise both kinds of question: what will tomorrow's temperature be is a regression question, while whether it will rain is a classification one.
In the overwhelming majority of cases the input to an ML model is a table. The rows are objects — examples or observations, which might be particular patients, apartments, or messages. The columns are features: the measured characteristics of each object. Formally, a sample of objects with features is written as the matrix
where the row is one object and is the vector of correct answers. This notation is used throughout the course. In the chapter on neural networks the expression will mean pushing every object-row through a set of weights.
What matters is that features come in different types, and the types must not be confused. Numeric features such as area and age are ordinary numbers, fit for arithmetic. Nominal categorical features such as city or colour are labels with no order relation, so Kyiv is not "greater than" Lviv. Ordinal features such as size S/M/L or a grade of poor/good/excellent do have a definite order, but the distances between neighbouring values are unknown. Beyond these there are binary features, and dates and times. The type of a feature determines how it is prepared, and getting the type wrong quietly distorts the results. The classic case of miscoded cities appears later in this chapter.