Introduction to Computer Vision
30 min readreadingImage Fundamentals and Preprocessing
1 of 30Computer Vision with Deep Learning
Up next · Image Representation and Color Spaces
It is 7:40 in the morning and a radiologist is already behind. Overnight the hospital's scanners produced several hundred chest X-rays, and somewhere in that pile — statistically, almost certainly — there is a tumour the size of a grain of rice. Her eyes are good, but they are human eyes: they tire, they skip, they see what they expect to see. Down the hall, a piece of software reviews the same images in ninety seconds and flags eleven of them for a closer look. It never gets tired, never gets bored, and nobody wrote rules telling it what a tumour looks like — it learned from a million labeled examples.
That software is computer vision: the discipline of taking an array of pixel values and extracting meaning from it — a face, a tumour, a stop sign, a defect on a circuit board. The field went from "researchers can recognize handwritten digits" to "self-driving cars and medical imaging diagnostics" in roughly a decade, driven almost entirely by deep learning. What changed was not the cameras and not the pixels; it was the shift from telling machines what to look for to letting them learn it from examples.
This lesson sets the map for the next 29 — what computer vision is, why it stubbornly resisted decades of clever engineering, what happened in 2012 that broke the dam, and how the decade after — from AlexNet through ResNet to ViT, CLIP, and SAM — carried the field into its foundation-model era, where one giant pretrained model does work that used to take a bespoke system per task. Along the way you will run your first "image": eight pixels by eight, built by hand, so you can see for yourself that a picture is nothing more than a grid of numbers.
1. What Computer Vision Actually Does
Strip away the demos and the products, and every computer-vision system does the same thing: it takes an image (or a stream of them) and produces structured output that some downstream decision can consume. The interesting differences lie in what that output looks like. Five canonical tasks cover almost everything practitioners build:
| Task | Output | Example |
|---|---|---|
| Classification | One label per image | "cat" / "dog" |
| Detection | Boxes + labels | "person at (x, y, w, h)" |
| Segmentation | Pixel-wise label map | "this pixel is road, this is sky" |
| Generation | A new image | "a photo of an astronaut on a horse" |
| Multi-modal | Text + image jointly | "describe this picture" |
The course covers all five. Sections 1-2 build classification; Section 3 adds detection; Section 4 adds segmentation; Section 5 generation; Section 6 the modern multi-modal landscape.
Keep this table in your pocket. When someone describes a vision problem — "we need to know whether the weld is defective", "we need to blur every face in the frame" — your first job is to decide which row it belongs to, because that choice determines the architecture you reach for, the labels you must collect, and the metric you will be judged by. Before any of that, though, it is worth being honest about why this problem is hard at all.
2. Why Vision Is Hard (For Machines)
Humans recognize a cat from any angle, in any lighting, behind any occlusion. A computer sees a 224 × 224 × 3 array of integers from 0 to 255 — about 150 000 numbers, none of which individually says anything about whiskers. The challenge is the gap between those numbers and "cat", and the gap is wider than intuition suggests because so many things change the numbers without changing the answer:
Walk around the cat and every pixel changes with your viewpoint; wait for dusk and illumination rewrites the numbers again. Step back and the animal shrinks from a thousand pixels to thirty. Let it slip behind the sofa so only the tail shows, or photograph it in a cluttered living room where the background supplies most of the image, and the array changes yet again. And even holding all of that fixed, "cat" is not one appearance but a family of them: a Persian, a Maine Coon, and a hairless Sphynx share a label while sharing remarkably few pixels.
Every one of these variations leaves the label unchanged — it is still a cat — while scrambling the raw numbers dramatically. Two photographs of the same animal can disagree on virtually every pixel, while a cat and a dog shot in the same studio pose can agree on most of them. Whatever a vision system learns, it must be a function that is wildly insensitive to some enormous changes and exquisitely sensitive to some subtle ones. That is the entire difficulty of the field, compressed into one sentence.
To feel the gap yourself, run the cell below. It builds an 8 × 8 "image" out of nothing but a numpy array, prints the raw numbers, then displays them as pixels. Press Run and compare the two panels: the left is what you see, the right is the same picture with the number the computer sees written on every pixel. Then edit the values — change the 0.6 to 0.2, or move the bars — and run it again to watch the "image" change.
The plus sign is obvious to you and completely invisible to the machine — it has no concept of "plus sign", only 64 floating-point numbers. Everything in this course is about closing that gap. Pre-deep-learning systems tried to close it with hand-engineered features (SIFT, HOG, SURF). Deep nets learn the right features directly from data — that one shift is what unlocked the modern era.
3. The Pre-Deep-Learning Era (Briefly)
It was not for lack of trying. Before 2012, a state-of-the-art vision system was a carefully engineered pipeline: hand-designed feature extractors such as SIFT, HOG, and SURF turned pixels into descriptors, a classical classifier (an SVM or a random forest) made the decision on top of them, and a layer of domain-specific tricks patched up whatever the first two stages missed for each particular task.
Performance plateaued in the high 70s % on ImageNet classification. Each new task required new features, designed by hand, by experts, over months — and a descriptor that worked beautifully for pedestrian detection told you nothing about reading street signs. The field was research-heavy, brittle, and didn't generalize, and everyone inside it knew. Then came the autumn of 2012, and a result that people who were in the room still describe with something like awe.
4. The 2012 Inflection Point
A ten-point jump, on a benchmark where progress had been measured in fractions of a point, was not an improvement — it was a change of regime. Within five years, ImageNet classification went from "human performance is the goal" to "humans are the bottleneck". By 2017, top models were beating the human baseline. The same architectural ideas powered detection, segmentation, and eventually generation. So what did the neural network know that twenty years of feature engineering didn't?