Vectors, Matrices, and Tensors
1 of 39Mathematics for Machine Learning
It is late, the demo is tomorrow, and your semantic search feature has started
returning nonsense: a query about "refund policy" surfaces a paragraph about
parking validation. The embedding model is fine — everyone uses it — so you
drop into a notebook to score the similarity by hand. The traceback that greets
you is not about meaning at all:
ValueError: shapes (768,) and (1,768) not aligned. Somewhere
between the query encoder and the document store a vector picked up an extra
axis, and NumPy has been broadcasting your dot product into a 768×768 matrix
of garbage instead of the single number you wanted.
That single number is a dot product, the quiet workhorse underneath nearly every model you will ship. Similarity search, attention scores, the linear layer inside a neural network, the loss you minimize during training: all of them are dot products, sums, and reshapes stacked into towers. The bug that wrecked your evening was not a machine-learning bug. It was a shape bug — a mismatch between the mathematical object you imagined and the array NumPy actually held.
This lesson builds the vocabulary that keeps those two in sync: vectors as points and directions, the dot product and cosine similarity that read off alignment, the norms that measure "size" (and quietly power regularization), matrices as data and transformations, and tensors with the broadcasting rules that turn a shape mismatch from a 2 a.m. mystery into a glance. The dot product you sharpen here becomes matrix multiplication in Lesson 2 (Matrix Operations and Properties) and a geometric transformation in Lesson 3 (Linear Transformations in Practice).
1. Vectors: Points and Directions at Once
A vector is an ordered list of numbers, written as a column:
The notation says " is a real vector with components" — and yet a vector wears two hats you must switch between at will. As a point, is a location in -dimensional space, the tip of an arrow anchored at the origin. As a direction, the same numbers describe a displacement: how far and which way to travel. A word embedding is a point (where does "refund" live?); the difference between two embeddings is a direction (which way is "more formal"?). The math never changes; only your reading of it does.
In practice a vector is a one-dimensional NumPy array
(u = np.array([2.0, 1.0, 3.0])), and the two operations you build
everything from are elementwise: adding two vectors (u + v,
tip-to-tail travel) and scaling one by a number (2.5 * u, same
direction, longer arrow). Its u.shape is (3,) — one
axis, length 3.
2. The Dot Product: Alignment as a Single Number
Multiply two same-length vectors componentwise and add up the results. That sum is the dot product:
It eats two vectors and returns one scalar, collapsing all dimensions into a single number measuring how much the two agree. It deserves the spotlight because of a second, geometric formula for the exact same quantity:
Here is the length of (next section) and is the angle between the arrows. The two formulas are equal — one arithmetic you can run on an array, one geometry you can picture — and together they say: the dot product is large and positive when the vectors point the same way, zero when perpendicular, negative when they oppose. Alignment, as a number.
One consequence you will lean on constantly: two nonzero vectors are orthogonal (perpendicular, ) exactly when their dot product is zero, since . Orthogonality is the backbone of the eigenvectors of Lesson 4 (Eigenvalues and Eigenvectors) and the projections behind least-squares regression in Lesson 31 (Math Behind Linear Regression).
import numpy as np
u = np.array([2.0, 1.0, 3.0])
v = np.array([1.0, 4.0, 0.0])
print("dot, explicit =", np.sum(u * v)) # elementwise then sum
print("dot, idiomatic =", u @ v) # the @ operator: prefer this
print("dot, function =", np.dot(u, v)) # identical result
w = np.array([1.0, -2.0, 0.0])
print("u . w =", u @ w, "-> orthogonal" if u @ w == 0 else "")
dot, explicit = 6.0
dot, idiomatic = 6.0
dot, function = 6.0
u . w = 0.0 -> orthogonal
3. Cosine Similarity: Why Embeddings Only Care About Angle
Rearrange the geometric formula to isolate the angle and you get the most important quantity in modern retrieval, the cosine similarity:
Dividing by both lengths cancels magnitude entirely, leaving a number in that depends only on direction: aligned, orthogonal, opposite. This is why a vector database compares embeddings by cosine rather than raw dot product. A long document and a short query can produce embeddings of wildly different length for reasons unrelated to meaning; cosine discards that nuisance and keeps only the semantic direction. Your "refund policy" query matches the refund paragraph because their embeddings point almost the same way — cosine near — while the parking paragraph sits off at a wide angle.
The dot product also has a clean geometric child: the projection of onto , the shadow casts along :
The figure below shows , , their tip-to-tail sum, the angle , and the projection of onto . As you swing toward , the angle shrinks, the cosine climbs toward one, and the projection stretches to nearly the full length of — the same three quantities moving together.
import numpy as np
def cosine(a, b):
return (a @ b) / (np.linalg.norm(a) * np.linalg.norm(b))
query = np.array([0.9, 0.1, 0.4]) # "refund policy" (toy embedding)
refund = np.array([0.8, 0.2, 0.5]) # a refund paragraph
parking = np.array([0.1, 0.9, 0.2]) # a parking paragraph
print(f"query vs refund : {cosine(query, refund):.3f}")
print(f"query vs parking : {cosine(query, parking):.3f}")
query vs refund : 0.985
query vs parking : 0.283
4. Norms: Three Honest Ways to Measure Size
We keep writing for "length," but there is more than one sensible way to measure the size of a vector, and ML uses several. A norm assigns a nonnegative size, is zero only for the zero vector, scales linearly, and obeys the triangle inequality. The three you meet daily are the -norms for :
The L2 norm is ordinary Euclidean length — Pythagoras in dimensions — the one hiding in the cosine denominator above. The L1 norm is "taxicab" distance, the total absolute travel if you may only move along the axes. The L∞ norm is the largest component. They agree on which vector is zero and disagree, usefully, on everything else.
| Norm | Formula | Nickname | Where it shows up in ML |
|---|---|---|---|
| L2 | Euclidean | Distances, cosine, ridge (weight decay) | |
| L1 | Manhattan / taxicab | Lasso — drives weights to exactly zero | |
| L∞ | Chebyshev / max | Worst-case bounds, gradient clipping |
The difference between L1 and L2 is not academic; it is why two common regularizers behave so differently. Penalizing the L2 norm of a weight vector (ridge) shrinks every weight smoothly toward zero; penalizing the L1 norm (lasso) drives many weights exactly to zero, producing a sparse model. You will derive both as constrained optimization in Lesson 28 (Lagrange Multipliers and Constraints) and see ridge fall out of the regression math in Lesson 31 — but the reason they differ is entirely the geometry of these two norms.
import numpy as np
v = np.array([3.0, -4.0, 0.0])
print("L2 :", np.linalg.norm(v)) # sqrt(9 + 16) = 5
print("L1 :", np.linalg.norm(v, ord=1)) # 3 + 4 = 7
print("Linf:", np.linalg.norm(v, ord=np.inf)) # max(3, 4) = 4
unit = v / np.linalg.norm(v) # normalize: the prep for cosine search
print("unit:", np.linalg.norm(unit))
L2 : 5.0
L1 : 7.0
Linf: 4.0
unit: 1.0
A norm is a promise about what "big" means — choose the one that matches the cost you care about.
5. Matrices: Data Tables and Transformations
Stack vectors side by side and you have a matrix, a grid of numbers with two axes. An matrix has rows and columns, with the entry in row , column :
Like a vector, a matrix wears two hats. As data, each of the
rows is one example (a customer, an image, a document) and each of the
columns one feature — tabular ML is just a design matrix of
shape (n_samples, n_features). As a
transformation, the same grid is a machine that takes a vector
in and produces one out, via the matrix–vector product:
Row of the output is the dot product of row of with — a matrix–vector product is just a stack of dot products, the operation from Section 2 run once per row. That idea seeds everything downstream: a linear layer is ; rotating an image is a matrix times a coordinate vector; the transformations of Lesson 3 and the decompositions of Lesson 4 all start here.
import numpy as np
# The SAME grid as a TRANSFORM: rotate a 2-D vector 90 degrees counterclockwise
theta = np.pi / 2
R = np.array([[np.cos(theta), -np.sin(theta)],
[np.sin(theta), np.cos(theta)]])
print("R @ [1, 0] :", np.round(R @ np.array([1.0, 0.0]), 6)) # now points along +y
R @ [1, 0] : [0. 1.]
6. Tensors: Stacking Into More Dimensions
A scalar has zero axes, a vector one, a matrix two — and the pattern does not stop. A tensor is simply an array with any number of axes: "a NumPy array, possibly with more than two dimensions." Its number of axes is its rank (or ndim); its shape is the tuple of lengths along each axis.
| Object | Rank | Example shape | Typical meaning |
|---|---|---|---|
| Vector | 1 | (768,) | One embedding |
| Matrix | 2 | (32, 768) | A batch of 32 embeddings |
| 3-tensor | 3 | (32, 128, 768) | Batch × sequence × features |
| 4-tensor | 4 | (32, 3, 224, 224) | Batch × channels × height × width |
The rules generalize cleanly: a batch of images is a rank-4 tensor, the hidden states in a Transformer are rank-3 (batch, position, feature). Nothing new happens conceptually — you are still adding and multiplying numbers — but the bookkeeping of which axis is which becomes the whole game, and it is where broadcasting enters.
7. Broadcasting: The Rule That Saves and Betrays You
You will constantly combine arrays of different shapes — add a bias vector to every row of a matrix, subtract a per-feature mean from a whole batch. Broadcasting is NumPy's rule for stretching a smaller array across a larger one without copying data. Line the two shapes up from the right and compare axis by axis: two axes are compatible when equal, or when one is 1 (that axis stretches). Missing leading axes count as 1.
So (3, 4) and (4,) broadcast (the vector applies to
each row); (3, 4) with (3, 1) broadcasts the other
way (one value per row). But (3, 4) and (3,) do
not, because aligning from the right pairs 4 with 3 and neither is 1.
That failure is exactly the opening bug: a (768,) where the code
assumed a matching row, or a stray (1, 768) that silently
stretched into an outer product.
import numpy as np
X = np.array([[1.0, 2.0, 3.0, 4.0],
[5.0, 6.0, 7.0, 8.0],
[9.0, 10., 11., 12.]]) # shape (3, 4)
bias = np.array([100., 200., 300., 400.]) # shape (4,) -> per column
print("X + bias:\n", X + bias) # bias added to every row
col = np.array([[1.], [2.], [3.]]) # shape (3, 1) -> per row
print("\nX * col:\n", X * col) # each row scaled by its factor
X + bias:
[[101. 202. 303. 404.]
[105. 206. 307. 408.]
[109. 210. 311. 412.]]
X * col:
[[ 1. 2. 3. 4.]
[10. 12. 14. 16.]
[27. 30. 33. 36.]]
8. Shapes in Practice: Vectorize or Perish
One last habit separates code that runs from code that runs in time to matter.
Everything here can be written as an explicit Python loop — and runs orders of
magnitude faster as a single vectorized array operation, because the loop then
happens in optimized C over contiguous memory rather than the interpreter. The
dot product of two million-element vectors is a loop you can write, but
a @ b beats it by a factor of dozens, and the gap widens with
size.
Rather than take that on faith, run it yourself. The playground below races a hand-written Python loop against the vectorized NumPy equivalent, live in your browser. Push the array size up and watch the loop's bar grow while NumPy's barely moves — the reason every formula in this course is written to be expressible as array math.
Same-length arrays in, one scalar out, no stray axes to broadcast: that discipline is the throughline of the course. Dot products power the attention scores and embeddings you will keep meeting; norms return as the regularizers of Lesson 28 (Lagrange Multipliers and Constraints); matrices as transforms drive Lesson 3 (Linear Transformations in Practice) and the eigen-analysis of Lesson 4 (Eigenvalues and Eigenvectors); and the gradients you descend in Lesson 9 (Gradient Descent from Scratch) are themselves vectors, measured with these very norms.