Matrix Operations and Properties
2 of 39Mathematics for Machine Learning
A recommender system is misbehaving. The pipeline is supposed to take a batch of
user vectors, rotate them into a "taste" space, then score them against an item
catalog — three matrices multiplied in a row. It runs without error. It returns
numbers. And the numbers are garbage: every user gets recommended the same three
items. You stare at the code and everything looks fine, until a colleague leans
over and points at one line. You wrote scores = catalog @ taste @ users.
The math you meant was users @ taste @ catalog. Same three matrices,
same data, reversed order — and the result went from "personalized ranking" to
"meaningless projection onto the wrong subspace."
The bug is not a typo you can grep for. It is a fact about matrices that trips up everyone at least once: matrix multiplication does not commute. and are usually different matrices, and often one of them does not even exist. In the dot products of Lesson 1 (Vectors, Matrices, and Tensors), order did not matter — . The moment you stack those dot products into a matrix product, order becomes the whole story.
This lesson makes that story precise. You will see why matrix multiplication is really function composition in disguise — which instantly explains why order matters and why the shapes have to line up. Then you will work through the operations and properties that every ML algorithm quietly leans on: transpose, the identity and inverse, the determinant as a volume-scaling factor, and rank — the single number that decides whether a system of equations has an answer at all, and whether your linear regression in Lesson 31 (Math Behind Linear Regression) will even run.
1. Matrix Multiplication Is Composition
Forget, for a moment, the "row times column" recipe you may have memorized. The reason matrix multiplication is defined the strange way it is comes from a single idea: a matrix is a function that transforms vectors, and multiplying two matrices means doing one transformation, then the other. Lesson 3 (Linear Transformations in Practice) develops this at length; here you need just enough to understand the algebra.
Say rotates vectors and stretches them. Apply to a vector , then apply to the result: you get . The matrix product is defined to be the single matrix that does both in one shot, so that
Read strictly right to left: acts first, acts second. That reading is the source of every "why is it backwards?" moment in linear algebra, and it is exactly the bug in the opening — reversing the matrices reverses which transformation happens first.
Composition also tells you immediately why order matters. Rotating a vector 90° and then stretching the -axis is a genuinely different operation from stretching first and then rotating — the stretch lands on different directions. Concretely:
Different matrices, as promised. There are special pairs that do commute — a matrix with itself, or a matrix with the identity — but they are the exception, not the rule.
2. Shapes and the Inner-Dimension Rule
Because means "apply , then ," the output of has to be a legal input to . That single requirement is the whole shape rule. If is and is , the product is . The two inner dimensions — the that expects as input and the that produces as output — must match and then cancel; the outer dimensions survive as the shape of the result:
Entry by entry, the element of is the dot product of row of with column of — which is why the shared dimension has to be the length of both:
This is the exact rule that catches the shape-mismatch bug from Lesson 1. When
NumPy raises ValueError: matmul: ... size 3 is different from 4, it is
telling you the inner dimensions did not agree. Trace the shapes of the opening's
three-matrix chain and the reordering bug shows up as a shape error or —
worse — as a chain that happens to be conformable in both orders and silently
computes the wrong thing.
import numpy as np
users = np.random.randn(32, 16) # 32 users, 16 raw features
taste = np.random.randn(16, 16) # feature -> taste-space rotation
catalog = np.random.randn(16, 8) # 16 taste dims, 8 catalog items
# Intended: (32x16)(16x16)(16x8) -> 32 users scored over 8 items
scores = users @ taste @ catalog
print("intended:", scores.shape)
# The bug from the opening: same matrices, reversed.
try:
wrong = catalog @ taste @ users
print("reversed:", wrong.shape)
except ValueError as e:
print("reversed failed:", str(e)[:45])
intended: (32, 8)
reversed failed: matmul: Input operand 1 has a mismatch in its
Here the reversal is caught for free, because the shapes refuse to line up. That
is the lucky case. When your matrices are square — as taste is — a
reversed product is still conformable and NumPy will happily return a
wrong-but-well-shaped answer. Shape checks catch some order bugs; understanding
composition catches the rest.
3. Transpose: Flipping Rows and Columns
The transpose reflects a matrix across its main diagonal: row becomes column , so . An matrix becomes . It looks like bookkeeping, but transpose is the connective tissue of nearly every ML formula, because it is how you turn "a stack of examples" into "a stack of features" and back.
The one property to internalize is how transpose interacts with a product — it reverses the order, just like putting on and taking off socks and shoes:
That reversal is not a quirk; it is forced by the shape rule. If is and is , then is and is . The only way to build a matrix from the transposed pieces () and () is — the inner 's meet in the middle. Any other order does not even conform.
Transpose is also where the dot product from Lesson 1 reappears in matrix clothing. For column vectors and , the dot product is — a product that collapses to a single number. Two patterns you will see constantly: (an matrix of feature-feature dot products, the heart of the normal equations in Lesson 31) and (an matrix of example-example similarities, the kernel matrix). A matrix that equals its own transpose, , is called symmetric; both and always are, which is exactly the structure that makes the eigen-decomposition of Lesson 4 (Eigenvalues and Eigenvectors) so well-behaved.
4. Identity, Inverse, and When There Isn't One
Among all matrices, one does nothing: the identity , with ones on the diagonal and zeros elsewhere. It is the "1" of matrix multiplication — for every compatible . Geometrically it is the transformation that leaves every vector exactly where it was.
Once you have a "1," you can ask about "division." The inverse of a square matrix is the matrix that undoes it:
In the composition picture, is the transformation that reverses whatever did — if rotates 30° and doubles length, halves and rotates . This is what lets you solve \( A\mathbf{x} = \mathbf{b} \) in closed form as : apply the undo-transformation to both sides. The inverse also reverses order under products, for the same socks-and-shoes reason as the transpose: .
But — and this is the crucial part — not every matrix has an inverse. A matrix with no inverse is called singular. Singularity is not a numerical accident; it means the transformation destroyed information that no later transformation can recover. If collapses the whole plane onto a single line, then countless different input vectors landed on the same output point, and there is no function that can send that point back to all of them. You cannot un-flatten a pancake.
5. The Determinant: How Much Volume Does It Scale?
Every square matrix carries a single number that tells you whether it is invertible and, more evocatively, how it changes area or volume. That number is the determinant . Take the unit square with corners at the origin and the two basis vectors; apply ; it becomes a parallelogram. The determinant is the signed area of that parallelogram. For a matrix the formula is short:
A determinant of 2 means the transformation doubles every area; a determinant of shrinks areas by half; a negative determinant means the transformation also flips orientation (a reflection). In three dimensions "area" becomes "volume," and in dimensions it is the -dimensional volume scaling factor. This is not a metaphor — it is why the change-of-variables formula in probability (which returns in the density transformations of Lesson 15) carries a Jacobian determinant.
That collapse is the whole point. Push the sliders until the parallelogram flattens onto a line: its area goes to zero, so . And a zero determinant is precisely the signature of a singular matrix. It makes intuitive sense — if the transformation squashed a 2-D region down to a 1-D line, it obliterated an entire dimension's worth of information, so it cannot be undone. This gives you the single most useful fact about determinants:
is invertible if and only if . Zero determinant means singular, means information destroyed, means no inverse.
In practice a determinant is rarely exactly zero in floating point; it creeps toward zero. A determinant that is tiny relative to the entries of the matrix signals a nearly singular — "ill-conditioned" — matrix, where the inverse exists on paper but amplifies noise catastrophically. This is the numerical face of multicollinearity, and it is why Lesson 31 adds a ridge penalty: to nudge a nearly-singular back to solidly invertible.
6. Rank: How Many Directions Survive
The determinant gives a yes/no verdict on invertibility, but it does not tell you how badly a matrix is degenerate, and it only applies to square matrices. The finer, more general tool is rank: the number of dimensions the matrix's output actually spans. Equivalently, it is the number of linearly independent columns (which always equals the number of linearly independent rows). Rank counts the directions that survive the transformation.
A matrix has full rank 3 if its columns point in three genuinely independent directions — it fills all of 3-D space and is invertible. If one column is a combination of the other two (say, feature C is just "A plus twice B" in your dataset), the columns span only a plane: rank 2, a flat pancake, singular. If everything collapses to a line, rank 1. The rank deficit — how far rank falls below full — is the number of dimensions the matrix annihilated.
This is not abstract. Rank is what decides whether your data problems are solvable:
| Where it shows up | What rank tells you |
|---|---|
| Design matrix in regression | Full column rank invertible a unique least-squares solution (Lesson 31). Rank-deficient means redundant features and infinitely many solutions. |
| Covariance matrix | Its rank is the true dimensionality of your data — the basis of PCA (Lessons 4 and 5). |
| Low-rank approximation | Keeping only the top few directions compresses images and embeddings — the engine of SVD in Lesson 5 (Singular Value Decomposition). |
| Weight matrices in LLMs | LoRA fine-tunes giant models by adding a deliberately low-rank update — a rank-8 correction to a huge weight matrix. |
NumPy ties all of this section together. Build a matrix whose third column is a copy of the first two combined, and watch rank, determinant, and invertibility agree that it is degenerate:
import numpy as np
# Column 3 = column 1 + 2 * column 2 -> a linear dependency
A = np.array([[1.0, 0.0, 1.0],
[0.0, 1.0, 2.0],
[2.0, 1.0, 4.0]])
print("rank :", np.linalg.matrix_rank(A), "of 3")
print("determinant :", round(np.linalg.det(A), 10))
try:
np.linalg.inv(A)
print("inverse : exists")
except np.linalg.LinAlgError:
print("inverse : singular, no inverse")
rank : 2 of 3
determinant : 0.0
inverse : singular, no inverseRank 2, determinant 0, no inverse — three ways of saying the same thing: one of the three directions was fictional. Break the dependency by changing a single entry (say, the corner from 4.0 to 4.1) and all three verdicts flip at once — rank jumps to 3, the determinant becomes nonzero, and the inverse springs into existence. These properties are not independent facts to memorize; they are one fact — does this transformation preserve every dimension? — viewed from three angles.