Linear Transformations in Practice

14 min read · ~45 min with practicenotebookLinear Algebra Foundations
3 of 39Mathematics for Machine Learning

You are staring at a photograph that came in sideways. A phone recorded it in portrait, the metadata lied, and now every face in your training set is lying on its side. The fix is a single line of code — rotate(image, 90) — and it works, and you move on. But underneath that convenience is a machine that takes every pixel coordinate , multiplies it by a small grid of numbers, and returns where that pixel should now live. The image did not "turn." A matrix reached into the plane and moved every point at once, by a rule so uniform that straight lines stayed straight and the origin never budged.

That same machine is hiding in far more than image utilities. When Principal Component Analysis "rotates your data onto its axes of greatest variance," it applies a matrix. When a neural network layer computes , it applies a matrix. The dot product and matrix multiplication you built in Lesson 1 (Vectors, Matrices, and Tensors) and Lesson 2 (Matrix Operations and Properties) were the mechanics. This lesson is about the meaning: a matrix is not a table of numbers, it is a function that transforms space — and once you can see the function, you can predict what it will do before you ever run the code.

We will build that intuition from the ground up: why a matrix is literally a function, why its columns tell you everything it does, how stacking transforms is just multiplication, what the famous special cases look like, and how "changing basis" sets up the eigenvectors of Lesson 4 (Eigenvalues and Eigenvectors) and the PCA that follows. Along the way you will run the transforms in NumPy and implement the core operation — matrix times vector — by hand.

1. A Matrix Is a Function

Start with the most useful reframe in all of linear algebra. A function takes an input and returns an output; takes a number and doubles it. A matrix does exactly this, except its inputs and outputs are vectors. Feed it a vector , and it returns a new vector . The function is linear, which is a precise and consequential promise: it respects addition and scaling.

Read those two equations as guarantees about behavior. Whatever does, it does the same thing to a sum as to the parts, and scaling the input just scales the output. A direct corollary: . The origin is nailed in place — no linear transformation can shift it. That is why "rotating an image" about its center needs a little extra bookkeeping (a translation) that a pure matrix cannot express; translations move the origin, so they are affine, not linear.

Because the transformation is linear, it is completely determined by a tiny amount of information — far less than the infinite number of points it moves. Section 2 turns that into a superpower.

2. Basis Vectors Tell You Everything

Any 2D vector is a recipe written in terms of two reference directions: points east, points north, and . Now apply a linear map and let its two rules from Section 1 do the work:

Look at what just happened. To know where any point goes, you only need to know where the two basis vectors go — and — then blend them with the same weights . Here is the punchline that makes matrices legible for the rest of your career: the columns of are exactly the landing spots of the basis vectors.

So the matrix–vector product is just a weighted sum of the columns — a linear combination — where the vector supplies the weights:

Watch this happen. The animation below applies a matrix to the unit square step by step, so you can see the columns become the new axes as the whole grid follows.

loading animation…

As it plays, keep your eye on the two highlighted basis arrows. Every other point in the grid is carried along by the linear-combination rule — the axes lead, and space follows. When the grid ends up sheared or flipped, trace it back to what the columns did.

3. Applying Transforms with NumPy

Enough theory to make the code honest. In NumPy a linear transformation is A @ x, and if you stack many points as the columns of a matrix , then transforms all of them at once — the vectorized win you raced for in Lesson 1's playground. Let us push the unit square's corners through a shear.

python
import numpy as np

# Four corners of the unit square, one point per column.
square = np.array([[0, 1, 1, 0],
                   [0, 0, 1, 1]], dtype=float)

# A horizontal shear: e1 stays put, e2 lands at (0.5, 1).
A = np.array([[1.0, 0.5],
              [0.0, 1.0]])

transformed = A @ square
print("columns of A (where the axes land):")
print(A[:, 0], A[:, 1])
print("original corners:\n", square)
print("sheared corners:\n", transformed)
code
columns of A (where the axes land):
[1. 0.] [0.5 1. ]
original corners:
 [[0. 1. 1. 0.]
 [0. 0. 1. 1.]]
sheared corners:
 [[0.  1.  1.5 0.5]
 [0.  0.  1.  1. ]]

The bottom edge (both points at ) did not move, and the top edge slid right by — precisely because the second column, the landing spot of , moved to . The determinant is , so area is preserved: a shear slants the square into a parallelogram of the same area, the geometric meaning of the determinant from Lesson 2. Change the to and rerun — the top edge slides twice as far, area unchanged.

4. Composition Is Matrix Multiplication

Real pipelines never apply just one transform — you rotate, then scale, then project. The beautiful fact is that applying and then — the composition — is itself a linear transformation, with matrix the product :

This is why matrix multiplication uses that strange row-times-column rule from Lesson 2 rather than elementwise: it is engineered so the product matrix is the composed function. It also explains the property that trips up every beginner — order matters, because in general. Function composition is not commutative: socks-then-shoes is not shoes-then-socks.

python
import numpy as np

theta = np.pi / 2                      # 90 degrees
R = np.array([[np.cos(theta), -np.sin(theta)],
              [np.sin(theta),  np.cos(theta)]])
S = np.array([[3.0, 0.0],              # stretch x by 3, y unchanged
              [0.0, 1.0]])

v = np.array([1.0, 0.0])
rotate_then_scale = S @ R              # apply R first, then S
scale_then_rotate = R @ S             # apply S first, then R

print("rotate then scale:", np.round(rotate_then_scale @ v, 3))
print("scale then rotate:", np.round(scale_then_rotate @ v, 3))
print("matrices equal?  ", np.allclose(rotate_then_scale, scale_then_rotate))
code
rotate then scale: [0. 1.]
scale then rotate: [0. 3.]
matrices equal?   False

Rotating by 90° gives ; a horizontal stretch then leaves it alone, so you land at . Reverse the order and the stretch acts first, turning into , which the rotation swings up to . Same ingredients, different result — and the matrices themselves disagree. Read composition right-to-left: in , the matrix nearest the vector acts first. That convention is exactly how a neural network's forward pass chains its layers, a thread we pick up in Lesson 32 (Math Behind Neural Networks).

5. The Special Transforms You Will Meet Everywhere

A handful of matrices show up again and again, and knowing their shapes on sight is worth more than any product grinding by hand.

TransformMatrixWhat it doesDeterminant
Rotation by Turns space; preserves lengths & angles
ScalingStretches each axis independently
ShearSlants; preserves area
Reflection (over )Flips; reverses orientation
Projection onto Flattens onto a line; loses a dimension

Two of these anchor later lessons. A rotation is orthogonal: its columns are unit vectors at right angles, so it preserves every length and angle and its inverse is its transpose, . That property is the quiet hero of Lesson 5 (Singular Value Decomposition), where every matrix factors into rotate–scale–rotate.

A projection is the opposite: it throws information away. Projecting onto a line spanned by unit vector is the outer product

Read it right: is the dot product from Lesson 1 — how much of lies along — and rebuilds a vector of that length pointing along . Projections are idempotent: , because once you have flattened onto the line, flattening again does nothing. That fact is the heart of least-squares regression in Lesson 31, where fitting a line is projecting your target onto the span of your features.

python
import numpy as np

u = np.array([1.0, 1.0])
u = u / np.linalg.norm(u)          # unit vector along the 45-degree line
P = np.outer(u, u)                 # projection matrix u u^T

x = np.array([3.0, 1.0])
px = P @ x
print("projection matrix:\n", np.round(P, 3))
print("P @ x     =", np.round(px, 3))
print("P @ (P@x) =", np.round(P @ px, 3), "  (idempotent: same as above)")
print("residual x - Px is orthogonal to u:", np.round(np.dot(x - px, u), 12))
code
projection matrix:
 [[0.5 0.5]
 [0.5 0.5]]
P @ x     = [2. 2.]
P @ (P@x) = [2. 2.]   (idempotent: same as above)
residual x - Px is orthogonal to u: 0.0

The residual — what is left after projecting — is perfectly perpendicular to the line. That orthogonality is the defining property of the "best" approximation, and the geometric reason least squares works at all.

6. Change of Basis: A Preview of PCA

Here is the idea that turns everything above into machine learning. Coordinates are not sacred — they are a description relative to a chosen set of axes, and a different basis gives the same point different numbers. Suppose your new axes are the columns of a matrix . A point written in the new basis as has world coordinates , so to express a world point in the new basis you invert:

And a transformation that you understand in world coordinates becomes, in the new basis, the conjugated matrix

Why would anyone want this? Because the right basis makes a hard matrix simple. Choose so its columns are the invariant directions of — its eigenvectors — and comes out diagonal: the transformation reduces to independent stretches along each new axis, no cross-talk. That is the entire content of the eigendecomposition you will derive in Lesson 4.

PCA is this move applied to data instead of a transform. Your features are correlated — height and weight lean together — so the cloud of points sits at an angle in the original axes. PCA finds the basis in which the covariance matrix becomes diagonal: new axes (the principal components) along which the data varies independently. Rotate onto that basis with (the components are orthonormal, so ) and correlations vanish. Below is the setup — the eigenvectors wait for Lesson 4, but the change of basis is exactly this section's machinery.

python
import numpy as np

# A tilted, correlated cloud: x and y rise together.
rng = np.random.default_rng(0)
base = rng.normal(size=(2, 400))
tilt = np.array([[2.0, 1.0],
                 [1.0, 1.0]])
data = tilt @ base                       # correlated 2D points, columns

# New basis B = the two orthonormal directions of greatest variance.
cov = np.cov(data)
values, B = np.linalg.eigh(cov)          # columns of B are the axes
rotated = B.T @ data                     # change of basis: express in B

print("covariance in original axes:\n", np.round(cov, 2))
print("covariance after change of basis:\n", np.round(np.cov(rotated), 2))
code
covariance in original axes:
 [[5.33 3.26]
 [3.26 2.18]]
covariance after change of basis:
 [[ 0.14 -0.  ]
 [-0.    7.37]]

The off-diagonal covariance — the correlation — dropped to zero. Same points, better axes. You did not change the data; you changed the description, and in the eigenvector basis the features are decorrelated. That is the through-line from here to Lesson 4 (Eigenvalues and Eigenvectors) and Lesson 5 (Singular Value Decomposition), and to the whitening and dimensionality reduction you will lean on across the rest of the course.

7. Build It Yourself: Matrix Times Vector

Everything in this lesson stands on one operation — the matrix–vector product from Section 2. Before you let NumPy do it forever, implement it once with plain Python lists. Remember the two readings you now hold: the -th output entry is the dot product of row with , and equivalently the whole output is a weighted sum of the columns. Either view leads to correct code.

Exercise · gradedImplement matvec(A, x)

Implement matvec(A, x), the matrix–vector product, using only plain Python — no NumPy. A is a list of rows (each row a list of numbers) and x is a list of numbers whose length equals the number of columns of A. Return a new list whose i-th entry is the dot product of row i of A with x. For an matrix and a length- vector, the result has length .

def matvec(A, x):
    # A: list of rows (list of lists); x: list of numbers.
    # Return a list: entry i is the dot product of row i with x.
    ...

Notice the last test: passing to that matrix returns times the first column plus times the second. If your code passes it, you have proven that the matrix–vector product really is a linear combination of columns — the claim Section 2 asked you to take on faith.

8. What You Can Now Read Off a Matrix

Step back and take stock of the new sense you have grown. A matrix arrives, and you no longer see a grid of numbers to push through a formula — you see a function acting on space. Its columns are where the axes land, so you can sketch its effect before computing anything. Its determinant tells you whether it grows or shrinks area and whether it flips orientation; zero warns that it has collapsed a dimension. Multiplying two matrices composes their functions, right to left, and the order is not negotiable. And the right basis turns a tangled transform into clean, independent stretches.

A matrix is a verb, not a noun. It does not sit there being a table of numbers; it acts, and its columns confess exactly how.

This is the lens the whole linear-algebra arc is built around. In Lesson 4 you will hunt for the vectors a matrix does not rotate — its eigenvectors; Lesson 5 factors any matrix into rotate–scale–rotate. And far downstream, when a neural network computes in Lesson 32 and gradients flow backward through it in Lesson 33 (Deriving Backpropagation by Hand), it is these same transformations, stacked dozens deep, learning to reshape data until the answer falls out.

Up next · Eigenvalues and Eigenvectors