The Math You Actually Need for Machine Learning

Every "math for ML" reading list is the same three courses: linear algebra, calculus, probability. Each is a semester. Together they are a year you do not have, and most of what is in them you will never use.

The useful version of the question is narrower. Which ideas keep reappearing in the code, and what does knowing them let you do that you could not do otherwise? That list is short. It is maybe six ideas, and none of them are hard. What makes them worth learning properly is that they turn library calls from incantations into things with reasons.

If you came through an engineering degree you have already met all of this. I did civil engineering, and every piece below was in those courses — matrices for structural analysis, derivatives for everything, probability for load factors. What changed was not the mathematics. It was what it is pointed at.

A matrix is a function, not a grid of numbers

This is the single reframing that makes linear algebra click for ML.

At school a matrix is a table you learn rules for multiplying. In ML a matrix is a linear map: a function that takes a vector in and gives a vector out. Multiplying two matrices is not a grid operation, it is composing two functions. AB means do B, then do A.

Once you see it that way, a neural network layer stops being mysterious:

h = W @ x + b

W maps the input vector into a different space, possibly of a different dimension, and b shifts it. That is the whole layer. A network is a stack of these maps.

Which immediately raises a question worth sitting with. If every layer is a linear map, and composing linear maps gives another linear map, then a hundred stacked layers collapse into a single equivalent matrix. All that depth buys nothing.

That is exactly why the activation function is not optional. ReLU, sigmoid, tanh — their job is to break linearity so composition actually adds expressive power. The activation is not a detail bolted on for convenience; it is the reason depth means anything. You cannot get to that from the grid-of- numbers view, and you get to it immediately from the function view.

The dot product is the workhorse

Two vectors, multiply elementwise, add it up. Trivially simple, and it is doing most of the work in modern ML, because the dot product measures alignment.

similarity = a @ b        # large when a and b point the same way

Every embedding-based system runs on this. "Find similar documents" is a dot product against stored vectors. Attention scores are dot products between queries and keys — how much should this token care about that one — and the answer is how aligned their vectors are.

Here is a detail that shows what the math buys you. Attention is not q @ k, it is:

scores = (q @ k.T) / math.sqrt(d_k)

Why divide by the square root of the dimension? Because if the components of q and k are roughly independent with unit variance, their dot product over d_k terms has variance proportional to d_k. So as the dimension grows, the scores spread out, and softmax over spread-out values saturates — it goes nearly one-hot, gradients vanish, and the model stops learning what to attend to. Dividing by the square root of d_k holds the variance steady.

That is a line of code that looks arbitrary and is not. One paragraph of probability explains it.

Shapes are your debugger

The least glamorous and most useful habit: track the shape of every tensor.

x        # (batch, seq_len, d_model)
W_q      # (d_model, d_head)
q = x @ W_q   # (batch, seq_len, d_head)

A large share of real bugs are shape errors that do not crash — a broadcast that silently did the wrong thing, a transpose that should not have been there, a batch dimension that got summed over. If you can predict the output shape of every line before you run it, you catch these by reading. If you cannot, you are debugging by print statement.

Derivatives: which way is downhill

Training is one idea repeated: measure how wrong you are, work out which direction reduces the wrongness, take a small step that way.

The derivative answers "if I nudge this input slightly, how much does the output change, and in which direction?" The gradient is that answer for every parameter at once, collected into a vector. It points in the direction of steepest increase, so you step the opposite way:

for p in parameters:
    p -= learning_rate * p.grad

That is gradient descent. It is the entire optimisation story before the refinements.

Backpropagation is the chain rule and nothing else

The chain rule says: to differentiate nested functions, multiply the derivatives of each layer of nesting.

Take a network so small you can do it by hand:

z = w * x + b          # linear
a = sigmoid(z)         # activation
L = (a - y) ** 2       # loss

How does the loss change with w? Walk backwards, multiplying:

dL/dw = dL/da * da/dz * dz/dw
      = 2 * (a - y)   *  a * (1 - a)  *  x

Three local derivatives, each obvious on its own, multiplied together. That is backprop. A framework does it for a million parameters instead of one, and caches the intermediate values so nothing is recomputed, but there is no additional idea.

And now the payoff, because this explains a piece of history. Look at the middle term, the derivative of sigmoid: a * (1 - a). Its largest possible value is 0.25, at a = 0.5. In a deep network you multiply one of these per layer. Ten sigmoid layers and the gradient reaching the first layer is scaled by at most 0.25 ** 10, which is about one in a million. The early layers receive essentially nothing and never learn.

That is the vanishing gradient problem, and it is why ReLU — whose derivative is exactly 1 for positive inputs, so it multiplies through without shrinking — was such a large practical unlock. Two lines of chain rule tell you why an architectural choice mattered.

Probability: your model outputs a distribution

The step that reorganises everything: a classifier does not output an answer, it outputs a probability distribution over answers. A language model does not output the next token, it outputs a distribution over the whole vocabulary.

Once you accept that, the loss function stops being a design choice and becomes a derivation.

Maximum likelihood, and where the losses come from

You have data and a model with parameters. Ask: which parameters make the observed data most probable?

Assuming the examples are independent, the probability of the whole dataset is the product of the individual probabilities:

likelihood = p(y_1 | x_1) * p(y_2 | x_2) * ... * p(y_n | x_n)

Products of many small numbers underflow, so take the log, which turns the product into a sum and is monotonic so it does not move the maximum. Flip the sign to make it a minimisation, and you have the negative log-likelihood:

NLL = -sum( log p(y_i | x_i) )

Now specialise it, and watch the familiar losses fall out.

Classification. The model gives a probability per class. The negative log probability of the correct class, summed over the data, is exactly cross-entropy loss. It was not chosen because it works well. It is what maximum likelihood is for a categorical output.

Regression. Assume the target is the prediction plus Gaussian noise with fixed variance. Write down the Gaussian density, take its log, and everything except the squared term is a constant with respect to the parameters. What is left is the squared error. Mean squared error is maximum likelihood under an assumption of Gaussian noise.

This is worth more than trivia. It tells you when MSE is the wrong tool: when your noise is not Gaussian. Heavy-tailed targets with occasional large outliers violate the assumption, the squared term lets one outlier dominate the gradient, and the fix — Huber loss, or predicting a quantile — comes from questioning the assumption rather than from a list of alternative losses.

Information theory: three quantities

Three definitions, tightly related, and they show up constantly.

Entropy is the average surprise of a distribution — how uncertain it is. A fair coin has high entropy; a coin that always lands heads has none.

Cross-entropy is the average surprise you get when you believe distribution Q but reality follows P. It is minimised when Q equals P, which is why it works as a loss: you are penalised for the gap between your predicted distribution and the truth.

KL divergence is the difference between the two:

KL(P || Q) = cross_entropy(P, Q) - entropy(P)

Read it as: how much extra surprise you pay for using Q instead of the true P. It is zero when they match and grows as they diverge.

Two consequences. Since entropy(P) does not depend on your model, minimising cross-entropy and minimising KL divergence are the same optimisation — the standard classification loss is already a KL minimisation, just written without the constant.

And KL is not symmetric. KL(P || Q) is not KL(Q || P), and the asymmetry has teeth. Minimising one direction produces a Q that spreads out to cover everywhere P has mass; the other produces a Q that concentrates on one mode and ignores the rest. Which you pick changes what your model does.

If you read the RLHF post, this is the same KL that appears there as a penalty term. The policy is charged for diverging from the model it started from, and the coefficient on that charge is the exchange rate between reward and drift. Same quantity, used as a leash instead of a loss.

Optimisation: why the simple thing works

Gradient descent has an obvious failure mode. The gradient is local — it only knows the slope right where you stand — so it walks downhill into whatever valley it happens to be in, not the deepest one.

For a convex loss, shaped like a bowl, there is only one valley and this is not a problem. Linear and logistic regression are convex, which is why they train reliably.

Neural networks are wildly non-convex. Enormously many local minima and saddle points. By the textbook, gradient descent should be hopeless.

It works anyway, and the reasons are worth knowing because they justify things that otherwise look like folklore:

In very high dimensions, bad local minima are rare. For a point to be a local minimum, the loss must curve upward in every one of millions of directions. That is an extraordinary coincidence. Far more common are saddle points, which curve up in some directions and down in others — and a saddle has a downhill direction, so you can leave.

The noise helps. Stochastic gradient descent computes the gradient on a small batch, so each step is a noisy estimate of the true gradient. That noise is not just a computational compromise; it lets the optimiser rattle out of narrow valleys. It is a feature.

Most minima are about as good. Empirically, the many minima a large network can land in reach similar loss. You are not searching for one special solution among millions of bad ones.

This is also where the learning rate gets its outsized reputation. Too large and you step past the minimum and oscillate or diverge; too small and training takes forever and settles into the first shallow dip. Nothing else you tune matters as much.

What you can skip, for now

Being honest about the other direction matters as much.

You do not need to compute eigenvalues by hand, invert matrices by hand, or prove convergence theorems. You do not need measure theory. You do not need most of a real analysis course. You do not need to derive backprop for an arbitrary architecture — you need to know it is the chain rule so that when a gradient is NaN you know where to look.

Determinants, eigendecomposition and SVD are genuinely useful, but useful later and in specific places: PCA, some initialisation schemes, analysing what a trained model has learned. Learn them when you hit one, not before.

The failure mode is not skipping too much. It is spending six months on a linear algebra course before writing any code, and arriving with a lot of theorems and no idea which ones matter.

A workable order

Learn it interleaved with building things, not before.

1. Vectors, matrices, shapes. Enough to read a model definition and predict every tensor shape. Do this first; it pays off immediately.
2. Derivatives and the chain rule. Enough to hand-derive the two-layer example above. Then read a from-scratch backprop implementation and follow every line.
3. Probability, up to maximum likelihood. Enough to derive cross-entropy yourself rather than accept it.
4. Entropy, cross-entropy, KL. Small, and they unlock a lot of papers.
5. Everything else, on demand. When a paper uses something you do not know, learn that thing. This is a better filter than any curriculum, because the things you keep meeting are by definition the things that matter.

If you want one source rather than three courses, Mathematics for Machine Learning (Deisenroth, Faisal and Ong) is free online and written for exactly this — the mathematics that ML uses, without the rest of the degree. For intuition rather than mechanics, 3Blue1Brown's Essence of linear algebra and Essence of calculus are the best few hours you can spend on either.

The point

None of this is about being able to prove things. It is about the difference between reading scores / math.sqrt(d_k) as a line someone told you to write and reading it as variance control. Between "cross-entropy is the loss for classification" and "cross-entropy is what maximum likelihood reduces to here, so it stops being right when this assumption stops holding."

That difference is what lets you debug a model instead of restarting the training run with a different random seed and hoping.