The Objective: Minimizing Loss

Training data constrains the hypothesis space, but we still need a precise, computable criterion for which hypothesis is best. We need to turn 'fits the data well' into a number that a computer can minimize. That number is the loss — and the function that computes it is the loss function. The choice of loss function encodes your values: what kinds of errors you care about and how much.

What Is a Loss Function?

A loss function L(y_hat, y) takes two arguments — the model's prediction y_hat and the true label y — and returns a non-negative number measuring how wrong the prediction is. A perfect prediction yields loss 0; wrong predictions yield positive loss; the worse the prediction, the higher the loss. The total training loss is the average (or sum) of the per-example losses over the entire training dataset: L_total = (1/n) * sum over i of L(f(xi), yi) Training is the process of searching for parameter values that minimize L_total. This is why the loss function is called the objective — it is the thing the algorithm is literally trying to optimize. Two loss functions are foundational and worth knowing precisely. Mean Squared Error (MSE): used for regression (predicting a continuous number). MSE = (1/n) * sum of (y_hat_i - y_i)^2 Example: true prices are [200k, 350k, 500k]; predictions are [210k, 340k, 480k]. Errors: [10k, -10k, -20k]. Squared errors: [100M, 100M, 400M]. MSE = 600M / 3 = 200,000,000 (dollars squared). Root MSE = $14,142. Cross-Entropy Loss: used for classification (predicting a probability over categories). For binary classification: L = −[y * log(y_hat) + (1−y) * log(1−y_hat)] If the true label is 1 (spam) and the model outputs probability 0.9, loss = −log(0.9) ≈ 0.105. Low. If the true label is 1 and the model outputs 0.1, loss = −log(0.1) ≈ 2.303. High — severely penalizes confident wrong predictions.

A subtlety: minimizing training loss does not mean finding the true function. It means finding the hypothesis in your hypothesis space that best explains the training data according to your loss function. If your hypothesis space does not contain the true function, the minimum-loss hypothesis is still an approximation. If your training data is unrepresentative, minimizing training loss can move you away from the true function on real-world inputs. This separation — between the objective (minimize training loss) and the goal (perform well on future data) — is one of the deepest tensions in machine learning. We will examine it closely in Lesson 5. A useful mental model: the loss function is a landscape. Each point in the landscape corresponds to one hypothesis (one set of parameter values). The height of the landscape at any point is the training loss for that hypothesis. Training is the problem of finding the lowest point in the landscape. Lesson 7 will show how gradient descent navigates this landscape.

The Loss Landscape and Multiple Minima

For a linear model with MSE loss, the loss landscape has a pleasing property: it is convex — shaped like a bowl — meaning there is exactly one global minimum, reachable by any competent optimizer. For deep neural networks with millions of parameters, the loss landscape is high-dimensional and complex. It has many local minima (valleys that are not the lowest point globally), saddle points (flat regions that curve upward in some directions), and plateaus (vast flat regions where the gradient gives little information about which direction to move). Navigating this landscape efficiently is the problem of optimization, to be explored in Lesson 7. A remarkable empirical finding: in modern large neural networks, many local minima have nearly the same loss value and generalize comparably well. This suggests the landscape, while complex, may be more navigable than early theory feared. But this is an active area of research — our theoretical understanding of why large neural networks train successfully is still incomplete.