Learning From Mistakes

You now know that every model starts by making predictions, and that we can measure exactly how wrong those predictions are. But measuring error is only half the story. The other half — and the part that makes machine learning remarkable — is what happens next: the model adjusts itself to do better. That adjustment process is called learning, and it is the engine inside every neural network, every decision tree booster, every model you will ever use.

Weights and the Direction of Change

Recall from Lesson 1 that a model's predictions come from weights — numbers that scale the influence of each input feature. When the model makes an error, the question is: which weights caused the error, and which direction should each weight shift to reduce it? Imagine a dial on a stereo. If the music is too quiet, you turn the dial up. If it is too loud, you turn it down. Training a model is like adjusting hundreds or millions of dials simultaneously — each one nudged in the direction that makes the total error go down. The algorithm that figures out which direction to nudge each weight is called gradient descent. 'Gradient' means slope — the algorithm computes how steeply the loss changes as each weight changes. A weight with a large positive gradient is pulling the loss up, so the algorithm decreases it. A weight with a negative gradient is pulling the loss down, so the algorithm increases it slightly.

Here is a concrete analogy. Imagine you are blindfolded on a hilly landscape and your goal is to reach the lowest valley. You cannot see the whole map, but you can feel the ground under your feet — which direction slopes downward. You take a small step downhill. Then feel again. Another step downhill. Repeat. Eventually you reach a low point. Gradient descent works identically, except the 'landscape' is the loss function and the 'position' is the set of all weights. Each step is one update to the weights. The low point — where loss is smallest — is what we call the trained model. The size of each step is controlled by a setting called the learning rate. Too large a step and you overshoot the valley and bounce around. Too small a step and training takes forever. Choosing a good learning rate is one of the tuning challenges you will meet in Lesson 8.

Backpropagation: Spreading the Credit

For simple models with only a few weights, computing gradients is straightforward. For deep neural networks with millions of weights, computing which weight contributed how much to the error is a bigger challenge. The algorithm that solves this is called backpropagation (often shortened to backprop). Backpropagation works backward through the network — from the output (where the error is known) back toward the input — applying the chain rule of calculus layer by layer. Each weight receives a number saying 'you contributed this much to the error, move by this amount.' You do not need to derive calculus here. What matters: backprop is how neural networks assign blame for an error across every single weight, so each one can be adjusted.