Activation Functions

You established in the previous lesson that activation functions are necessary — without them, a deep network collapses into a single linear transformation. But which nonlinearity should you use, and does the choice matter? The answer is yes, substantially. Different activation functions have different behaviors, computational costs, and failure modes. This lesson studies three functions in depth: sigmoid, tanh, and ReLU, and explains why the field moved from the first two to the third.

Sigmoid and Tanh

The sigmoid function is defined as: sigma(z) = 1 / (1 + e^(-z)) It maps any real number to the interval (0, 1). When z is very large and positive, sigma approaches 1. When z is very large and negative, sigma approaches 0. At z = 0, sigma(0) = 0.5. Computed example: sigma(2) = 1 / (1 + e^(-2)) = 1 / (1 + 0.135) ≈ 0.881. The hyperbolic tangent (tanh) is: tanh(z) = (e^z - e^(-z)) / (e^z + e^(-z)) It maps any real number to (-1, 1). At z = 0, tanh(0) = 0. Computed example: tanh(1) = (e - e^(-1)) / (e + e^(-1)) ≈ (2.718 - 0.368) / (2.718 + 0.368) ≈ 2.350 / 3.086 ≈ 0.762. Both functions are smooth (infinitely differentiable) and squeeze their input into a bounded range. Historically they were the default choices. But both suffer from a critical failure mode called the vanishing gradient problem.

ReLU — Rectified Linear Unit — is defined as: ReLU(z) = max(0, z) It is disarmingly simple: if z is positive, output z unchanged; if z is negative or zero, output 0. Computed examples: ReLU(3.5) = 3.5 ReLU(-1.2) = 0 ReLU(0) = 0 The gradient of ReLU is also simple: dReLU/dz = 1 if z > 0, 0 if z <= 0 When z > 0, the gradient is 1 — it does not shrink. This directly solves the vanishing gradient problem for positive activations: gradients flow backward through ReLU neurons without being scaled down. This is the primary reason ReLU became the default activation function for hidden layers in modern deep networks around 2012 and has remained so since.

Choosing and Using Activation Functions

In practice, hidden layers almost always use ReLU or a variant (Leaky ReLU, ELU, GELU). The output layer uses a different function chosen to match the prediction target: - Binary classification (yes/no): sigmoid output, producing a probability in (0,1) - Multi-class classification (one of K classes): softmax, which produces K probabilities summing to 1 - Regression (predicting a real number): often no activation (linear output), so predictions are unbounded Leaky ReLU addresses ReLU's own failure mode — the dying ReLU problem. If a neuron's pre-activation is always negative (because weights drift negative during training), its gradient is always zero and it never updates. Leaky ReLU solves this by using a small slope alpha (e.g., 0.01) for negative inputs: LeakyReLU(z) = z if z > 0, alpha*z if z <= 0