📘 Lesson · Lesson 58

Activation Functions in Neural Networks

In this lesson: Activation functions in neural networks explained — ReLU, sigmoid, tanh, softmax — graphs, formulas and use cases.

What is an activation function

An activation function decides the output of a neuron in a neural network. It takes the neuron's input and transforms it, adding non-linearity so the network can learn complex patterns. Common ones are Sigmoid, Tanh, ReLU and Softmax — each with a different shape and use.

💡 Activation functions add non-linearity — the key that lets deep networks learn curves, not just straight lines.

Why they are needed

Without an activation function, every layer would just be a linear equation, and stacking many linear layers still gives one linear result. That could never model real, curvy patterns. Activation functions bend the output, so many layers together can learn rich, complex relationships. This is what makes deep learning powerful.

Sigmoid

Squashes any input into a value between 0 and 1.

Range: 0 to 1.
Best for: the output layer of binary (yes/no) problems.
Weakness: can cause the vanishing gradient problem in deep networks.

Tanh

Similar to Sigmoid but outputs between -1 and 1.

Range: -1 to 1 (centred on 0).
Best for: hidden layers where centred output helps.
Weakness: still suffers from vanishing gradients in very deep networks.

ReLU

The most popular choice — outputs the input if positive, else 0.

def relu(x):
    return max(0, x)

print(relu(5))    # 5
print(relu(-3))   # 0

5
0

ReLU is simple and fast, and largely avoids the vanishing gradient problem, so it is the default for hidden layers in most modern networks.

Softmax

Turns a list of numbers into probabilities that add up to 1.

Output: a set of probabilities (each 0 to 1, summing to 1).
Best for: the output layer of multi-class problems (e.g. classifying into 10 categories).
Idea: the biggest input gets the highest probability.

Which to use

Function	Range	Typical use
ReLU	0 to ∞	Hidden layers (default)
Sigmoid	0 to 1	Binary output
Tanh	-1 to 1	Some hidden layers
Softmax	0 to 1 (sum 1)	Multi-class output

Common confusions

Using Sigmoid everywhere. Prefer ReLU in hidden layers; Sigmoid mainly for binary output.
Forgetting non-linearity matters. Without activations, extra layers add nothing.
Softmax vs Sigmoid mix-up. Sigmoid for one yes/no output; Softmax for choosing among many classes.
Ignoring vanishing gradients. Deep Sigmoid/Tanh networks can train very slowly — ReLU helps.

Summary

Activation functions add non-linearity so networks learn complex patterns.
ReLU is the fast, popular default for hidden layers.
Sigmoid (0-1) suits binary output; Tanh (-1 to 1) suits some hidden layers.
Softmax gives class probabilities for multi-class output.
Without activations, deep layers behave like one linear layer.

Next, learn model quality in overfitting and underfitting, or compare architectures in CNN vs RNN.

Frequently Asked Questions

What is an activation function in a neural network?

An activation function decides the output of a neuron based on its input. It adds non-linearity, which lets the network learn complex patterns. Without it, a deep network would behave like a single simple layer.

Why do neural networks need activation functions?

They introduce non-linearity. Without an activation function, stacking layers just gives another linear equation, so the network could not learn curves or complex relationships. Activations make deep learning possible.

What is ReLU and why is it popular?

ReLU (Rectified Linear Unit) outputs the input if it is positive, else 0. It is simple, fast, and avoids the vanishing gradient problem that slows Sigmoid and Tanh, which is why it is the default in most hidden layers.

When do I use Sigmoid or Softmax?

Use Sigmoid in the output layer for binary (yes/no) problems, since it gives a value between 0 and 1. Use Softmax in the output layer for multi-class problems, since it gives probabilities that add up to 1.

What is the vanishing gradient problem?

With Sigmoid and Tanh, gradients can become tiny in deep networks, so early layers learn very slowly. ReLU largely avoids this because its gradient stays 1 for positive inputs, helping deep networks train faster.

← Back to Python Tutorial

💻 Live Code Editor

This page's programs are ready here — run them, edit them, and learn. No installation needed.

Powered by OneCompiler. The code loads into the editor automatically — press Run to see the output. If the editor does not open, open it in a new tab.