Activation Functions

Activation Functions

Activation functions are a critical part of neural networks. They determine whether a neuron should be activated based on the input it receives.

Without activation functions, the neural network would behave as a linear model, limiting its ability to learn complex patterns.

Common types of activation functions include:

• Sigmoid: Outputs values between 0 and 1.
• ReLU (Rectified Linear Unit): Outputs the input if it's positive, otherwise 0.
• Softmax: Converts outputs into probabilities for classification tasks.

The sigmoid function maps any input value to a range between 0 and 1, making it suitable for binary classification.

Formula:

σ(x) = 1 / (1 + e^(-x))

It squashes large inputs and outputs values close to 0 or 1.

The ReLU (Rectified Linear Unit) function is defined as:

ReLU(x) = max(0, x)

It outputs 0 for negative inputs and the input value itself for positive inputs. ReLU is computationally efficient and helps address the vanishing gradient problem.

The softmax function is used in the output layer for multi-class classification. It converts raw scores into probabilities:

Softmax(x_i) = e^(x_i) / Σ(e^(x_j))

All output values are normalized to sum up to 1.

The choice of activation function depends on the task:

• Sigmoid: For binary classification.
• ReLU: For hidden layers in deep networks.
• Softmax: For multi-class classification problems.

Activation functions enable neural networks to:

• Learn complex, non-linear patterns.
• Differentiate between inputs effectively.
• Adapt to various types of data and tasks.

Some issues include:

• Sigmoid: Can cause vanishing gradients during training.
• ReLU: Can lead to 'dead neurons' where some neurons stop learning.
• Softmax: Computationally expensive for large outputs.

In this lesson, we:

• Explored activation functions like Sigmoid, ReLU, and Softmax.
• Learned their roles and use cases in neural networks.
• Discussed their advantages and challenges.

Next, we will build a simple feedforward neural network and understand its structure and working.