preface

Sigmoid function (Logistic function) is a very commonly used activation function in neural networks. Today, we will have an in-depth understanding of Sigmoid function.

Functional form

Function image

Code implementation

Code run: Colab

import matplotlib.pyplot as plt
import numpy as np
import math

x = np.linspace(-10.10.100)
z = 1 / (1 + np.exp(-x))

plt.title("Sigmoid")
plt.plot(x, z)
plt.xlabel("x")
plt.ylabel("Sigmoid(X)")

plt.savefig("sigmoid.png")
plt.show()
plt.close()
Copy the code

Nature and Problems

The range of the function value S(x) is (0, 1), often used in binary problems, the function is smooth, easy to find the derivative. But as the activation function, the large amount of calculation and error back propagation and gradient, derivative division, prone to gradient disappeared, the input is close to infinite or negative infinity, gradient approach to zero, gradient dispersion (with the increase of network layer, using the back propagation algorithm to calculate the gradient, from the output layer to the first few, disappearance of gradient is obvious. As a result, the derivative of the overall loss function to the weight of the first few layers is very small, so when the gradient descent algorithm is used, the weight of the first few layers changes very slowly, and even useful features cannot be learned). Because the Sigmoid function value is greater than 0, weight update can only be updated in one direction, which may affect the convergence speed.

conclusion

Sigmoid function is a very common activation function in neural network, which is widely used in logistic regression and widely used in statistics and machine learning fields.

  • Originally posted on: RAIS