Weight initialization

In <a href="/facts/Deep_learning/JLuwD3ea">deep learning</a>, weight initialization or parameter initialization describes the initial step in creating a <a href="/facts/Neural_network_(machine_learning)/6V1jMlkx">neural network</a>. A neural network contains trainable parameters that are modified during training: weight initialization is the pre-training step of assigning initial values to these parameters.
The choice of weight initialization method affects the speed of convergence, the scale of neural <a href="/facts/Activation_function/S4NImL6L">activation</a> within the network, the scale of gradient signals during <a href="/facts/Backpropagation/lCsIdKHc">backpropagation</a>, and the quality of the final model. Proper initialization is necessary for avoiding issues such as <a href="/facts/Vanishing_gradient_problem/3hteovPr">vanishing and exploding gradients</a> and activation function <a href="/facts/Activation_function/S4NImL6L">saturation</a>.
Note that even though this article is titled "weight initialization", both weights and biases are used in a neural network as trainable parameters, so this article describes how both of these are initialized. Similarly, trainable parameters in <a href="/facts/Convolutional_neural_network/kHEdnmGU">convolutional neural networks</a> (CNNs) are called <a href="/facts/Kernel_(image_processing)/mKV4r1Gv">kernels</a> and biases, and this article also describes these.

Weight initialization open-in-new

Weight initialization