Batch normalization

Batch normalization (also known as batch norm) is a technique used to make training of <a href="/facts/Artificial_neural_network/6V1jMlkx">artificial neural networks</a> faster and more stable by adjusting the inputs to each layer—re-centering them around zero and re-scaling them to a standard size. It was introduced by Sergey Ioffe and Christian Szegedy in 2015.
Experts still debate why batch normalization works so well. It was initially thought to tackle internal covariate shift, a problem where parameter initialization and changes in the distribution of the inputs of each layer affect the learning rate of the network. However, newer research suggests it doesn’t fix this shift but instead smooths the <a href="/facts/Intelligent_agent/8C2Q93Vw">objective function</a>—a mathematical guide the network follows to improve—enhancing performance. In very deep networks, batch normalization can initially cause a severe <a href="/facts/Vanishing_gradient_problem/3hteovPr">gradient explosion</a>—where updates to the network grow uncontrollably large—but this is managed with shortcuts called skip connections in residual networks. Another theory is that batch normalization adjusts data by handling its size and path separately, speeding up training.

Batch normalization open-in-new

Batch normalization