Learning rate

In <a href="/facts/Machine_learning/e0w0XJTu">machine learning</a> and <a href="/facts/Statistics/DNPsKGYU">statistics</a>, the learning rate is a <a href="/facts/Hyperparameter_(machine_learning)/jCxm3fTA">tuning parameter</a> in an <a href="/facts/Mathematical_optimization/oRn8Iv5I">optimization algorithm</a> that determines the step size at each iteration while moving toward a minimum of a <a href="/facts/Loss_function/xv5ozuhl">loss function</a>. Since it influences to what extent newly acquired information overrides old information, it metaphorically represents the speed at which a machine learning model "learns". In the <a href="/facts/Adaptive_control/9CUsn1wv">adaptive control</a> literature, the learning rate is commonly referred to as gain.
In setting a learning rate, there is a trade-off between the rate of convergence and <a href="/facts/Overshoot_(signal)/l450BHuI">overshooting</a>. While the <a href="/facts/Descent_direction/pj86tpeJ">descent direction</a> is usually determined from the <a href="/facts/Gradient_descent/pFFrek0F">gradient</a> of the loss function, the learning rate determines how big a step is taken in that direction. A too high learning rate will make the learning jump over minima but a too low learning rate will either take too long to converge or get stuck in an undesirable local minimum.
In order to achieve faster convergence, prevent oscillations and getting stuck in undesirable local minima the learning rate is often varied during training either in accordance to a learning rate schedule or by using an adaptive learning rate. The learning rate and its adjustments may also differ per parameter, in which case it is a <a href="/facts/Diagonal_matrix/tjIAHrE6">diagonal matrix</a> that can be interpreted as an approximation to the <a href="/facts/Invertible_matrix/fPqXk3V8">inverse</a> of the <a href="/facts/Hessian_matrix/RguMNr3m">Hessian matrix</a> in <a href="/facts/Newton%27s_method_in_optimization/H46TN721">Newton's method</a>. The learning rate is related to the step length determined by inexact <a href="/facts/Line_search/GQWhNEsR">line search</a> in <a href="/facts/Quasi-Newton_method/1YX1vMRa">quasi-Newton methods</a> and related optimization algorithms.

Learning rate open-in-new

Learning rate