Residual neural network

A residual neural network (also referred to as a residual network or ResNet) is a <a href="/facts/Deep_learning/JLuwD3ea">deep learning</a> architecture in which the layers learn residual functions with reference to the layer inputs. It was developed in 2015 for <a href="/facts/Image_recognition/Tl2Yyk66">image recognition</a>, and won the <a href="/facts/ImageNet/DPaK4GLC">ImageNet</a> Large Scale Visual Recognition Challenge (<a href="https://image-net.org/challenges/LSVRC/">ILSVRC</a>) of that year.
As a point of terminology, "residual connection" refers to the specific architectural motif of 
 
 
 
 x
 ↦
 f
 (
 x
 )
 +
 x
 
 
 {\displaystyle x\mapsto f(x)+x}
 
, where 
 
 
 
 f
 
 
 {\displaystyle f}
 
 is an arbitrary neural network module. The motif had been used previously (see §History for details). However, the publication of ResNet made it widely popular for <a href="/facts/Feedforward_neural_network/CP0pPGDF">feedforward networks</a>, appearing in neural networks that are seemingly unrelated to ResNet.
The residual connection stabilizes the training and convergence of deep neural networks with hundreds of layers, and is a common motif in deep neural networks, such as <a href="/facts/Transformer_(deep_learning_architecture)/cDbjx6a8">transformer</a> models (e.g., <a href="/facts/BERT_(language_model)/rDSueM4E">BERT</a>, and <a href="/facts/Generative_pre-trained_transformer/myah4Tjj">GPT</a> models such as <a href="/facts/ChatGPT/ptB1y6J5">ChatGPT</a>), the <a href="/facts/AlphaGo_Zero/oUm1jEdz">AlphaGo Zero</a> system, the <a href="/facts/AlphaStar_(software)/Gphq3yDA">AlphaStar</a> system, and the <a href="/facts/AlphaFold/DwBhppKC">AlphaFold</a> system.

Residual neural network open-in-new

Residual neural network