• Images
• Videos
• Documents
• Books
• Articles

Advantages

Advantages of ReLU include:

• Sparse activation: for example, in a randomly initialized network, only about 50% of hidden units are activated (i.e. have a non-zero output).
• Better gradient propagation: fewer vanishing gradient problems compared to sigmoidal activation functions that saturate in both directions.¹⁷
• Efficiency: only requires comparison and addition.
• Scale-invariant (homogeneous):

max ( 0 , a x ) = a max ( 0 , x )  for  a ≥ 0 {\displaystyle \max(0,ax)=a\max(0,x){\text{ for }}a\geq 0} .

Potential problems

Possible downsides can include:

• Non-differentiability at zero (however, it is differentiable anywhere else, and the value of the derivative at zero can be chosen to be 0 or 1 arbitrarily).
• Not zero-centered: ReLU outputs are always non-negative. This can make it harder for the network to learn during backpropagation, because gradient updates tend to push weights in one direction (positive or negative). Batch normalization can help address this.
• ReLU is unbounded.
• Dying ReLU: ReLU neurons can sometimes be pushed into states in which they become inactive for essentially all inputs. In this state, no gradients flow backward through the neuron, and so the neuron becomes stuck in a perpetually inactive state (it "dies"). This is a form of the vanishing gradient problem. In some cases, large numbers of neurons in a network can become stuck in dead states, effectively decreasing the model capacity and potentially even halting the learning process. This problem typically arises when the learning rate is set too high. It may be mitigated by using "leaky" ReLU instead, where a small positive slope is assigned for x < 0 {\displaystyle x<0} . However, depending on the task, performance may be reduced.

Variants

Piecewise-linear variants

Leaky ReLU allows a small, positive gradient when the unit is inactive,¹⁸ helping to mitigate the vanishing gradient problem. This gradient is defined by a parameter α {\displaystyle \alpha } , typically set to 0.01–0.3.¹⁹²⁰

f ( x ) = { x x > 0 , α x x ≤ 0 , f ′ ( x ) = { 1 x > 0 , α x ≤ 0. {\displaystyle f(x)={\begin{cases}x&x>0,\\\alpha x&x\leq 0,\end{cases}}\qquad f'(x)={\begin{cases}1&x>0,\\\alpha &x\leq 0.\end{cases}}}

The same function can also be expressed without the piecewise notation as:

f ( x ) = 1 + α 2 x + 1 − α 2 | x | {\displaystyle f(x)={\frac {1+\alpha }{2}}x+{\frac {1-\alpha }{2}}|x|}

Parametric ReLU (PReLU) takes this idea further by making α {\displaystyle \alpha } a learnable parameter along with the other network parameters.²¹

Note that for α ≤ 1 {\displaystyle \alpha \leq 1} , this is equivalent to

f ( x ) = max ( x , α x ) {\displaystyle f(x)=\max(x,\alpha x)}

and thus has a relation to "maxout" networks.²²

Concatenated ReLU (CReLU) preserves positive and negative phase information:²³

f ( x ) = [ ReLU ⁡ ( x ) , ReLU ⁡ ( − x ) ] . {\displaystyle f(x)=[\operatorname {ReLU} (x),\operatorname {ReLU} (-x)].}

Other non-linear variants

DELU

ExtendeD Exponential Linear Unit (DELU) is an activation function which is smoother within the neighborhood of zero and sharper for bigger values, allowing better allocation of neurons in the learning process for higher performance. Thanks to its unique design, it has been shown that DELU may obtain higher classification accuracy than ReLU and ELU.²⁴

f ( x ) = { x x > x c , ( e a x − 1 ) / b x ≤ x c f ′ ( x ) = { 1 x > x c , ( a / b ) e a x x ≤ x c {\displaystyle f(x)={\begin{cases}x&x>x_{c},\\(e^{ax}-1)/b&x\leq x_{c}\end{cases}}\qquad f'(x)={\begin{cases}1&x>x_{c},\\(a/b)e^{ax}&x\leq x_{c}\end{cases}}}

In these formulas, a {\displaystyle a} , b {\displaystyle b} and x c {\displaystyle x_{c}} are hyperparameter values which could be set as default constraints a = 1 {\displaystyle a=1} , b = 2 {\displaystyle b=2} and x c = 1.25643 {\displaystyle x_{c}=1.25643} , as done in the original work.

Gaussian-error linear unit (GELU)

GELU is a smooth approximation to the rectifier:

f ( x ) = x Φ ( x ) , {\displaystyle f(x)=x\Phi (x),} f ′ ( x ) = x Φ ′ ( x ) + Φ ( x ) {\displaystyle f'(x)=x\Phi '(x)+\Phi (x)}

where Φ ( x ) = P ( X ⩽ x ) {\displaystyle \Phi (x)=P(X\leqslant x)} is the cumulative distribution function of the standard normal distribution.

This activation function is illustrated in the figure at the start of this article. It has a "bump" to the left of x < 0 and serves as the default activation for models such as BERT.²⁵

SiLU

Main article: Swish function

The SiLU (sigmoid linear unit) or swish function²⁶ is another smooth approximation which uses the sigmoid function, first introduced in the GELU paper:²⁷

f ( x ) = x ⋅ sigmoid ⁡ ( x ) , {\displaystyle f(x)=x\cdot \operatorname {sigmoid} (x),} f ′ ( x ) = x ⋅ sigmoid ′ ⁡ ( x ) + sigmoid ⁡ ( x ) {\displaystyle f'(x)=x\cdot \operatorname {sigmoid} '(x)+\operatorname {sigmoid} (x)}

Softplus

Main article: Softplus

A smooth approximation to the rectifier is the analytic function

f ( x ) = ln ⁡ ( 1 + e x ) , f ′ ( x ) = e x 1 + e x = 1 1 + e − x {\displaystyle f(x)=\ln(1+e^{x}),\qquad f'(x)={\frac {e^{x}}{1+e^{x}}}={\frac {1}{1+e^{-x}}}}

which is called the softplus²⁸²⁹ or SmoothReLU function.³⁰ For large negative x {\displaystyle x} it is roughly ln ⁡ 1 {\displaystyle \ln 1} , so just above 0, while for large positive x {\displaystyle x} it is roughly ln ⁡ ( e x ) {\displaystyle \ln(e^{x})} , so just above x {\displaystyle x} .

This function can be approximated as:

ln ⁡ ( 1 + e x ) ≈ { ln ⁡ 2 , x = 0 , x 1 − e − x / ln ⁡ 2 , x ≠ 0 {\displaystyle \ln \left(1+e^{x}\right)\approx {\begin{cases}\ln 2,&x=0,\\[6pt]{\frac {x}{1-e^{-x/\ln 2}}},&x\neq 0\end{cases}}}

By making the change of variables x = y ln ⁡ ( 2 ) {\displaystyle x=y\ln(2)} , this is equivalent to

log 2 ⁡ ( 1 + 2 y ) ≈ { 1 , y = 0 , y 1 − e − y , y ≠ 0 {\displaystyle \log _{2}(1+2^{y})\approx {\begin{cases}1,&y=0,\\[6pt]{\frac {y}{1-e^{-y}}},&y\neq 0\end{cases}}}

A sharpness parameter k {\displaystyle k} may be included:

f ( x ) = ln ⁡ ( 1 + e k x ) k , f ′ ( x ) = e k x 1 + e k x = 1 1 + e − k x {\displaystyle f(x)={\frac {\ln(1+e^{kx})}{k}},\qquad f'(x)={\frac {e^{kx}}{1+e^{kx}}}={\frac {1}{1+e^{-kx}}}}

The derivative of softplus is the logistic function.

The logistic sigmoid function is a smooth approximation of the derivative of the rectifier, the Heaviside step function.

The multivariable generalization of single-variable softplus is the LogSumExp with the first argument set to zero:

L S E 0 + ⁡ ( x 1 , … , x n ) := LSE ⁡ ( 0 , x 1 , … , x n ) = ln ⁡ ( 1 + e x 1 + ⋯ + e x n ) {\displaystyle \operatorname {LSE_{0}} ^{+}(x_{1},\dots ,x_{n}):=\operatorname {LSE} (0,x_{1},\dots ,x_{n})=\ln(1+e^{x_{1}}+\cdots +e^{x_{n}})}

The LogSumExp function is

LSE ⁡ ( x 1 , … , x n ) = ln ⁡ ( e x 1 + ⋯ + e x n ) {\displaystyle \operatorname {LSE} (x_{1},\dots ,x_{n})=\ln(e^{x_{1}}+\cdots +e^{x_{n}})}

and its gradient is the softmax; the softmax with the first argument set to zero is the multivariable generalization of the logistic function. Both LogSumExp and softmax are used in machine learning.

ELU

Exponential linear units try to make the mean activations closer to zero, which speeds up learning. It has been shown that ELUs can obtain higher classification accuracy than ReLUs.³¹

f ( x ) = { x x > 0 , α ( e x − 1 ) x ≤ 0 f ′ ( x ) = { 1 x > 0 , α e x x ≤ 0 {\displaystyle f(x)={\begin{cases}x&x>0,\\\alpha \left(e^{x}-1\right)&x\leq 0\end{cases}}\qquad f'(x)={\begin{cases}1&x>0,\\\alpha e^{x}&x\leq 0\end{cases}}}

In these formulas, α {\displaystyle \alpha } is a hyperparameter to be tuned with the constraint α ≥ 0 {\displaystyle \alpha \geq 0} .

Given the same interpretation of α {\displaystyle \alpha } , ELU can be viewed as a smoothed version of a shifted ReLU (SReLU), which has the form f ( x ) = max ( − α , x ) {\displaystyle f(x)=\max(-\alpha ,x)} .

Mish

The mish function can also be used as a smooth approximation of the rectifier.³² It is defined as

f ( x ) = x tanh ⁡ ( softplus ⁡ ( x ) ) , {\displaystyle f(x)=x\tanh {\big (}\operatorname {softplus} (x){\big )},}

where tanh ⁡ ( x ) {\displaystyle \tanh(x)} is the hyperbolic tangent, and softplus ⁡ ( x ) {\displaystyle \operatorname {softplus} (x)} is the softplus function.

Mish is non-monotonic and self-gated.³³ It was inspired by Swish, itself a variant of ReLU.³⁴

Squareplus

Squareplus³⁵ is the function

f ( x ) = x + x 2 + b 2 {\displaystyle f(x)={\frac {x+{\sqrt {x^{2}+b}}}{2}}}

where b ≥ 0 {\displaystyle b\geq 0} is a hyperparameter that determines the "size" of the curved region near x = 0 {\displaystyle x=0} . (For example, letting b = 0 {\displaystyle b=0} yields ReLU, and letting b = 4 {\displaystyle b=4} yields the metallic mean function.) Squareplus shares many properties with softplus: It is monotonic, strictly positive, approaches 0 as x → − ∞ {\displaystyle x\to -\infty } , approaches the identity as x → + ∞ {\displaystyle x\to +\infty } , and is C ∞ {\displaystyle C^{\infty }} smooth. However, squareplus can be computed using only algebraic functions, making it well-suited for settings where computational resources or instruction sets are limited. Additionally, squareplus requires no special consideration to ensure numerical stability when x {\displaystyle x} is large.

See also

• Softmax function
• Sigmoid function
• Tobit model
• Layer (deep learning)

References

1. Brownlee, Jason (8 January 2019). "A Gentle Introduction to the Rectified Linear Unit (ReLU)". Machine Learning Mastery. Retrieved 8 April 2021. https://machinelearningmastery.com/rectified-linear-activation-function-for-deep-learning-neural-networks/ ↩

2. Liu, Danqing (30 November 2017). "A Practical Guide to ReLU". Medium. Retrieved 8 April 2021. https://medium.com/@danqing/a-practical-guide-to-relu-b83ca804f1f7 ↩

3. Ramachandran, Prajit; Barret, Zoph; Quoc, V. Le (October 16, 2017). "Searching for Activation Functions". arXiv:1710.05941 [cs.NE]. /wiki/ArXiv_(identifier) ↩

4. Xavier Glorot; Antoine Bordes; Yoshua Bengio (2011). Deep sparse rectifier neural networks (PDF). AISTATS. Rectifier and softplus activation functions. The second one is a smooth version of the first. /wiki/Yoshua_Bengio ↩

5. László Tóth (2013). Phone Recognition with Deep Sparse Rectifier Neural Networks (PDF). ICASSP. http://www.inf.u-szeged.hu/~tothl/pubs/ICASSP2013.pdf ↩

6. Andrew L. Maas, Awni Y. Hannun, Andrew Y. Ng (2014). Rectifier Nonlinearities Improve Neural Network Acoustic Models. https://ai.stanford.edu/~amaas/papers/relu_hybrid_icml2013_final.pdf ↩

7. Hansel, D.; van Vreeswijk, C. (2002). "How noise contributes to contrast invariance of orientation tuning in cat visual cortex". J. Neurosci. 22 (12): 5118–5128. doi:10.1523/JNEUROSCI.22-12-05118.2002. PMC 6757721. PMID 12077207. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6757721 ↩

8. Kadmon, Jonathan; Sompolinsky, Haim (2015-11-19). "Transition to Chaos in Random Neuronal Networks". Physical Review X. 5 (4): 041030. arXiv:1508.06486. Bibcode:2015PhRvX...5d1030K. doi:10.1103/PhysRevX.5.041030. S2CID 7813832. /wiki/ArXiv_(identifier) ↩

9. Engelken, Rainer; Wolf, Fred; Abbott, L. F. (2020-06-03). "Lyapunov spectra of chaotic recurrent neural networks". arXiv:2006.02427 [nlin.CD]. /wiki/ArXiv_(identifier) ↩

10. Householder, Alston S. (June 1941). "A theory of steady-state activity in nerve-fiber networks: I. Definitions and preliminary lemmas". The Bulletin of Mathematical Biophysics. 3 (2): 63–69. doi:10.1007/BF02478220. ISSN 0007-4985. http://link.springer.com/10.1007/BF02478220 ↩

11. Fukushima, K. (1969). "Visual feature extraction by a multilayered network of analog threshold elements". IEEE Transactions on Systems Science and Cybernetics. 5 (4): 322–333. doi:10.1109/TSSC.1969.300225. /wiki/Doi_(identifier) ↩

12. Fukushima, K.; Miyake, S. (1982). "Neocognitron: A Self-Organizing Neural Network Model for a Mechanism of Visual Pattern Recognition". Competition and Cooperation in Neural Nets. Lecture Notes in Biomathematics. Vol. 45. Springer. pp. 267–285. doi:10.1007/978-3-642-46466-9_18. ISBN 978-3-540-11574-8. {{cite book}}: |journal= ignored (help) 978-3-540-11574-8 ↩

13. Hahnloser, R.; Sarpeshkar, R.; Mahowald, M. A.; Douglas, R. J.; Seung, H. S. (2000). "Digital selection and analogue amplification coexist in a cortex-inspired silicon circuit". Nature. 405 (6789): 947–951. Bibcode:2000Natur.405..947H. doi:10.1038/35016072. PMID 10879535. S2CID 4399014. /wiki/Nature_(journal) ↩

14. Hahnloser, R.; Seung, H. S. (2001). Permitted and Forbidden Sets in Symmetric Threshold-Linear Networks. NIPS 2001. ↩

15. Xavier Glorot; Antoine Bordes; Yoshua Bengio (2011). Deep sparse rectifier neural networks (PDF). AISTATS. Rectifier and softplus activation functions. The second one is a smooth version of the first. /wiki/Yoshua_Bengio ↩

16. Yann LeCun; Leon Bottou; Genevieve B. Orr; Klaus-Robert Müller (1998). "Efficient BackProp" (PDF). In G. Orr; K. Müller (eds.). Neural Networks: Tricks of the Trade. Springer. /wiki/Yann_LeCun ↩

17. Xavier Glorot; Antoine Bordes; Yoshua Bengio (2011). Deep sparse rectifier neural networks (PDF). AISTATS. Rectifier and softplus activation functions. The second one is a smooth version of the first. /wiki/Yoshua_Bengio ↩

18. Andrew L. Maas, Awni Y. Hannun, Andrew Y. Ng (2014). Rectifier Nonlinearities Improve Neural Network Acoustic Models. https://ai.stanford.edu/~amaas/papers/relu_hybrid_icml2013_final.pdf ↩

19. "PyTorch Leaky ReLU docs". https://pytorch.org/docs/stable/generated/torch.nn.LeakyReLU.html ↩

20. "TensorFlow Leaky ReLU docs". https://www.tensorflow.org/api_docs/python/tf/keras/layers/LeakyReLU ↩

21. He, Kaiming; Zhang, Xiangyu; Ren, Shaoqing; Sun, Jian (2015). "Delving Deep into Rectifiers: Surpassing Human-Level Performance on Image Net Classification". arXiv:1502.01852 [cs.CV]. /wiki/ArXiv_(identifier) ↩

22. He, Kaiming; Zhang, Xiangyu; Ren, Shaoqing; Sun, Jian (2015). "Delving Deep into Rectifiers: Surpassing Human-Level Performance on Image Net Classification". arXiv:1502.01852 [cs.CV]. /wiki/ArXiv_(identifier) ↩

23. Shang, Wenling; Sohn, Kihyuk; Almeida, Diogo; Lee, Honglak (2016-06-11). "Understanding and Improving Convolutional Neural Networks via Concatenated Rectified Linear Units". Proceedings of the 33rd International Conference on Machine Learning. PMLR: 2217–2225. arXiv:1603.05201. https://proceedings.mlr.press/v48/shang16.html ↩

24. Çatalbaş, Burak; Morgül, Ömer (16 August 2023). "Deep learning with ExtendeD Exponential Linear Unit (DELU)". Neural Computing and Applications. 35 (30): 22705–22724. Retrieved 20 April 2025. https://link.springer.com/article/10.1007/s00521-023-08932-z ↩

25. Hendrycks, Dan; Gimpel, Kevin (2016). "Gaussian Error Linear Units (GELUs)". arXiv:1606.08415 [cs.LG]. /wiki/ArXiv_(identifier) ↩

26. Diganta Misra (23 Aug 2019), Mish: A Self Regularized Non-Monotonic Activation Function (PDF), arXiv:1908.08681v1, retrieved 26 March 2022. https://www.bmvc2020-conference.com/assets/papers/0928.pdf ↩

27. Hendrycks, Dan; Gimpel, Kevin (2016). "Gaussian Error Linear Units (GELUs)". arXiv:1606.08415 [cs.LG]. /wiki/ArXiv_(identifier) ↩

28. Dugas, Charles; Bengio, Yoshua; Bélisle, François; Nadeau, Claude; Garcia, René (2000-01-01). "Incorporating second-order functional knowledge for better option pricing" (PDF). Proceedings of the 13th International Conference on Neural Information Processing Systems (NIPS'00). MIT Press: 451–457. Since the sigmoid h has a positive first derivative, its primitive, which we call softplus, is convex. http://papers.nips.cc/paper/1920-incorporating-second-order-functional-knowledge-for-better-option-pricing.pdf ↩

29. Xavier Glorot; Antoine Bordes; Yoshua Bengio (2011). Deep sparse rectifier neural networks (PDF). AISTATS. Rectifier and softplus activation functions. The second one is a smooth version of the first. /wiki/Yoshua_Bengio ↩

30. "Smooth Rectifier Linear Unit (SmoothReLU) Forward Layer". Developer Guide for Intel Data Analytics Acceleration Library. 2017. Retrieved 2018-12-04. https://software.intel.com/sites/products/documentation/doclib/daal/daal-user-and-reference-guides/daal_prog_guide/GUID-FAC73B9B-A597-4F7D-A5C4-46707E4A92A0.htm ↩

31. Clevert, Djork-Arné; Unterthiner, Thomas; Hochreiter, Sepp (2015). "Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)". arXiv:1511.07289 [cs.LG]. /wiki/ArXiv_(identifier) ↩

32. Diganta Misra (23 Aug 2019), Mish: A Self Regularized Non-Monotonic Activation Function (PDF), arXiv:1908.08681v1, retrieved 26 March 2022. https://www.bmvc2020-conference.com/assets/papers/0928.pdf ↩

33. Shaw, Sweta (2020-05-10). "Activation Functions Compared with Experiments". W&B. Retrieved 2022-07-11. https://wandb.ai/shweta/Activation%20Functions/reports/Activation-Functions-Compared-with-Experiments--VmlldzoxMDQwOTQ ↩

34. Shaw, Sweta (2020-05-10). "Activation Functions Compared with Experiments". W&B. Retrieved 2022-07-11. https://wandb.ai/shweta/Activation%20Functions/reports/Activation-Functions-Compared-with-Experiments--VmlldzoxMDQwOTQ ↩

35. Barron, Jonathan T. (22 December 2021). "Squareplus: A Softplus-Like Algebraic Rectifier". arXiv:2112.11687 [cs.NE]. /wiki/ArXiv_(identifier) ↩