In artificial intelligence, especially computer vision and artificial neural networks, a hard sigmoid is non-smooth function used in place of a sigmoid function. These retain the basic shape of a sigmoid, rising from 0 to 1, but using simpler functions, especially piecewise linear functions or piecewise constant functions. These are preferred where speed of computation is more important than precision.
Examples
The most extreme examples are the sign function or Heaviside step function, which go from −1 to 1 or 0 to 1 (which to use depends on normalization) at 0.1
Other examples include the Theano library, which provides two approximations: ultra_fast_sigmoid, which is a multi-part piecewise approximation and hard_sigmoid, which is a 3-part piecewise linear approximation (output 0, line with slope 0.2, output 1).23
References
Curves and Surfaces in Computer Vision and Graphics, Volume 1610, SPIE, 1992, p. 301 https://books.google.com/books?id=9fpRAQAAIAAJ&q=%22hard+sigmoid%22 ↩
"nnet – Ops for neural networks". Archived from the original on 2018-08-14. Retrieved 2018-09-03. https://web.archive.org/web/20180814165920/http://deeplearning.net/software/theano/library/tensor/nnet/nnet.html ↩
Theano/sigm.py at 38a6331ae23250338290e886a72daadb33441bc4 · Theano/Theano · GitHub https://github.com/Theano/Theano/blob/38a6331ae23250338290e886a72daadb33441bc4/theano/tensor/nnet/sigm.py#L279 ↩