Menu
Home
People
Places
Arts
History
Plants & Animals
Science
Life & Culture
Technology
Reference.org
Mixed-precision arithmetic
open-in-new
<h2 id="overview">Overview</h2> <p>A common usage of mixed-precision arithmetic is for operating on inaccurate numbers with a small width and expanding them to a larger, more accurate representation. For example, two <a href="/facts/Half-precision_floating-point_format/7xPj7B3w">half-precision</a> or <a href="/facts/Bfloat16_floating-point_format/wo1Q6IJY">bfloat16</a> (16-bit) floating-point numbers may be multiplied together to result in a more accurate <a href="/facts/Single-precision_floating-point_format/e6Fez7BM">single-precision</a> (32-bit) float.<a class="footnote-ref" id="fnref:1" href="#fn:1"><sup>1</sup></a> In this way, mixed-precision arithmetic approximates <a href="/facts/Arbitrary-precision_arithmetic/cXxc8yB2">arbitrary-precision arithmetic</a>, albeit with a low number of possible precisions. </p><p><a href="/facts/Iterative_method/uKMAJhEh">Iterative algorithms</a> (like <a href="/facts/Gradient_descent/pFFrek0F">gradient descent</a>) are good candidates for mixed-precision arithmetic. In an iterative algorithm like <a href="/facts/Square_root/AfrzfBdQ">square root</a>, a coarse integral guess can be made and refined over many iterations until the error in precision makes it such that the smallest addition or subtraction to the guess is still too coarse to be an acceptable answer. When this happens, the precision can be increased to something more precise, which allows for smaller increments to be used for the approximation. </p><p><a href="/facts/Supercomputer/HT9NwGAN">Supercomputers</a> such as <a href="/facts/Summit_(supercomputer)/BmKWTpEk">Summit</a> utilize mixed-precision arithmetic to be more efficient with regards to memory and processing time, as well as power consumption.<a class="footnote-ref" id="fnref:2" href="#fn:2"><sup>2</sup></a><a class="footnote-ref" id="fnref:3" href="#fn:3"><sup>3</sup></a><a class="footnote-ref" id="fnref:4" href="#fn:4"><sup>4</sup></a> </p> <h3>Floating point format</h3> <p>A floating-point number is typically packed into a single bit-string, as the sign bit, the exponent field, and the significand or mantissa, from left to right. As an example, a <a href="/facts/IEEE_754/dD3e7zAl">IEEE 754</a> standard 32-bit float ("FP32", "float32", or "binary32") is packed as follows: </p><p>The IEEE 754 binary floats are: </p> <table><tbody><tr><th rowspan="2">Type</th><th colspan="4">Bits</th><td rowspan="7"></td><th rowspan="2">Exponent<p>bias</p></th><th rowspan="2">Bits<p>precision</p></th><th rowspan="2">Number of<p>decimal digits</p></th></tr><tr><th>Sign</th><th>Exponent</th><th>Significand</th><th>Total</th></tr><tr><td><a href="/facts/Half_precision/7xPj7B3w">Half</a> (<a href="/facts/IEEE_floating_point/dD3e7zAl">IEEE 754-2008</a>)</td><td>1</td><td>5</td><td>10</td><td>16</td><td>15</td><td>11</td><td>~3.3</td></tr><tr><td><a href="/facts/Single_precision/e6Fez7BM">Single</a></td><td>1</td><td>8</td><td>23</td><td>32</td><td>127</td><td>24</td><td>~7.2</td></tr><tr><td><a href="/facts/Double_precision/JYyXXYFM">Double</a></td><td>1</td><td>11</td><td>52</td><td>64</td><td>1023</td><td>53</td><td>~15.9</td></tr><tr><td><a href="/facts/Extended_precision/CVpdQYeC">x86 extended precision</a></td><td>1</td><td>15</td><td>64</td><td>80</td><td>16383</td><td>64</td><td>~19.2</td></tr><tr><td><a href="/facts/Quad_precision/50oPmapW">Quad</a></td><td>1</td><td>15</td><td>112</td><td>128</td><td>16383</td><td>113</td><td>~34.0</td></tr></tbody></table> <h2 id="machine-learning">Machine learning</h2> <p>Mixed-precision arithmetic is used in the field of <a href="/facts/Machine_learning/e0w0XJTu">machine learning</a>, since <a href="/facts/Gradient_descent/pFFrek0F">gradient descent</a> algorithms can use coarse and efficient half-precision floats for certain tasks, but can be more accurate if they use more precise but slower single-precision floats. Some platforms, including <a href="/facts/Nvidia/8QfCgqbu">Nvidia</a>, <a href="/facts/Intel/SMF0gJJX">Intel</a>, and <a href="/facts/Advanced_Micro_Devices/CB1G7QTb">AMD</a> CPUs and GPUs, provide mixed-precision arithmetic for this purpose, using coarse floats when possible, but expanding them to higher precision when necessary.<a class="footnote-ref" id="fnref:5" href="#fn:5"><sup>5</sup></a><a class="footnote-ref" id="fnref:6" href="#fn:6"><sup>6</sup></a><a class="footnote-ref" id="fnref:7" href="#fn:7"><sup>7</sup></a><a class="footnote-ref" id="fnref:8" href="#fn:8"><sup>8</sup></a> </p> <h3>Automatic mixed precision</h3> <p><a href="/facts/PyTorch/4Gv1t13B">PyTorch</a> implements automatic mixed-precision (AMP), which performs autocasting, gradient scaling, and loss scaling.<a class="footnote-ref" id="fnref:9" href="#fn:9"><sup>9</sup></a><a class="footnote-ref" id="fnref:10" href="#fn:10"><sup>10</sup></a> </p> <ul><li>The weights are stored in a master copy at a high precision, usually in FP32.</li> <li>Autocasting means automatically converting a floating-point number between different precisions, such as from FP32 to FP16, during training. For example, <a href="/facts/Matrix_multiplication/lYDB6Ro2">matrix multiplications</a> can often be performed in FP16 without loss of accuracy, even if the master copy weights are stored in FP32. Low-precision weights are used during forward pass.</li> <li>Gradient scaling means multiplying <a href="/facts/Gradient/TsR7uukj">gradients</a> by a constant factor during training, typically before the <a href="/facts/Stochastic_gradient_descent/HbcaYqQP">weight optimizer</a> update. This is done to prevent the gradients from underflowing to zero when using low-precision data types like FP16. Mathematically, if the unscaled gradient is g {\displaystyle \mathbf {g} } , the scaled gradient is s g {\displaystyle s\mathbf {g} } where s {\displaystyle s} is the scaling factor. Within the optimizer update, the scaled gradient is cast to a higher precision before it is scaled down (no longer underflowing, as it is in a higher precision) to update the weights.</li> <li>Loss scaling means multiplying the loss function by a constant factor during training, typically before <a href="/facts/Backpropagation/lCsIdKHc">backpropagation</a>. This is done to prevent the gradients from underflowing to zero when using low-precision data types. If the unscaled loss is L {\displaystyle {\mathcal {L}}} , the scaled loss is k L {\displaystyle k{\mathcal {L}}} where k {\displaystyle k} is the scaling factor. Since gradient scaling and loss scaling are mathematically equivalent by ∂ ( k L ) ∂ w = k ∂ L ∂ w {\displaystyle {\frac {\partial (k{\mathcal {L}})}{\partial \mathbf {w} }}=k{\frac {\partial {\mathcal {L}}}{\partial \mathbf {w} }}} , loss scaling is an implementation of gradient scaling.</li></ul> <p>PyTorch AMP uses <a href="/facts/Exponential_backoff/pk2cAbqm">exponential backoff</a> to automatically adjust the scale factor for loss scaling. That is, it periodically increase the scale factor. Whenever the gradients contain a <a href="/facts/NaN/DNnYJ4FU">NaN</a> (indicating overflow), the weight update is skipped, and the scale factor is decreased. </p> <h2 id="references">References</h2> <ol> <li id="fn:1"><p>"Difference Between Single-, Double-, Multi-, Mixed-Precision". NVIDIA Blog. 15 November 2019. Retrieved 30 December 2020. <a href="https://blogs.nvidia.com/blog/2019/11/15/whats-the-difference-between-single-double-multi-and-mixed-precision-computing/" target="_blank">https://blogs.nvidia.com/blog/2019/11/15/whats-the-difference-between-single-double-multi-and-mixed-precision-computing/</a> <a href="#fnref:1" class="footnote-back-ref">↩</a></p></li> <li id="fn:2"><p>"Difference Between Single-, Double-, Multi-, Mixed-Precision". NVIDIA Blog. 15 November 2019. Retrieved 30 December 2020. <a href="https://blogs.nvidia.com/blog/2019/11/15/whats-the-difference-between-single-double-multi-and-mixed-precision-computing/" target="_blank">https://blogs.nvidia.com/blog/2019/11/15/whats-the-difference-between-single-double-multi-and-mixed-precision-computing/</a> <a href="#fnref:2" class="footnote-back-ref">↩</a></p></li> <li id="fn:3"><p>Abdelfattah, Ahmad; Anzt, Hartwig; Boman, Erik G.; Carson, Erin; Cojean, Terry; Dongarra, Jack; Gates, Mark; Grützmacher, Thomas; Higham, Nicholas J.; Li, Sherry; Lindquist, Neil; Liu, Yang; Loe, Jennifer; Luszczek, Piotr; Nayak, Pratik; Pranesh, Sri; Rajamanickam, Siva; Ribizel, Tobias; Smith, Barry; Swirydowicz, Kasia; Thomas, Stephen; Tomov, Stanimire; Tsai, Yaohung M.; Yamazaki, Ichitaro; Urike Meier Yang (2020). "A Survey of Numerical Methods Utilizing Mixed Precision Arithmetic". arXiv:2007.06674 [cs.MS]. <a href="/wiki/Ulrike_Meier_Yang" target="_blank">/wiki/Ulrike_Meier_Yang</a> <a href="#fnref:3" class="footnote-back-ref">↩</a></p></li> <li id="fn:4"><p>Holt, Kris (8 June 2018). "The US again has the world's most powerful supercomputer". Engadget. Retrieved 20 July 2018. <a href="https://www.engadget.com/2018/06/08/summit-supercomputer-research-ai-oak-ridge/" target="_blank">https://www.engadget.com/2018/06/08/summit-supercomputer-research-ai-oak-ridge/</a> <a href="#fnref:4" class="footnote-back-ref">↩</a></p></li> <li id="fn:5"><p>"Difference Between Single-, Double-, Multi-, Mixed-Precision". NVIDIA Blog. 15 November 2019. Retrieved 30 December 2020. <a href="https://blogs.nvidia.com/blog/2019/11/15/whats-the-difference-between-single-double-multi-and-mixed-precision-computing/" target="_blank">https://blogs.nvidia.com/blog/2019/11/15/whats-the-difference-between-single-double-multi-and-mixed-precision-computing/</a> <a href="#fnref:5" class="footnote-back-ref">↩</a></p></li> <li id="fn:6"><p>Abdelfattah, Ahmad; Anzt, Hartwig; Boman, Erik G.; Carson, Erin; Cojean, Terry; Dongarra, Jack; Gates, Mark; Grützmacher, Thomas; Higham, Nicholas J.; Li, Sherry; Lindquist, Neil; Liu, Yang; Loe, Jennifer; Luszczek, Piotr; Nayak, Pratik; Pranesh, Sri; Rajamanickam, Siva; Ribizel, Tobias; Smith, Barry; Swirydowicz, Kasia; Thomas, Stephen; Tomov, Stanimire; Tsai, Yaohung M.; Yamazaki, Ichitaro; Urike Meier Yang (2020). "A Survey of Numerical Methods Utilizing Mixed Precision Arithmetic". arXiv:2007.06674 [cs.MS]. <a href="/wiki/Ulrike_Meier_Yang" target="_blank">/wiki/Ulrike_Meier_Yang</a> <a href="#fnref:6" class="footnote-back-ref">↩</a></p></li> <li id="fn:7"><p>Micikevicius, Paulius; Narang, Sharan; Alben, Jonah; Diamos, Gregory; Elsen, Erich; Garcia, David; Ginsburg, Boris; Houston, Michael; Kuchaiev, Oleksii (2018-02-15). "Mixed Precision Training". arXiv:1710.03740 [cs.AI]. <a href="/wiki/ArXiv_(identifier)" target="_blank">/wiki/ArXiv_(identifier)</a> <a href="#fnref:7" class="footnote-back-ref">↩</a></p></li> <li id="fn:8"><p>"Mixed-Precision Training of Deep Neural Networks". NVIDIA Technical Blog. 2017-10-11. Retrieved 2024-09-10. <a href="https://developer.nvidia.com/blog/mixed-precision-training-deep-neural-networks/" target="_blank">https://developer.nvidia.com/blog/mixed-precision-training-deep-neural-networks/</a> <a href="#fnref:8" class="footnote-back-ref">↩</a></p></li> <li id="fn:9"><p>"Mixed Precision — PyTorch Training Performance Guide". residentmario.github.io. Retrieved 2024-09-10. <a href="https://residentmario.github.io/pytorch-training-performance-guide/mixed-precision.html" target="_blank">https://residentmario.github.io/pytorch-training-performance-guide/mixed-precision.html</a> <a href="#fnref:9" class="footnote-back-ref">↩</a></p></li> <li id="fn:10"><p>"What Every User Should Know About Mixed Precision Training in PyTorch". PyTorch. Retrieved 2024-09-10. <a href="https://pytorch.org/blog/what-every-user-should-know-about-mixed-precision-training-in-pytorch/" target="_blank">https://pytorch.org/blog/what-every-user-should-know-about-mixed-precision-training-in-pytorch/</a> <a href="#fnref:10" class="footnote-back-ref">↩</a></p></li> </ol>