Entropy in thermodynamics and information theory

<h2 id="equivalence-of-form-of-the-defining-expressions">Equivalence of form of the defining expressions</h2>

The defining expression for <a href="/facts/Entropy/s52IXeFz">entropy</a> in the theory of <a href="/facts/Statistical_mechanics/6zMsNYYG">statistical mechanics</a> established by <a href="/facts/Ludwig_Boltzmann/xDeLroqt">Ludwig Boltzmann</a> and <a href="/facts/J._Willard_Gibbs/xyWwxQtA">J. Willard Gibbs</a> in the 1870s, is of the form:

S
        =
        −
        
          k
          
            B
          
        
        
          ∑
          
            i
          
        
        
          p
          
            i
          
        
        ln
        ⁡
        
          p
          
            i
          
        
        ,
      
    
    {\displaystyle S=-k_{\text{B}}\sum _{i}p_{i}\ln p_{i},}

where 
 
 
 
 
 p
 
 i
 
 
 
 
 {\displaystyle p_{i}}
 
 is the probability of the <a href="/facts/Microstate_(statistical_mechanics)/ArXuNn08">microstate</a> i taken from an <a href="/facts/Ensemble_(mathematical_physics)/Nsxv4FAI">equilibrium ensemble</a>, and 
 
 
 
 
 k
 
 B
 
 
 
 
 {\displaystyle k_{B}}
 
 is the <a href="/facts/Boltzmann_constant/5DIejyR3">Boltzmann constant</a>.
The defining expression for <a href="/facts/Information_entropy/NLg4NLvt">entropy</a> in the theory of <a href="/facts/Information_theory/qNLH3U5P">information</a> established by <a href="/facts/Claude_E._Shannon/EremsCws">Claude E. Shannon</a> in 1948 is of the form:

H
        =
        −
        
          ∑
          
            i
          
        
        
          p
          
            i
          
        
        
          log
          
            b
          
        
        ⁡
        
          p
          
            i
          
        
        ,
      
    
    {\displaystyle H=-\sum _{i}p_{i}\log _{b}p_{i},}

where 
 
 
 
 
 p
 
 i
 
 
 
 
 {\displaystyle p_{i}}
 
 is the probability of the message 
 
 
 
 
 m
 
 i
 
 
 
 
 {\displaystyle m_{i}}
 
 taken from the message space M, and b is the <a href="/facts/Base_of_a_logarithm/QeXaV97q">base</a> of the <a href="/facts/Logarithm/QeXaV97q">logarithm</a> used. Common values of b are 2, <a href="/facts/E_(mathematical_constant)/coEtSBk8">Euler's number e</a>, and 10, and the unit of entropy is <a href="/facts/Shannon_(unit)/VEevYzKM">shannon</a> (or <a href="/facts/Bit/S972NSaD">bit</a>) for b = 2, <a href="/facts/Nat_(unit)/UgNgfVgI">nat</a> for b = e, and <a href="/facts/Hartley_(unit)/HHnoxYMR">hartley</a> for b = 10.<a class="footnote-ref" id="fnref:1" href="#fn:1">1</a>
Mathematically H may also be seen as an average information, taken over the message space, because when a certain message occurs with probability pi, the information quantity −log(pi) (called <a href="/facts/Information_content/EkUdL669">information content</a> or self-information) will be obtained.
If all the microstates are equiprobable (a <a href="/facts/Microcanonical_ensemble/L4fE6SxT">microcanonical ensemble</a>), the statistical thermodynamic entropy reduces to the form, as given by Boltzmann,

S
        =
        
          k
          
            B
          
        
        ln
        ⁡
        W
        ,
      
    
    {\displaystyle S=k_{\text{B}}\ln W,}

where W is the number of microstates that corresponds to the macroscopic thermodynamic state. Therefore S depends on temperature.
If all the messages are equiprobable, the information entropy reduces to the <a href="/facts/Hartley_entropy/xA8E8Dp1">Hartley entropy</a>

H
        =
        
          log
          
            b
          
        
        ⁡
        
          |
        
        M
        
          |
        
         
        ,
      
    
    {\displaystyle H=\log _{b}|M|\ ,}

where 
 
 
 
 
 |
 
 M
 
 |
 
 
 
 {\displaystyle |M|}
 
 is the <a href="/facts/Cardinality/m6VPojwT">cardinality</a> of the message space M.
The logarithm in the thermodynamic definition is the <a href="/facts/Natural_logarithm/PnreKEzI">natural logarithm</a>. It can be shown that the <a href="/facts/Entropy_(statistical_thermodynamics)/gMFXXIck">Gibbs entropy</a> formula, with the natural logarithm, reproduces all of the properties of the macroscopic <a href="/facts/Classical_thermodynamics/q4hIx3yP">classical thermodynamics</a> of <a href="/facts/Rudolf_Clausius/RMgFgY7I">Rudolf Clausius</a>. (See article: <a href="/facts/Entropy_(statistical_views)/gMFXXIck">Entropy (statistical views)</a>).
The <a href="/facts/Logarithm/QeXaV97q">logarithm</a> can also be taken to the natural base in the case of information entropy. This is equivalent to choosing to measure information in nats instead of the usual <a href="/facts/Bit/S972NSaD">bits</a> (or more formally, shannons). In practice, information entropy is almost always calculated using base-2 logarithms, but this distinction amounts to nothing other than a change in units. One nat is about 1.44 shannons.
For a simple compressible system that can only perform volume work, the <a href="/facts/First_law_of_thermodynamics/hjL95WCr">first law of thermodynamics</a> becomes

d
        E
        =
        −
        p
        d
        V
        +
        T
        d
        S
        .
      
    
    {\displaystyle dE=-pdV+TdS.}

But one can equally well write this equation in terms of what physicists and chemists sometimes call the 'reduced' or dimensionless entropy, σ = S/k, so that

d
        E
        =
        −
        p
        d
        V
        +
        
          k
          
            B
          
        
        T
        d
        σ
        .
      
    
    {\displaystyle dE=-pdV+k_{\text{B}}Td\sigma .}

Just as S is conjugate to T, so σ is conjugate to kBT (the energy that is characteristic of T on a molecular scale).
Thus the definitions of entropy in statistical mechanics (The <a href="/facts/Entropy_(statistical_thermodynamics)/gMFXXIck"> Gibbs entropy formula</a> 
 
 
 
 S
 =
 −
 
 k
 
 
 B
 
 
 
 
 ∑
 
 i
 
 
 
 p
 
 i
 
 
 log
 ⁡
 
 p
 
 i
 
 
 
 
 {\displaystyle S=-k_{\mathrm {B} }\sum _{i}p_{i}\log p_{i}}
 
) and in classical thermodynamics (
 
 
 
 d
 S
 =
 
 
 
 δ
 
 Q
 
 rev
 
 
 
 T
 
 
 
 
 {\displaystyle dS={\frac {\delta Q_{\text{rev}}}{T}}}
 
, and the <a href="/facts/Fundamental_thermodynamic_relation/T4rHbSdV">fundamental thermodynamic relation</a>) are equivalent for <a href="/facts/Microcanonical_ensemble/L4fE6SxT">microcanonical ensemble</a>, and statistical ensembles describing a thermodynamic system in equilibrium with a reservoir, such as the <a href="/facts/Canonical_ensemble/GIhyzkb8">canonical ensemble</a>, <a href="/facts/Grand_canonical_ensemble/cvztr5Y0">grand canonical ensemble</a>, <a href="/facts/Isothermal%E2%80%93isobaric_ensemble/pFcQL5hQ">isothermal–isobaric ensemble</a>. This equivalence is commonly shown in textbooks. However, the equivalence between the thermodynamic definition of entropy and the <a href="/facts/Entropy_(statistical_thermodynamics)/gMFXXIck"> Gibbs entropy</a> is not general but instead an exclusive property of the <a href="/facts/Boltzmann_distribution/XBU530uu">generalized Boltzmann distribution</a>.<a class="footnote-ref" id="fnref:2" href="#fn:2">2</a>
Furthermore, it has been shown that the definitions of entropy in statistical mechanics is the only entropy that is equivalent to the classical thermodynamics entropy under the following postulates:<a class="footnote-ref" id="fnref:3" href="#fn:3">3</a>

<ol><li>The probability density function is proportional to some function of the ensemble parameters and random variables.</li><li>Thermodynamic state functions are described by ensemble averages of random variables.</li><li>At infinite temperature, all the microstates have the same probability.</li></ol>
<h2 id="theoretical-relationship">Theoretical relationship</h2>
Despite the foregoing, there is a difference between the two quantities. The <a href="/facts/Information_entropy/NLg4NLvt">information entropy</a> Η can be calculated for any <a href="/facts/Probability_distribution/EpsKKVRu">probability distribution</a> (if the "message" is taken to be that the event i which had probability pi occurred, out of the space of the events possible), while the <a href="/facts/Entropy/s52IXeFz">thermodynamic entropy</a> S refers to thermodynamic probabilities pi specifically. The difference is more theoretical than actual, however, because any probability distribution can be approximated arbitrarily closely by some thermodynamic system.
Moreover, a direct connection can be made between the two. If the probabilities in question are the thermodynamic probabilities pi: the (reduced) <a href="/facts/Gibbs_entropy/gMFXXIck">Gibbs entropy</a> σ can then be seen as simply the amount of Shannon information needed to define the detailed microscopic state of the system, given its macroscopic description. Or, in the words of <a href="/facts/G._N._Lewis/wWnV6QXL">G. N. Lewis</a> writing about chemical entropy in 1930, "Gain in entropy always means loss of information, and nothing more". To be more concrete, in the discrete case using base two logarithms, the reduced Gibbs entropy is equal to the average of the minimum number of yes–no questions needed to be answered in order to fully specify the <a href="/facts/Microstate_(statistical_mechanics)/ArXuNn08">microstate</a>, given that we know the macrostate.
Furthermore, the prescription to find the equilibrium distributions of statistical mechanics—such as the Boltzmann distribution—by maximising the Gibbs entropy subject to appropriate constraints (the <a href="/facts/Gibbs_algorithm/4sT6udmC">Gibbs algorithm</a>) can be seen as something not unique to thermodynamics, but as a principle of general relevance in statistical inference, if it is desired to find a <a href="/facts/Principle_of_maximum_entropy/T0gybmPF">maximally uninformative probability distribution</a>, subject to certain constraints on its averages. (These perspectives are explored further in the article <a href="/facts/Maximum_entropy_thermodynamics/lQWFMKU6">Maximum entropy thermodynamics</a>.)
The Shannon entropy in information theory is sometimes expressed in units of bits per symbol. The physical entropy may be on a "per quantity" basis (h) which is called "<a href="/facts/Intensive_and_extensive_properties/UcgSh8EG">intensive</a>" entropy instead of the usual total entropy which is called "extensive" entropy. The "shannons" of a message (Η) are its total "extensive" information entropy and is h times the number of bits in the message.
A direct and physically real relationship between h and S can be found by assigning a symbol to each microstate that occurs per mole, kilogram, volume, or particle of a homogeneous substance, then calculating the 'h' of these symbols. By theory or by observation, the symbols (microstates) will occur with different probabilities and this will determine h. If there are N moles, kilograms, volumes, or particles of the unit substance, the relationship between h (in bits per unit substance) and physical extensive entropy in nats is:

S
        =
        
          k
          
            
              B
            
          
        
        ln
        ⁡
        (
        2
        )
        N
        h
      
    
    {\displaystyle S=k_{\mathrm {B} }\ln(2)Nh}

where ln(2) is the conversion factor from base 2 of Shannon entropy to the natural base e of physical entropy. N h is the amount of information in bits needed to describe the state of a physical system with entropy S. <a href="/facts/Landauer%27s_principle/G6Y0m3FX">Landauer's principle</a> demonstrates the reality of this by stating the minimum energy E required (and therefore heat Q generated) by an ideally efficient memory change or logic operation by irreversibly erasing or merging N h bits of information will be S times the temperature which is

E
        =
        Q
        =
        T
        
          k
          
            
              B
            
          
        
        ln
        ⁡
        (
        2
        )
        N
        h
        ,
      
    
    {\displaystyle E=Q=Tk_{\mathrm {B} }\ln(2)Nh,}

where h is in informational bits and E and Q are in physical Joules. This has been experimentally confirmed.<a class="footnote-ref" id="fnref:4" href="#fn:4">4</a>
Temperature is a measure of the average kinetic energy per particle in an ideal gas (kelvins = ⁠2/3⁠ joules/kB) so the J/K units of kB is dimensionless (joule/joule). kb is the conversion factor from energy in ⁠3/2⁠ kelvins to joules for an ideal gas. If kinetic energy measurements per particle of an ideal gas were expressed as joules instead of kelvins, kb in the above equations would be replaced by 3/2. This shows that S is a true statistical measure of microstates that does not have a fundamental physical unit other than the units of information, in this case nats, which is just a statement of which logarithm base was chosen by convention.

<h2 id="information-is-physical">Information is physical</h2>
<h3>Szilard's engine</h3>

A physical <a href="/facts/Thought_experiment/GXlr0lz9">thought experiment</a> demonstrating how just the possession of information might in principle have thermodynamic consequences was established in 1929 by <a href="/facts/Le%C3%B3_Szil%C3%A1rd/VslvyPwI">Leó Szilárd</a>, in a refinement of the famous <a href="/facts/Maxwell%27s_demon/V8tqskvy">Maxwell's demon</a> scenario<a class="footnote-ref" id="fnref:5" href="#fn:5">5</a> (and a reversal of the <a href="/facts/Joule_expansion/kXcheXjn">Joule expansion</a> thought experiment).
Consider Maxwell's set-up, but with only a single gas particle in a box. If the demon knows which half of the box the particle is in (equivalent to a single bit of information), it can close a shutter between the two halves of the box, close a piston unopposed into the empty half of the box, and then extract 
 
 
 
 
 k
 
 B
 
 
 T
 ln
 ⁡
 2
 
 
 {\displaystyle k_{\text{B}}T\ln 2}
 
 joules of useful work if the shutter is opened again. The particle can then be left to isothermally expand back to its original equilibrium occupied volume. In just the right circumstances therefore, the possession of a single bit of Shannon information (a single bit of <a href="/facts/Negentropy/WHqI45gv">negentropy</a> in Brillouin's term) really does correspond to a reduction in the entropy of the physical system. The global entropy is not decreased, but information to free energy conversion is possible.
This thought experiment has been physically demonstrated, using a <a href="/facts/Phase-contrast_microscopy/BmUkx69q">phase-contrast microscope</a> equipped with a high speed camera connected to a computer, acting as the demon.<a class="footnote-ref" id="fnref:6" href="#fn:6">6</a> In this experiment, information to energy conversion is performed on a <a href="/facts/Brownian_motion/Th0g0wkr">Brownian</a> particle by means of feedback control; that is, synchronizing the work given to the particle with the information obtained on its position. Computing energy balances for different feedback protocols, has confirmed that the <a href="/facts/Jarzynski_equality/mrtcV4rR">Jarzynski equality</a> requires a generalization that accounts for the amount of information involved in the feedback.

<h3>Landauer's principle</h3>
Main article: <a href="/facts/Landauer%27s_principle/G6Y0m3FX">Landauer's principle</a>
In fact one can generalise: any information that has a physical representation must somehow be embedded in the statistical mechanical degrees of freedom of a physical system.
Thus, <a href="/facts/Rolf_Landauer/1yIDBbeQ">Rolf Landauer</a> argued in 1961, if one were to imagine starting with those degrees of freedom in a thermalised state, there would be a real reduction in thermodynamic entropy if they were then re-set to a known state. This can only be achieved under information-preserving microscopically deterministic dynamics if the uncertainty is somehow dumped somewhere else – i.e. if the entropy of the environment (or the non information-bearing degrees of freedom) is increased by at least an equivalent amount, as required by the Second Law, by gaining an appropriate quantity of heat: specifically kT ln(2) of heat for every 1 bit of randomness erased.
On the other hand, Landauer argued, there is no thermodynamic objection to a logically reversible operation potentially being achieved in a physically reversible way in the system. It is only logically irreversible operations – for example, the erasing of a bit to a known state, or the merging of two computation paths – which must be accompanied by a corresponding entropy increase. When information is physical, all processing of its representations, i.e. generation, encoding, transmission, decoding and interpretation, are natural processes where entropy increases by consumption of free energy.<a class="footnote-ref" id="fnref:7" href="#fn:7">7</a>
Applied to the Maxwell's demon/Szilard engine scenario, this suggests that it might be possible to "read" the state of the particle into a computing apparatus with no entropy cost; but only if the apparatus has already been SET into a known state, rather than being in a thermalised state of uncertainty. To SET (or RESET) the apparatus into this state will cost all the entropy that can be saved by knowing the state of Szilard's particle.
In 2008 and 2009, researchers showed that Landauer's principle can be derived from the <a href="/facts/Second_law_of_thermodynamics/m7SJ2eeH">second law of thermodynamics</a> and the entropy change associated with information gain, developing the thermodynamics of quantum and classical feedback-controlled systems.<a class="footnote-ref" id="fnref:8" href="#fn:8">8</a><a class="footnote-ref" id="fnref:9" href="#fn:9">9</a>

<h2 id="negentropy">Negentropy</h2>
Main article: <a href="/facts/Negentropy/WHqI45gv">Negentropy</a>
Shannon entropy has been related by physicist <a href="/facts/L%C3%A9on_Brillouin/8AAtqyeH">Léon Brillouin</a> to a concept sometimes called <a href="/facts/Negentropy/WHqI45gv">negentropy</a>. In 1953, Brillouin derived a general equation<a class="footnote-ref" id="fnref:10" href="#fn:10">10</a> stating that the changing of an information bit value requires at least kT ln(2) energy. This is the same energy as the work <a href="/facts/Leo_Szilard/VslvyPwI">Leo Szilard</a>'s engine produces in the idealistic case, which in turn equals the same quantity found by <a href="/facts/Rolf_Landauer/1yIDBbeQ">Landauer</a>. In his book,<a class="footnote-ref" id="fnref:11" href="#fn:11">11</a> he further explored this problem concluding that any cause of a bit value change (measurement, decision about a yes/no question, erasure, display, etc.) will require the same amount, kT ln(2), of energy. Consequently, acquiring information about a system's microstates is associated with an <a href="/facts/Entropy_production/HxNR6Yza">entropy production</a>, while erasure yields entropy production only when the bit value is changing. Setting up a bit of information in a sub-system originally in thermal equilibrium results in a local entropy reduction. However, there is no violation of the second law of thermodynamics, according to Brillouin, since a reduction in any local system's thermodynamic entropy results in an increase in thermodynamic entropy elsewhere. In this way, Brillouin clarified the meaning of negentropy which was considered as controversial because its earlier understanding can yield Carnot efficiency higher than one. Additionally, the relationship between energy and information formulated by Brillouin has been proposed as a connection between the amount of bits that the brain processes and the energy it consumes: Collell and Fauquet <a class="footnote-ref" id="fnref:12" href="#fn:12">12</a> argued that De Castro <a class="footnote-ref" id="fnref:13" href="#fn:13">13</a> analytically found the Landauer limit as the thermodynamic lower bound for brain computations. However, even though evolution is supposed to have "selected" the most energetically efficient processes, the physical lower bounds are not realistic quantities in the brain. Firstly, because the minimum processing unit considered in physics is the atom/molecule, which is distant from the actual way that brain operates; and, secondly, because neural networks incorporate important redundancy and noise factors that greatly reduce their efficiency.<a class="footnote-ref" id="fnref:14" href="#fn:14">14</a> Laughlin et al. <a class="footnote-ref" id="fnref:15" href="#fn:15">15</a> was the first to provide explicit quantities for the energetic cost of processing sensory information. Their findings in blowflies revealed that for visual sensory data, the cost of transmitting one bit of information is around 5 × 10−14 Joules, or equivalently 104 ATP molecules. Thus, neural processing efficiency is still far from Landauer's limit of kT ln(2) J, but as a curious fact, it is still much more efficient than modern computers. 
In 2009, Mahulikar & Herwig redefined thermodynamic negentropy as the specific entropy deficit of the dynamically ordered sub-system relative to its surroundings.<a class="footnote-ref" id="fnref:16" href="#fn:16">16</a> This definition enabled the formulation of the Negentropy Principle, which is mathematically shown to follow from the 2nd Law of Thermodynamics, during order existence.

<h2 id="quantum-theory">Quantum theory</h2>
See also: <a href="/facts/Holographic_principle/EBVaAgc5">Holographic principle § Energy, matter, and information equivalence</a>; and <a href="/facts/Quantum_relative_entropy/cdoXwgDh">Quantum relative entropy</a>
Hirschman showed,<a class="footnote-ref" id="fnref:17" href="#fn:17">17</a> cf. <a href="/facts/Hirschman_uncertainty/fT9UajM8">Hirschman uncertainty</a>, that <a href="/facts/Heisenberg%27s_uncertainty_principle/UUSoSp4T">Heisenberg's uncertainty principle</a> can be expressed as a particular lower bound on the sum of the classical distribution entropies of the quantum observable probability distributions of a quantum mechanical state, the square of the wave-function, in coordinate, and also momentum space, when expressed in <a href="/facts/Planck_units/xDU6KlDM">Planck units</a>. The resulting inequalities provide a tighter bound on the uncertainty relations of Heisenberg.
It is meaningful to assign a "<a href="/facts/Joint_entropy/V4WwuX6P">joint entropy</a>", because positions and momenta are quantum conjugate variables and are therefore not jointly observable. Mathematically, they have to be treated as <a href="/facts/Joint_distribution/klX2ksGY">joint distribution</a>.
Note that this joint entropy is not equivalent to the <a href="/facts/Von_Neumann_entropy/nzeJKowg">Von Neumann entropy</a>, −Tr ρ lnρ = −⟨lnρ⟩.
Hirschman's entropy is said to account for the full information content of a mixture of quantum states.<a class="footnote-ref" id="fnref:18" href="#fn:18">18</a>
(Dissatisfaction with the Von Neumann entropy from quantum information points of view has been expressed by Stotland, Pomeransky, Bachmat and Cohen, who have introduced a yet different definition of entropy that reflects the inherent uncertainty of quantum mechanical states. This definition allows distinction between the minimum uncertainty entropy of pure states, and the excess statistical entropy of mixtures.<a class="footnote-ref" id="fnref:19" href="#fn:19">19</a>)

<h2 id="see-also">See also</h2>

<ul><li><a href="/facts/Entropy/s52IXeFz">Thermodynamic entropy</a></li>
<li><a href="/facts/Information_entropy/NLg4NLvt">Information entropy</a></li>
<li><a href="/facts/Thermodynamics/q4hIx3yP">Thermodynamics</a></li>
<li><a href="/facts/Statistical_mechanics/6zMsNYYG">Statistical mechanics</a></li>
<li><a href="/facts/Information_theory/qNLH3U5P">Information theory</a></li>
<li><a href="/facts/Quantum_entanglement/afzslwN0">Quantum entanglement</a></li>
<li><a href="/facts/Quantum_decoherence/mngM3rxy">Quantum decoherence</a></li>
<li><a href="/facts/Fluctuation_theorem/z4JK6z7D">Fluctuation theorem</a></li>
<li><a href="/facts/Black_hole_entropy/LghBnFWD">Black hole entropy</a></li>
<li><a href="/facts/Black_hole_information_paradox/4FyhobyY">Black hole information paradox</a></li>
<li><a href="/facts/Entropy_(information_theory)/NLg4NLvt">Entropy (information theory)</a></li>
<li><a href="/facts/Entropy_(statistical_thermodynamics)/gMFXXIck">Entropy (statistical thermodynamics)</a></li>
<li><a href="/facts/Entropy_(order_and_disorder)/2aXkcJHM">Entropy (order and disorder)</a></li>
<li><a href="/facts/Orders_of_magnitude_(entropy)/vfTcrlAq">Orders of magnitude (entropy)</a></li></ul>

<h2 id="further-reading">Further reading</h2>
<ul><li>Bennett, C.H. (1973). <a href="https://ieeexplore.ieee.org/xpl/tocresult.jsp?isnumber=5391321&punumber=5288520">"Logical reversibility of computation"</a>. IBM J. Res. Dev. 17 (6): 525–532. <a href="/facts/Doi_(identifier)/muM9Etpq">doi</a>:<a href="https://doi.org/10.1147%2Frd.176.0525">10.1147/rd.176.0525</a>.</li>
<li><a href="/facts/L%C3%A9on_Brillouin/8AAtqyeH">Brillouin, Léon</a> (2004), <a href="https://books.google.com/books?id=DWo7lVRVnhcC">Science And Information Theory</a> (second ed.), Dover, <a href="/facts/ISBN_(identifier)/15AdSPa9">ISBN</a> 978-0-486-43918-1. [Republication of 1962 original.]</li>
<li>Frank, Michael P. (May–June 2002). <a href="http://www.cise.ufl.edu/research/revcomp/physlim/plpaper.html">"Physical Limits of Computing"</a>. Computing in Science and Engineering. 4 (3): 16–25. <a href="/facts/Bibcode_(identifier)/9HtdQSGB">Bibcode</a>:<a href="https://ui.adsabs.harvard.edu/abs/2002CSE.....4c..16F">2002CSE.....4c..16F</a>. <a href="/facts/CiteSeerX_(identifier)/SceDmd3c">CiteSeerX</a> <a href="https://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.429.1618">10.1.1.429.1618</a>. <a href="/facts/Doi_(identifier)/muM9Etpq">doi</a>:<a href="https://doi.org/10.1109%2F5992.998637">10.1109/5992.998637</a>. <a href="/facts/OSTI_(identifier)/9Q4WHJRV">OSTI</a> <a href="https://www.osti.gov/biblio/1373456">1373456</a>. <a href="/facts/S2CID_(identifier)/ldJsHa2Y">S2CID</a> <a href="https://api.semanticscholar.org/CorpusID:499628">499628</a>.</li>
<li>Greven, Andreas; Keller, Gerhard; Warnecke, Gerald, eds. (2003). <a href="https://books.google.com/books?id=ZKPGzjS0IhcC">Entropy</a>. Princeton University Press. <a href="/facts/ISBN_(identifier)/15AdSPa9">ISBN</a> 978-0-691-11338-8. (A highly technical collection of writings giving an overview of the concept of entropy as it appears in various disciplines.)</li>
<li>Kalinin, M.I.; Kononogov, S.A. (2005), "Boltzmann's constant, the energy meaning of temperature, and thermodynamic irreversibility", Measurement Techniques, 48 (7): 632–636, <a href="/facts/Bibcode_(identifier)/9HtdQSGB">Bibcode</a>:<a href="https://ui.adsabs.harvard.edu/abs/2005MeasT..48..632K">2005MeasT..48..632K</a>, <a href="/facts/Doi_(identifier)/muM9Etpq">doi</a>:<a href="https://doi.org/10.1007%2Fs11018-005-0195-9">10.1007/s11018-005-0195-9</a>, <a href="/facts/S2CID_(identifier)/ldJsHa2Y">S2CID</a> <a href="https://api.semanticscholar.org/CorpusID:118726162">118726162</a>.</li>
<li>Koutsoyiannis, D. (2011), "Hurst–Kolmogorov dynamics as a result of extremal entropy production", <a href="/facts/Physica_(journal)/P5RseF6f">Physica A</a>, 390 (8): 1424–1432, <a href="/facts/Bibcode_(identifier)/9HtdQSGB">Bibcode</a>:<a href="https://ui.adsabs.harvard.edu/abs/2011PhyA..390.1424K">2011PhyA..390.1424K</a>, <a href="/facts/Doi_(identifier)/muM9Etpq">doi</a>:<a href="https://doi.org/10.1016%2Fj.physa.2010.12.035">10.1016/j.physa.2010.12.035</a>.</li>
<li><a href="/facts/Rolf_Landauer/1yIDBbeQ">Landauer, R.</a> (1993). <a href="http://ieeexplore.ieee.org/xpl/tocresult.jsp?sortType%3Dasc_p_Sequence%26filter%3DAND%28p_IS_Number%3A13433%29&refinements=4294815729&pageNumber=1&resultAction=REFINE#">"Information is Physical"</a>. <a href="https://ieeexplore.ieee.org/xpl/conhome/4863/proceeding">Proc. Workshop on Physics and Computation PhysComp'92</a>. Los Alamitos: IEEE Comp. Sci.Press. pp. 1–4. <a href="/facts/Doi_(identifier)/muM9Etpq">doi</a>:<a href="https://doi.org/10.1109%2FPHYCMP.1992.615478">10.1109/PHYCMP.1992.615478</a>. <a href="/facts/ISBN_(identifier)/15AdSPa9">ISBN</a> 978-0-8186-3420-8. <a href="/facts/S2CID_(identifier)/ldJsHa2Y">S2CID</a> <a href="https://api.semanticscholar.org/CorpusID:60640035">60640035</a>.</li>
<li>Landauer, R. (1961). <a href="http://domino.research.ibm.com/tchjr/journalindex.nsf/c469af92ea9eceac85256bd50048567c/8a9d4b4e96887b8385256bfa0067fba2?OpenDocument">"Irreversibility and Heat Generation in the Computing Process"</a>. IBM J. Res. Dev. 5 (3): 183–191. <a href="/facts/Doi_(identifier)/muM9Etpq">doi</a>:<a href="https://doi.org/10.1147%2Frd.53.0183">10.1147/rd.53.0183</a>.</li>
<li>Leff, H.S.; Rex, A.F., eds. (1990). Maxwell's Demon: Entropy, Information, Computing. Princeton NJ: Princeton University Press. <a href="/facts/ISBN_(identifier)/15AdSPa9">ISBN</a> 978-0-691-08727-6.</li>
<li>Middleton, D. (1960). <a href="https://archive.org/details/introductiontost0000midd">An Introduction to Statistical Communication Theory</a>. McGraw-Hill.</li>
<li><a href="/facts/Claude_Shannon/EremsCws">Shannon, Claude E.</a> (July–October 1948). <a href="/facts/A_Mathematical_Theory_of_Communication/Dze1Bnpo">"A Mathematical Theory of Communication"</a>. Bell System Technical Journal. 27 (3): 379–423. <a href="/facts/Doi_(identifier)/muM9Etpq">doi</a>:<a href="https://doi.org/10.1002%2Fj.1538-7305.1948.tb01338.x">10.1002/j.1538-7305.1948.tb01338.x</a>. <a href="/facts/Hdl_(identifier)/rdebSxmC">hdl</a>:<a href="https://hdl.handle.net/10338.dmlcz%2F101429">10338.dmlcz/101429</a>. (<a href="http://www.alcatel-lucent.com/bstj/vol27-1948/articles/bstj27-3-379.pdf">as PDF</a>)</li></ul>
<h2 id="external-links">External links</h2>
<ul><li><a href="http://plato.stanford.edu/entries/information-entropy/">Information Processing and Thermodynamic Entropy</a> Stanford Encyclopedia of Philosophy.</li>
<li>An Intuitive Guide to the Concept of Entropy Arising in Various Sectors of Science — a wikibook on the interpretation of the concept of entropy.</li></ul>

<h2 id="references">References</h2>

<ol>
<li id="fn:1">Schneider, T.D, Information theory primer with an appendix on logarithms, National Cancer Institute, 14 April 2007. <a href="http://alum.mit.edu/www/toms/paper/primer/primer.pdf" target="_blank">http://alum.mit.edu/www/toms/paper/primer/primer.pdf</a> <a href="#fnref:1" class="footnote-back-ref">↩</a></li>
<li id="fn:2">Gao, Xiang; Gallicchio, Emilio; Roitberg, Adrian (2019). "The generalized Boltzmann distribution is the only distribution in which the Gibbs-Shannon entropy equals the thermodynamic entropy". The Journal of Chemical Physics. 151 (3): 034113. arXiv:1903.02121. Bibcode:2019JChPh.151c4113G. doi:10.1063/1.5111333. PMID 31325924. S2CID 118981017. <a href="/wiki/ArXiv_(identifier)" target="_blank">/wiki/ArXiv_(identifier)</a> <a href="#fnref:2" class="footnote-back-ref">↩</a></li>
<li id="fn:3">Gao, Xiang (March 2022). "The Mathematics of the Ensemble Theory". Results in Physics. 34: 105230. arXiv:2006.00485. Bibcode:2022ResPh..3405230G. doi:10.1016/j.rinp.2022.105230. S2CID 221978379. <a href="https://doi.org/10.1016%2Fj.rinp.2022.105230" target="_blank">https://doi.org/10.1016%2Fj.rinp.2022.105230</a> <a href="#fnref:3" class="footnote-back-ref">↩</a></li>
<li id="fn:4">Antoine Bérut; Artak Arakelyan; Artyom Petrosyan; Sergio Ciliberto; Raoul Dillenschneider; Eric Lutz (8 March 2012), "Experimental verification of Landauer's principle linking information and thermodynamics" (PDF), Nature, 483 (7388): 187–190, Bibcode:2012Natur.483..187B, doi:10.1038/nature10872, PMID 22398556, S2CID 9415026 <a href="http://www.physik.uni-kl.de/eggert/papers/raoul.pdf" target="_blank">http://www.physik.uni-kl.de/eggert/papers/raoul.pdf</a> <a href="#fnref:4" class="footnote-back-ref">↩</a></li>
<li id="fn:5">Szilard, Leo (1929). "Über die Entropieverminderung in einem thermodynamischen System bei Eingriffen intelligenter Wesen". Zeitschrift für Physik (in German). 53 (11–12): 840–856. Bibcode:1929ZPhy...53..840S. doi:10.1007/BF01341281. ISSN 0044-3328. S2CID 122038206. Available on-line in English at Aurellen.org. <a href="/wiki/Bibcode_(identifier)" target="_blank">/wiki/Bibcode_(identifier)</a> <a href="#fnref:5" class="footnote-back-ref">↩</a></li>
<li id="fn:6">Shoichi Toyabe; Takahiro Sagawa; Masahito Ueda; Eiro Muneyuki; Masaki Sano (2010-09-29). "Information heat engine: converting information to energy by feedback control". Nature Physics. 6 (12): 988–992. arXiv:1009.5287. Bibcode:2010NatPh...6..988T. doi:10.1038/nphys1821. S2CID 118444713. We demonstrated that free energy is obtained by a feedback control using the information about the system; information is converted to free energy, as the first realization of Szilard-type Maxwell's demon. <a href="/wiki/ArXiv_(identifier)" target="_blank">/wiki/ArXiv_(identifier)</a> <a href="#fnref:6" class="footnote-back-ref">↩</a></li>
<li id="fn:7">Karnani, M.; Pääkkönen, K.; Annila, A. (2009). "The physical character of information". Proc. R. Soc. A. 465 (2107): 2155–75. Bibcode:2009RSPSA.465.2155K. doi:10.1098/rspa.2009.0063. <a href="https://doi.org/10.1098%2Frspa.2009.0063" target="_blank">https://doi.org/10.1098%2Frspa.2009.0063</a> <a href="#fnref:7" class="footnote-back-ref">↩</a></li>
<li id="fn:8">Sagawa, Takahiro; Ueda, Masahito (2008-02-26). "Second Law of Thermodynamics with Discrete Quantum Feedback Control". Physical Review Letters. 100 (8): 080403. arXiv:0710.0956. Bibcode:2008PhRvL.100h0403S. doi:10.1103/PhysRevLett.100.080403. ISSN 0031-9007. PMID 18352605. <a href="https://link.aps.org/doi/10.1103/PhysRevLett.100.080403" target="_blank">https://link.aps.org/doi/10.1103/PhysRevLett.100.080403</a> <a href="#fnref:8" class="footnote-back-ref">↩</a></li>
<li id="fn:9">Cao, F. J.; Feito, M. (2009-04-10). "Thermodynamics of feedback controlled systems". Physical Review E. 79 (4): 041118. arXiv:0805.4824. Bibcode:2009PhRvE..79d1118C. doi:10.1103/PhysRevE.79.041118. ISSN 1539-3755. PMID 19518184. <a href="https://link.aps.org/doi/10.1103/PhysRevE.79.041118" target="_blank">https://link.aps.org/doi/10.1103/PhysRevE.79.041118</a> <a href="#fnref:9" class="footnote-back-ref">↩</a></li>
<li id="fn:10">Brillouin, Leon (1953). "The negentropy principle of information". Journal of Applied Physics. 24 (9): 1152–1163. Bibcode:1953JAP....24.1152B. doi:10.1063/1.1721463. <a href="/wiki/Bibcode_(identifier)" target="_blank">/wiki/Bibcode_(identifier)</a> <a href="#fnref:10" class="footnote-back-ref">↩</a></li>
<li id="fn:11">Leon Brillouin, Science and Information theory, Dover, 1956 <a href="#fnref:11" class="footnote-back-ref">↩</a></li>
<li id="fn:12">Collell, G; Fauquet, J. (June 2015). "Brain activity and cognition: a connection from thermodynamics and information theory". Frontiers in Psychology. 6 (4): 818. doi:10.3389/fpsyg.2015.00818. PMC 4468356. PMID 26136709. <a href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4468356" target="_blank">https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4468356</a> <a href="#fnref:12" class="footnote-back-ref">↩</a></li>
<li id="fn:13">De Castro, A. (November 2013). "The Thermodynamic Cost of Fast Thought". Minds and Machines. 23 (4): 473–487. arXiv:1201.5841. doi:10.1007/s11023-013-9302-x. S2CID 11180644. <a href="/wiki/ArXiv_(identifier)" target="_blank">/wiki/ArXiv_(identifier)</a> <a href="#fnref:13" class="footnote-back-ref">↩</a></li>
<li id="fn:14">Narayanan, N. S. at al. (2005). "Redundancy and synergy of neuronal ensembles in motor cortex". J. Neurosci. 25 (17): 4207–4216. doi:10.1523/JNEUROSCI.4697-04.2005. PMC 6725112. PMID 15858046. <a href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6725112" target="_blank">https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6725112</a> <a href="#fnref:14" class="footnote-back-ref">↩</a></li>
<li id="fn:15">Laughlin, S. B at al. (November 2013). "The metabolic cost of neural information". Nat. Neurosci. 1 (1): 36–41. doi:10.1038/236. PMID 10195106. S2CID 204995437. <a href="/wiki/Doi_(identifier)" target="_blank">/wiki/Doi_(identifier)</a> <a href="#fnref:15" class="footnote-back-ref">↩</a></li>
<li id="fn:16">Mahulikar, S.P.; Herwig, H. (August 2009). "Exact thermodynamic principles for dynamic order existence and evolution in chaos". Chaos, Solitons & Fractals. 41 (4): 1939–48. Bibcode:2009CSF....41.1939M. doi:10.1016/j.chaos.2008.07.051. <a href="/wiki/Bibcode_(identifier)" target="_blank">/wiki/Bibcode_(identifier)</a> <a href="#fnref:16" class="footnote-back-ref">↩</a></li>
<li id="fn:17">Hirschman, I.I. Jr. (January 1957). "A note on entropy". American Journal of Mathematics. 79 (1): 152–6. doi:10.2307/2372390. JSTOR 2372390. <a href="/wiki/Doi_(identifier)" target="_blank">/wiki/Doi_(identifier)</a> <a href="#fnref:17" class="footnote-back-ref">↩</a></li>
<li id="fn:18">Zachos, C. K. (2007). "A classical bound on quantum entropy". Journal of Physics A: Mathematical and Theoretical. 40 (21): F407 – F412. arXiv:hep-th/0609148. Bibcode:2007JPhA...40..407Z. doi:10.1088/1751-8113/40/21/F02. S2CID 1619604. <a href="/wiki/ArXiv_(identifier)" target="_blank">/wiki/ArXiv_(identifier)</a> <a href="#fnref:18" class="footnote-back-ref">↩</a></li>
<li id="fn:19">Alexander Stotland; Pomeransky; Eitan Bachmat; Doron Cohen (2004). "The information entropy of quantum mechanical states". Europhysics Letters. 67 (5): 700–6. arXiv:quant-ph/0401021. Bibcode:2004EL.....67..700S. CiteSeerX 10.1.1.252.8715. doi:10.1209/epl/i2004-10110-1. S2CID 51730529. <a href="/wiki/ArXiv_(identifier)" target="_blank">/wiki/ArXiv_(identifier)</a> <a href="#fnref:19" class="footnote-back-ref">↩</a></li>
</ol>

Entropy in thermodynamics and information theory open-in-new

Entropy in thermodynamics and information theory