Entropy rate

<h2 id="definition">Definition</h2>
<p>A process 
  
    
      
        X
      
    
    {\displaystyle X}
  
 with a <a href="/facts/Countable/Wj4DmWPv">countable</a> index gives rise to the sequence of its <a href="/facts/Joint_entropy/V4WwuX6P">joint entropies</a> 
  
    
      
        
          H
          
            n
          
        
        (
        
          X
          
            1
          
        
        ,
        
          X
          
            2
          
        
        ,
        …
        
          X
          
            n
          
        
        )
      
    
    {\displaystyle H_{n}(X_{1},X_{2},\dots X_{n})}
  
. If the limit exists, the entropy rate is defined as
</p>

H
        (
        X
        )
        :=
        
          lim
          
            n
            →
            ∞
          
        
        
          
            
              1
              n
            
          
        
        
          H
          
            n
          
        
        .
      
    
    {\displaystyle H(X):=\lim _{n\to \infty }{\tfrac {1}{n}}H_{n}.}

<p>Note that given any sequence 
  
    
      
        (
        
          a
          
            n
          
        
        
          )
          
            n
          
        
      
    
    {\displaystyle (a_{n})_{n}}
  
 with 
  
    
      
        
          a
          
            0
          
        
        =
        0
      
    
    {\displaystyle a_{0}=0}
  
 and letting 
  
    
      
        Δ
        
          a
          
            k
          
        
        :=
        
          a
          
            k
          
        
        −
        
          a
          
            k
            −
            1
          
        
      
    
    {\displaystyle \Delta a_{k}:=a_{k}-a_{k-1}}
  
, by <a href="/facts/Telescoping_series/9QHSco21">telescoping</a> one has 
  
    
      
        
          a
          
            n
          
        
        =
        
          
            
              ∑
              
                k
                =
                1
              
              
                n
              
            
          
        
        Δ
        
          a
          
            k
          
        
      
    
    {\displaystyle a_{n}={\textstyle \sum _{k=1}^{n}}\Delta a_{k}}
  
. The entropy rate thus computes the mean of the first 
  
    
      
        n
      
    
    {\displaystyle n}
  
 such entropy changes, with 
  
    
      
        n
      
    
    {\displaystyle n}
  
 <a href="/facts/Limit_(mathematics)/j8JBhj4e">going to</a> <a href="/facts/Infinity/cF506uD7">infinity</a>.
The behaviour of joint entropies from one index to the next is also explicitly subject in some <a href="/facts/Entropy_(information_theory)/NLg4NLvt">characterizations of entropy</a>.
</p>
<h2 id="discussion">Discussion</h2>
<p>While 
  
    
      
        X
      
    
    {\displaystyle X}
  
 may be understood as a sequence of random variables, the entropy rate 
  
    
      
        H
        (
        X
        )
      
    
    {\displaystyle H(X)}
  
 represents the average entropy change per one random variable, in the long term. 
</p><p>It can be thought of as a general property of stochastic sources - this is the subject of the <a href="/facts/Asymptotic_equipartition_property/Jcy9SLWc">asymptotic equipartition property</a>.
</p>
<h3>For strongly stationary processes</h3>
<p>A stochastic process also gives rise to a sequence of conditional entropies, comprising more and more random variables.
For strongly stationary stochastic processes, the entropy rate equals the limit of that sequence
</p>

H
        (
        X
        )
        =
        
          lim
          
            n
            →
            ∞
          
        
        H
        (
        
          X
          
            n
          
        
        
          |
        
        
          X
          
            n
            −
            1
          
        
        ,
        
          X
          
            n
            −
            2
          
        
        ,
        …
        
          X
          
            1
          
        
        )
      
    
    {\displaystyle H(X)=\lim _{n\to \infty }H(X_{n}|X_{n-1},X_{n-2},\dots X_{1})}

<p>The quantity given by the limit on the right is also denoted 
  
    
      
        
          H
          ′
        
        (
        X
        )
      
    
    {\displaystyle H'(X)}
  
, which is motivated to the extent that here this is then again a rate associated with the process, in the above sense.
</p>
<h3>For Markov chains</h3>
<p>Since a stochastic process defined by a <a href="/facts/Markov_chain/5oP5hntk">Markov chain</a> that is <a href="/facts/Markov_chain/5oP5hntk">irreducible</a>, <a href="/facts/Aperiodic/Sm8lJQsF">aperiodic</a>
and <a href="/facts/Markov_chain/5oP5hntk">positive recurrent</a> has a <a href="/facts/Stationary_distribution/IBLlwnPr">stationary distribution</a>, the entropy rate is independent of the initial distribution.
</p><p>For example, consider a Markov chain defined on a <a href="/facts/Countable/Wj4DmWPv">countable</a> number of states. Given its <a href="/facts/Stochastic_matrix/2cRWAqC3">right stochastic transition matrix</a> 
  
    
      
        
          P
          
            i
            j
          
        
      
    
    {\displaystyle P_{ij}}
  
 and an entropy
</p>

h
          
            i
          
        
        :=
        −
        
          ∑
          
            j
          
        
        
          P
          
            i
            j
          
        
        log
        ⁡
        
          P
          
            i
            j
          
        
      
    
    {\displaystyle h_{i}:=-\sum _{j}P_{ij}\log P_{ij}}

<p>associated with each state, one finds
</p>

H
          (
          X
          )
          =
          
            ∑
            
              i
            
          
          
            μ
            
              i
            
          
          
            h
            
              i
            
          
          ,
        
      
    
    {\displaystyle \displaystyle H(X)=\sum _{i}\mu _{i}h_{i},}

<p>where 
  
    
      
        
          μ
          
            i
          
        
      
    
    {\displaystyle \mu _{i}}
  
 is the <a href="/facts/Asymptotic_distribution/dN7jVHqm">asymptotic distribution</a> of the chain.
</p><p>In particular, it follows that the entropy rate of an <a href="/facts/Independent_and_identically_distributed/othIRaWt">i.i.d.</a> <a href="/facts/Stochastic_process/Ng1n1dnB">stochastic process</a> is the same as the entropy of any individual member in the process.
</p>
<h3>For hidden Markov models</h3>
<p>The entropy rate of hidden Markov models (HMM) has no known closed-form solution. However, it has known upper and lower bounds. Let the underlying Markov chain 
  
    
      
        
          X
          
            1
            :
            ∞
          
        
      
    
    {\displaystyle X_{1:\infty }}
  
 be stationary, and let 
  
    
      
        
          Y
          
            1
            :
            ∞
          
        
      
    
    {\displaystyle Y_{1:\infty }}
  
 be the observable states, then we have
  
    
      
        H
        (
        
          Y
          
            n
          
        
        
          |
        
        
          X
          
            1
          
        
        ,
        
          Y
          
            1
            :
            n
            −
            1
          
        
        )
        ≤
        H
        (
        Y
        )
        ≤
        H
        (
        
          Y
          
            n
          
        
        
          |
        
        
          Y
          
            1
            :
            n
            −
            1
          
        
        )
      
    
    {\displaystyle H(Y_{n}|X_{1},Y_{1:n-1})\leq H(Y)\leq H(Y_{n}|Y_{1:n-1})}
  
and at the limit of 
  
    
      
        n
        →
        ∞
      
    
    {\displaystyle n\to \infty }
  
, both sides converge to the middle.<a class="footnote-ref" id="fnref:1" href="#fn:1"><sup>1</sup></a>
</p>
<h2 id="applications">Applications</h2>
<p>The entropy rate may be used to estimate the complexity of stochastic processes. It is used in diverse applications ranging from characterizing the complexity of languages, blind source separation, through to optimizing quantizers and data compression algorithms. For example, a maximum entropy rate criterion may be used for <a href="/facts/Feature_selection/gxbuF4VG">feature selection</a> in <a href="/facts/Machine_learning/e0w0XJTu">machine learning</a>.<a class="footnote-ref" id="fnref:2" href="#fn:2"><sup>2</sup></a>
</p>
<h2 id="see-also">See also</h2>
<ul><li><a href="/facts/Information_source_(mathematics)/CVsytjcq">Information source (mathematics)</a></li>
<li><a href="/facts/Markov_information_source/66EqoHuD">Markov information source</a></li>
<li><a href="/facts/Asymptotic_equipartition_property/Jcy9SLWc">Asymptotic equipartition property</a></li>
<li><a href="/facts/Maximal_entropy_random_walk/vfEcXHhY">Maximal entropy random walk</a> - chosen to maximize entropy rate</li></ul>

<ul><li>Cover, T. and Thomas, J. (1991) Elements of Information Theory, John Wiley and Sons, Inc., <a href="/facts/ISBN_(identifier)/15AdSPa9">ISBN</a> 0-471-06259-6 <a href="https://archive.today/20121216133431/http://www3.interscience.wiley.com/cgi-bin/bookhome/110438582?CRETRY=1&SRETRY=0">[1]</a></li></ul>

<h2 id="references">References</h2>

<ol>
<li id="fn:1"><p>Cover, Thomas M.; Thomas, Joy A. (2006). "4.5. Functions of Markov chains". Elements of information theory (2nd ed.). Hoboken, N.J: Wiley-Interscience. ISBN 978-0-471-24195-9. <a href="978-0-471-24195-9" target="_blank">978-0-471-24195-9</a> <a href="#fnref:1" class="footnote-back-ref">↩</a></p></li>
<li id="fn:2"><p>Einicke, G. A. (2018). "Maximum-Entropy Rate Selection of Features for Classifying Changes in Knee and Ankle Dynamics During Running". IEEE Journal of Biomedical and Health Informatics. 28 (4): 1097–1103. doi:10.1109/JBHI.2017.2711487. hdl:10810/68978. PMID 29969403. S2CID 49555941. <a href="/wiki/Doi_(identifier)" target="_blank">/wiki/Doi_(identifier)</a> <a href="#fnref:2" class="footnote-back-ref">↩</a></p></li>
</ol>

Entropy rate open-in-new

Entropy rate