Rényi entropy

<h2 id="definition">Definition</h2>
The Rényi entropy of order ⁠
 
 
 
 α
 
 
 {\displaystyle \alpha }
 
⁠, where 
 
 
 
 0
 <
 α
 <
 ∞
 
 
 {\displaystyle 0<\alpha <\infty }
 
 and ⁠
 
 
 
 α
 ≠
 1
 
 
 {\displaystyle \alpha \neq 1}
 
⁠, is defined as<a class="footnote-ref" id="fnref:6" href="#fn:6">6</a>

H
          
          
            α
          
        
        (
        X
        )
        =
        
          
            1
            
              1
              −
              α
            
          
        
        log
        ⁡
        
          
            (
          
        
        
          ∑
          
            i
            =
            1
          
          
            n
          
        
        
          p
          
            i
          
          
            α
          
        
        
          
            )
          
        
        .
      
    
    {\displaystyle \mathrm {H} _{\alpha }(X)={\frac {1}{1-\alpha }}\log {\Bigg (}\sum _{i=1}^{n}p_{i}^{\alpha }{\Bigg )}.}

It is further defined at 
 
 
 
 α
 =
 0
 ,
 1
 ,
 ∞
 
 
 {\displaystyle \alpha =0,1,\infty }
 
 as

H
          
          
            α
          
        
        (
        X
        )
        =
        
          lim
          
            γ
            →
            α
          
        
        
          
            H
          
          
            γ
          
        
        (
        X
        )
        .
      
    
    {\displaystyle \mathrm {H} _{\alpha }(X)=\lim _{\gamma \to \alpha }\mathrm {H} _{\gamma }(X).}

Here, 
 
 
 
 X
 
 
 {\displaystyle X}
 
 is a discrete random variable with possible outcomes in the set 
 
 
 
 
 
 A
 
 
 =
 {
 
 x
 
 1
 
 
 ,
 
 x
 
 2
 
 
 ,
 .
 .
 .
 ,
 
 x
 
 n
 
 
 }
 
 
 {\displaystyle {\mathcal {A}}=\{x_{1},x_{2},...,x_{n}\}}
 
 and corresponding probabilities 
 
 
 
 
 p
 
 i
 
 
 ≐
 Pr
 (
 X
 =
 
 x
 
 i
 
 
 )
 
 
 {\displaystyle p_{i}\doteq \Pr(X=x_{i})}
 
 for ⁠
 
 
 
 i
 =
 1
 ,
 …
 ,
 n
 
 
 {\displaystyle i=1,\dots ,n}
 
⁠. The resulting <a href="/facts/Unit_of_information/lxjSrVJM">unit of information</a> is determined by the base of the <a href="/facts/Logarithm/QeXaV97q">logarithm</a>, e.g. <a href="/facts/Shannon_(unit)/VEevYzKM">shannon</a> for base 2, or <a href="/facts/Nat_(unit)/UgNgfVgI">nat</a> for base <a href="/facts/E_(mathematical_constant)/coEtSBk8">e</a>.
If the probabilities are 
 
 
 
 
 p
 
 i
 
 
 =
 1
 
 /
 
 n
 
 
 {\displaystyle p_{i}=1/n}
 
 for all ⁠
 
 
 
 i
 =
 1
 ,
 …
 ,
 n
 
 
 {\displaystyle i=1,\dots ,n}
 
⁠, then all the Rényi entropies of the distribution are equal: ⁠
 
 
 
 
 
 H
 
 
 α
 
 
 (
 X
 )
 =
 log
 ⁡
 n
 
 
 {\displaystyle \mathrm {H} _{\alpha }(X)=\log n}
 
⁠.
In general, for all discrete random variables ⁠
 
 
 
 X
 
 
 {\displaystyle X}
 
⁠, 
 
 
 
 
 
 H
 
 
 α
 
 
 (
 X
 )
 
 
 {\displaystyle \mathrm {H} _{\alpha }(X)}
 
 is a non-increasing function in ⁠
 
 
 
 α
 
 
 {\displaystyle \alpha }
 
⁠.
Applications often exploit the following relation between the Rényi entropy and the <a href="/facts/Lp_space/gbad0Rv1">α-norm</a> of the vector of probabilities:

H
          
          
            α
          
        
        (
        X
        )
        =
        
          
            α
            
              1
              −
              α
            
          
        
        log
        ⁡
        
          (
          
            ‖
            P
            
              ‖
              
                α
              
            
          
          )
        
        .
      
    
    {\displaystyle \mathrm {H} _{\alpha }(X)={\frac {\alpha }{1-\alpha }}\log \left(\|P\|_{\alpha }\right).}

Here, the discrete probability distribution 
 
 
 
 P
 =
 (
 
 p
 
 1
 
 
 ,
 …
 ,
 
 p
 
 n
 
 
 )
 
 
 {\displaystyle P=(p_{1},\dots ,p_{n})}
 
 is interpreted as a vector in 
 
 
 
 
 
 R
 
 
 n
 
 
 
 
 {\displaystyle \mathbb {R} ^{n}}
 
 with 
 
 
 
 
 p
 
 i
 
 
 ≥
 0
 
 
 {\displaystyle p_{i}\geq 0}
 
 and ⁠
 
 
 
 
 
 ∑
 
 i
 =
 1
 
 
 n
 
 
 
 p
 
 i
 
 
 =
 1
 
 
 
 {\displaystyle \textstyle \sum _{i=1}^{n}p_{i}=1}
 
⁠.
The Rényi entropy for any 
 
 
 
 α
 ≥
 0
 
 
 {\displaystyle \alpha \geq 0}
 
 is <a href="/facts/Schur-concave_function/HdRAVIWD">Schur concave</a>. Proven by the <a href="/facts/Schur-convex_function/HdRAVIWD">Schur–Ostrowski criterion.</a>

<h2 id="special-cases">Special cases</h2>

As 
 
 
 
 α
 
 
 {\displaystyle \alpha }
 
 approaches zero, the Rényi entropy increasingly weighs all events with nonzero probability more equally, regardless of their probabilities. In the limit for ⁠
 
 
 
 α
 →
 0
 
 
 {\displaystyle \alpha \to 0}
 
⁠, the Rényi entropy is just the logarithm of the size of the <a href="/facts/Support_(mathematics)/BTuMGbsb">support</a> of X. The limit for 
 
 
 
 α
 →
 1
 
 
 {\displaystyle \alpha \to 1}
 
 is the <a href="/facts/Shannon_entropy/NLg4NLvt">Shannon entropy</a>. As 
 
 
 
 α
 
 
 {\displaystyle \alpha }
 
 approaches infinity, the Rényi entropy is increasingly determined by the events of highest probability.

<h3>Hartley or max-entropy</h3>

 
 
 
 
 
 H
 
 
 0
 
 
 (
 X
 )
 
 
 {\displaystyle \mathrm {H} _{0}(X)}
 
 is 
 
 
 
 log
 ⁡
 n
 
 
 {\displaystyle \log n}
 
 where 
 
 
 
 n
 
 
 {\displaystyle n}
 
 is the number of non-zero probabilities.<a class="footnote-ref" id="fnref:7" href="#fn:7">7</a> If the probabilities are all nonzero, it is simply the logarithm of the <a href="/facts/Cardinality/m6VPojwT">cardinality</a> of the alphabet (⁠
 
 
 
 
 
 A
 
 
 
 
 {\displaystyle {\mathcal {A}}}
 
⁠) of ⁠
 
 
 
 X
 
 
 {\displaystyle X}
 
⁠, sometimes called the <a href="/facts/Hartley_entropy/xA8E8Dp1">Hartley entropy</a> of ⁠
 
 
 
 X
 
 
 {\displaystyle X}
 
⁠,

H
          
          
            0
          
        
        (
        X
        )
        =
        log
        ⁡
        n
        =
        log
        ⁡
        
          |
        
        
          
            A
          
        
        
          |
        
        
      
    
    {\displaystyle \mathrm {H} _{0}(X)=\log n=\log |{\mathcal {A}}|\,}

<h3>Shannon entropy</h3>
The limiting value of 
 
 
 
 
 
 H
 
 
 α
 
 
 
 
 {\displaystyle \mathrm {H} _{\alpha }}
 
 as 
 
 
 
 α
 →
 1
 
 
 {\displaystyle \alpha \to 1}
 
 is the <a href="/facts/Shannon_entropy/NLg4NLvt">Shannon entropy</a>:<a class="footnote-ref" id="fnref:8" href="#fn:8">8</a>

H
          
          
            1
          
        
        (
        X
        )
        ≡
        
          lim
          
            α
            →
            1
          
        
        
          
            H
          
          
            α
          
        
        (
        X
        )
        =
        −
        
          ∑
          
            i
            =
            1
          
          
            n
          
        
        
          p
          
            i
          
        
        log
        ⁡
        
          p
          
            i
          
        
      
    
    {\displaystyle \mathrm {H} _{1}(X)\equiv \lim _{\alpha \to 1}\mathrm {H} _{\alpha }(X)=-\sum _{i=1}^{n}p_{i}\log p_{i}}

<h3>Collision entropy</h3>
Collision entropy, sometimes just called "Rényi entropy", refers to the case ⁠
 
 
 
 α
 =
 2
 
 
 {\displaystyle \alpha =2}
 
⁠,

H
          
          
            2
          
        
        (
        X
        )
        =
        −
        log
        ⁡
        
          ∑
          
            i
            =
            1
          
          
            n
          
        
        
          p
          
            i
          
          
            2
          
        
        =
        −
        log
        ⁡
        P
        (
        X
        =
        Y
        )
        ,
      
    
    {\displaystyle \mathrm {H} _{2}(X)=-\log \sum _{i=1}^{n}p_{i}^{2}=-\log P(X=Y),}

where 
 
 
 
 X
 
 
 {\displaystyle X}
 
 and 
 
 
 
 Y
 
 
 {\displaystyle Y}
 
 are <a href="/facts/Independent_and_identically_distributed/othIRaWt">independent and identically distributed</a>. The collision entropy is related to the <a href="/facts/Index_of_coincidence/7uW1sGer">index of coincidence</a>. It is the negative logarithm of the <a href="/facts/Simpson_diversity_index/E6NWQEry">Simpson diversity index</a>.

<h3>Min-entropy</h3>
Main article: <a href="/facts/Min-entropy/U0j0SNcp">Min-entropy</a>
In the limit as ⁠
 
 
 
 α
 →
 ∞
 
 
 {\displaystyle \alpha \rightarrow \infty }
 
⁠, the Rényi entropy 
 
 
 
 
 
 H
 
 
 α
 
 
 
 
 {\displaystyle \mathrm {H} _{\alpha }}
 
 converges to the min-entropy ⁠
 
 
 
 
 
 H
 
 
 ∞
 
 
 
 
 {\displaystyle \mathrm {H} _{\infty }}
 
⁠:

H
          
          
            ∞
          
        
        (
        X
        )
        ≐
        
          min
          
            i
          
        
        (
        −
        log
        ⁡
        
          p
          
            i
          
        
        )
        =
        −
        (
        
          max
          
            i
          
        
        log
        ⁡
        
          p
          
            i
          
        
        )
        =
        −
        log
        ⁡
        
          max
          
            i
          
        
        
          p
          
            i
          
        
        
        .
      
    
    {\displaystyle \mathrm {H} _{\infty }(X)\doteq \min _{i}(-\log p_{i})=-(\max _{i}\log p_{i})=-\log \max _{i}p_{i}\,.}

Equivalently, the min-entropy 
 
 
 
 
 
 H
 
 
 ∞
 
 
 (
 X
 )
 
 
 {\displaystyle \mathrm {H} _{\infty }(X)}
 
 is the largest real number b such that all events occur with probability at most ⁠
 
 
 
 
 2
 
 −
 b
 
 
 
 
 {\displaystyle 2^{-b}}
 
⁠.
The name min-entropy stems from the fact that it is the smallest entropy measure in the family of Rényi entropies.
In this sense, it is the strongest way to measure the information content of a discrete random variable.
In particular, the min-entropy is never larger than the <a href="/facts/Shannon_entropy/NLg4NLvt">Shannon entropy</a>.
The min-entropy has important applications for <a href="/facts/Randomness_extractor/YxS1Ia7x">randomness extractors</a> in <a href="/facts/Theoretical_computer_science/jSkGGuvy">theoretical computer science</a>:
Extractors are able to extract randomness from random sources that have a large min-entropy; merely having a large <a href="/facts/Shannon_entropy/NLg4NLvt">Shannon entropy</a> does not suffice for this task.

<h2 id="inequalities-for-different-orders-">Inequalities for different orders α</h2>
That 
 
 
 
 
 
 H
 
 
 α
 
 
 
 
 {\displaystyle \mathrm {H} _{\alpha }}
 
 is non-increasing in 
 
 
 
 α
 
 
 {\displaystyle \alpha }
 
 for any given distribution of probabilities ⁠
 
 
 
 
 p
 
 i
 
 
 
 
 {\displaystyle p_{i}}
 
⁠,
which can be proven by differentiation,<a class="footnote-ref" id="fnref:9" href="#fn:9">9</a> as

−
        
          
            
              d
              
                
                  H
                
                
                  α
                
              
            
            
              d
              α
            
          
        
        =
        
          
            1
            
              (
              1
              −
              α
              
                )
                
                  2
                
              
            
          
        
        
          ∑
          
            i
            =
            1
          
          
            n
          
        
        
          z
          
            i
          
        
        log
        ⁡
        (
        
          z
          
            i
          
        
        
          /
        
        
          p
          
            i
          
        
        )
        =
        
          
            1
            
              (
              1
              −
              α
              
                )
                
                  2
                
              
            
          
        
        
          D
          
            K
            L
          
        
        (
        z
        ‖
        p
        )
      
    
    {\displaystyle -{\frac {d\mathrm {H} _{\alpha }}{d\alpha }}={\frac {1}{(1-\alpha )^{2}}}\sum _{i=1}^{n}z_{i}\log(z_{i}/p_{i})={\frac {1}{(1-\alpha )^{2}}}D_{KL}(z\|p)}

which is proportional to <a href="/facts/Kullback%E2%80%93Leibler_divergence/nh7SjlPE">Kullback–Leibler divergence</a> (which is always non-negative), where
⁠
 
 
 
 
 
 z
 
 i
 
 
 =
 
 p
 
 i
 
 
 α
 
 
 
 /
 
 
 ∑
 
 j
 =
 1
 
 
 n
 
 
 
 p
 
 j
 
 
 α
 
 
 
 
 
 {\displaystyle \textstyle z_{i}=p_{i}^{\alpha }/\sum _{j=1}^{n}p_{j}^{\alpha }}
 
⁠. In particular, it is strictly positive except when the distribution is uniform.
At the 
 
 
 
 α
 →
 1
 
 
 {\displaystyle \alpha \to 1}
 
 limit, we have ⁠
 
 
 
 −
 
 
 
 d
 
 
 H
 
 
 α
 
 
 
 
 d
 α
 
 
 
 →
 
 
 1
 2
 
 
 
 ∑
 
 i
 
 
 
 p
 
 i
 
 
 (
 ln
 ⁡
 
 p
 
 i
 
 
 +
 H
 (
 p
 )
 
 )
 
 2
 
 
 
 
 {\displaystyle -{\frac {d\mathrm {H} _{\alpha }}{d\alpha }}\to {\frac {1}{2}}\sum _{i}p_{i}(\ln p_{i}+H(p))^{2}}
 
⁠.
In particular cases inequalities can be proven also by <a href="/facts/Jensen%27s_inequality/sosRQWN1">Jensen's inequality</a>:<a class="footnote-ref" id="fnref:10" href="#fn:10">10</a><a class="footnote-ref" id="fnref:11" href="#fn:11">11</a>

log
        ⁡
        n
        =
        
          
            H
          
          
            0
          
        
        ≥
        
          
            H
          
          
            1
          
        
        ≥
        
          
            H
          
          
            2
          
        
        ≥
        
          
            H
          
          
            ∞
          
        
        .
      
    
    {\displaystyle \log n=\mathrm {H} _{0}\geq \mathrm {H} _{1}\geq \mathrm {H} _{2}\geq \mathrm {H} _{\infty }.}

For values of ⁠
 
 
 
 α
 >
 1
 
 
 {\displaystyle \alpha >1}
 
⁠, inequalities in the other direction also hold. In particular, we have<a class="footnote-ref" id="fnref:12" href="#fn:12">12</a><a class="footnote-ref" id="fnref:13" href="#fn:13">13</a>

H
          
          
            2
          
        
        ≤
        2
        
          
            H
          
          
            ∞
          
        
        .
      
    
    {\displaystyle \mathrm {H} _{2}\leq 2\mathrm {H} _{\infty }.}

On the other hand, the Shannon entropy 
 
 
 
 
 
 H
 
 
 1
 
 
 
 
 {\displaystyle \mathrm {H} _{1}}
 
 can be arbitrarily high for a random variable 
 
 
 
 X
 
 
 {\displaystyle X}
 
 that has a given min-entropy. An example of this is given by the sequence of random variables 
 
 
 
 
 X
 
 n
 
 
 ∼
 {
 0
 ,
 …
 ,
 n
 }
 
 
 {\displaystyle X_{n}\sim \{0,\ldots ,n\}}
 
 for 
 
 
 
 n
 ≥
 1
 
 
 {\displaystyle n\geq 1}
 
 such that 
 
 
 
 P
 (
 
 X
 
 n
 
 
 =
 0
 )
 =
 1
 
 /
 
 2
 
 
 {\displaystyle P(X_{n}=0)=1/2}
 
 and 
 
 
 
 P
 (
 
 X
 
 n
 
 
 =
 x
 )
 =
 1
 
 /
 
 (
 2
 n
 )
 
 
 {\displaystyle P(X_{n}=x)=1/(2n)}
 
 since 
 
 
 
 
 
 H
 
 
 ∞
 
 
 (
 
 X
 
 n
 
 
 )
 =
 log
 ⁡
 2
 
 
 {\displaystyle \mathrm {H} _{\infty }(X_{n})=\log 2}
 
 but ⁠
 
 
 
 
 
 H
 
 
 1
 
 
 (
 
 X
 
 n
 
 
 )
 =
 (
 log
 ⁡
 2
 +
 log
 ⁡
 2
 n
 )
 
 /
 
 2
 
 
 {\displaystyle \mathrm {H} _{1}(X_{n})=(\log 2+\log 2n)/2}
 
⁠.

<h2 id="rnyi-divergence">Rényi divergence</h2>
As well as the absolute Rényi entropies, Rényi also defined a spectrum of divergence measures generalising the <a href="/facts/Kullback%E2%80%93Leibler_divergence/nh7SjlPE">Kullback–Leibler divergence</a>.<a class="footnote-ref" id="fnref:14" href="#fn:14">14</a>
The Rényi divergence of order ⁠
 
 
 
 α
 
 
 {\displaystyle \alpha }
 
⁠ or alpha-divergence of a distribution P from a distribution Q is defined to be

D
          
            α
          
        
        (
        P
        ‖
        Q
        )
        =
        
          
            1
            
              α
              −
              1
            
          
        
        log
        ⁡
        
          
            (
          
        
        
          ∑
          
            i
            =
            1
          
          
            n
          
        
        
          
            
              p
              
                i
              
              
                α
              
            
            
              q
              
                i
              
              
                α
                −
                1
              
            
          
        
        
          
            )
          
        
        =
        
          
            1
            
              α
              −
              1
            
          
        
        log
        ⁡
        
          
            E
          
          
            i
            ∼
            p
          
        
        [
        (
        
          p
          
            i
          
        
        
          /
        
        
          q
          
            i
          
        
        
          )
          
            α
            −
            1
          
        
        ]
        
      
    
    {\displaystyle D_{\alpha }(P\Vert Q)={\frac {1}{\alpha -1}}\log {\Bigg (}\sum _{i=1}^{n}{\frac {p_{i}^{\alpha }}{q_{i}^{\alpha -1}}}{\Bigg )}={\frac {1}{\alpha -1}}\log \mathbb {E} _{i\sim p}[(p_{i}/q_{i})^{\alpha -1}]\,}

when ⁠
 
 
 
 0
 <
 α
 <
 ∞
 
 
 {\displaystyle 0<\alpha <\infty }
 
⁠ and ⁠
 
 
 
 α
 ≠
 1
 
 
 {\displaystyle \alpha \neq 1}
 
⁠. We can define the Rényi divergence for the special values α = 0, 1, ∞ by taking a limit, and in particular the limit α → 1 gives the Kullback–Leibler divergence.
Some special cases:

⁠
 
 
 
 
 D
 
 0
 
 
 (
 P
 ‖
 Q
 )
 =
 −
 log
 ⁡
 Q
 (
 {
 i
 :
 
 p
 
 i
 
 
 >
 0
 }
 )
 
 
 {\displaystyle D_{0}(P\Vert Q)=-\log Q(\{i:p_{i}>0\})}
 
⁠: minus the log probability under Q that pi > 0;
⁠
 
 
 
 
 D
 
 1
 
 /
 
 2
 
 
 (
 P
 ‖
 Q
 )
 =
 −
 2
 log
 ⁡
 
 ∑
 
 i
 =
 1
 
 
 n
 
 
 
 
 
 p
 
 i
 
 
 
 q
 
 i
 
 
 
 
 
 
 {\displaystyle D_{1/2}(P\Vert Q)=-2\log \sum _{i=1}^{n}{\sqrt {p_{i}q_{i}}}}
 
⁠: minus twice the logarithm of the <a href="/facts/Bhattacharyya_coefficient/3kw3a7SF">Bhattacharyya coefficient</a>; (Nielsen & Boltz (2010))
⁠
 
 
 
 
 D
 
 1
 
 
 (
 P
 ‖
 Q
 )
 =
 
 ∑
 
 i
 =
 1
 
 
 n
 
 
 
 p
 
 i
 
 
 log
 ⁡
 
 
 
 p
 
 i
 
 
 
 q
 
 i
 
 
 
 
 
 
 {\displaystyle D_{1}(P\Vert Q)=\sum _{i=1}^{n}p_{i}\log {\frac {p_{i}}{q_{i}}}}
 
⁠: the <a href="/facts/Kullback%E2%80%93Leibler_divergence/nh7SjlPE">Kullback–Leibler divergence</a>;
⁠
 
 
 
 
 D
 
 2
 
 
 (
 P
 ‖
 Q
 )
 =
 log
 ⁡
 
 
 ⟨
 
 
 
 
 
 p
 
 i
 
 
 
 q
 
 i
 
 
 
 
 
 
 ⟩
 
 
 
 
 {\displaystyle D_{2}(P\Vert Q)=\log {\Big \langle }{\frac {p_{i}}{q_{i}}}{\Big \rangle }}
 
⁠: the log of the expected ratio of the probabilities;
⁠
 
 
 
 
 D
 
 ∞
 
 
 (
 P
 ‖
 Q
 )
 =
 log
 ⁡
 
 sup
 
 i
 
 
 
 
 
 p
 
 i
 
 
 
 q
 
 i
 
 
 
 
 
 
 {\displaystyle D_{\infty }(P\Vert Q)=\log \sup _{i}{\frac {p_{i}}{q_{i}}}}
 
⁠: the log of the maximum ratio of the probabilities.
The Rényi divergence is indeed a <a href="/facts/Divergence_(statistics)/8qrn1djm">divergence</a>, meaning simply that 
 
 
 
 
 D
 
 α
 
 
 (
 P
 ‖
 Q
 )
 
 
 {\displaystyle D_{\alpha }(P\|Q)}
 
 is greater than or equal to zero, and zero only when P = Q. For any fixed distributions P and Q, the Rényi divergence is nondecreasing as a function of its order α, and it is continuous on the set of α for which it is finite,<a class="footnote-ref" id="fnref:15" href="#fn:15">15</a> or for the sake of brevity, the information of order α obtained if the distribution P is replaced by the distribution Q.<a class="footnote-ref" id="fnref:16" href="#fn:16">16</a>

<h2 id="financial-interpretation">Financial interpretation</h2>
A pair of probability distributions can be viewed as a game of chance in which one of the distributions defines official odds and the other contains the actual probabilities. Knowledge of the actual probabilities allows a player to profit from the game. The expected profit rate is connected to the Rényi divergence as follows<a class="footnote-ref" id="fnref:17" href="#fn:17">17</a>

E
            x
            p
            e
            c
            t
            e
            d
            R
            a
            t
            e
          
        
        =
        
          
            1
            R
          
        
        
        
          D
          
            1
          
        
        (
        b
        ‖
        m
        )
        +
        
          
            
              R
              −
              1
            
            R
          
        
        
        
          D
          
            1
            
              /
            
            R
          
        
        (
        b
        ‖
        m
        )
        
        ,
      
    
    {\displaystyle {\rm {ExpectedRate}}={\frac {1}{R}}\,D_{1}(b\|m)+{\frac {R-1}{R}}\,D_{1/R}(b\|m)\,,}

where 
 
 
 
 m
 
 
 {\displaystyle m}
 
 is the distribution defining the official odds (i.e. the "market") for the game, 
 
 
 
 b
 
 
 {\displaystyle b}
 
 is the investor-believed distribution and 
 
 
 
 R
 
 
 {\displaystyle R}
 
 is the investor's risk aversion (the <a href="/facts/Risk_aversion/erarEwDf">Arrow–Pratt relative risk aversion</a>).
If the true distribution is 
 
 
 
 p
 
 
 {\displaystyle p}
 
 (not necessarily coinciding with the investor's belief ⁠
 
 
 
 b
 
 
 {\displaystyle b}
 
⁠), the long-term realized rate converges to the true expectation which has a similar mathematical structure<a class="footnote-ref" id="fnref:18" href="#fn:18">18</a>

R
            e
            a
            l
            i
            z
            e
            d
            R
            a
            t
            e
          
        
        =
        
          
            1
            R
          
        
        
        
          
            (
          
        
        
          D
          
            1
          
        
        (
        p
        ‖
        m
        )
        −
        
          D
          
            1
          
        
        (
        p
        ‖
        b
        )
        
          
            )
          
        
        +
        
          
            
              R
              −
              1
            
            R
          
        
        
        
          D
          
            1
            
              /
            
            R
          
        
        (
        b
        ‖
        m
        )
        
        .
      
    
    {\displaystyle {\rm {RealizedRate}}={\frac {1}{R}}\,{\Big (}D_{1}(p\|m)-D_{1}(p\|b){\Big )}+{\frac {R-1}{R}}\,D_{1/R}(b\|m)\,.}

<h2 id="properties-specific-to--1">Properties specific to α = 1</h2>
The value ⁠
 
 
 
 α
 =
 1
 
 
 {\displaystyle \alpha =1}
 
⁠, which gives the <a href="/facts/Shannon_entropy/NLg4NLvt">Shannon entropy</a> and the <a href="/facts/Kullback%E2%80%93Leibler_divergence/nh7SjlPE">Kullback–Leibler divergence</a>, is the only value at which the <a href="/facts/Conditional_information/gHzPezGB">chain rule of conditional probability</a> holds exactly:

H
        
        (
        A
        ,
        X
        )
        =
        
          H
        
        (
        A
        )
        +
        
          
            E
          
          
            a
            ∼
            A
          
        
        
          
            [
          
        
        
          H
        
        (
        X
        
          |
        
        A
        =
        a
        )
        
          
            ]
          
        
      
    
    {\displaystyle \mathrm {H} (A,X)=\mathrm {H} (A)+\mathbb {E} _{a\sim A}{\big [}\mathrm {H} (X|A=a){\big ]}}

for the absolute entropies, and

D
          
            
              K
              L
            
          
        
        (
        p
        (
        x
        
          |
        
        a
        )
        p
        (
        a
        )
        ‖
        m
        (
        x
        ,
        a
        )
        )
        =
        
          D
          
            
              K
              L
            
          
        
        (
        p
        (
        a
        )
        ‖
        m
        (
        a
        )
        )
        +
        
          
            E
          
          
            p
            (
            a
            )
          
        
        {
        
          D
          
            
              K
              L
            
          
        
        (
        p
        (
        x
        
          |
        
        a
        )
        ‖
        m
        (
        x
        
          |
        
        a
        )
        )
        }
        ,
      
    
    {\displaystyle D_{\mathrm {KL} }(p(x|a)p(a)\|m(x,a))=D_{\mathrm {KL} }(p(a)\|m(a))+\mathbb {E} _{p(a)}\{D_{\mathrm {KL} }(p(x|a)\|m(x|a))\},}

for the relative entropies.
The latter in particular means that if we seek a distribution p(x, a) which minimizes the divergence from some underlying prior measure m(x, a), and we acquire new information which only affects the distribution of a, then the distribution of p(x|a) remains m(x|a), unchanged.
The other Rényi divergences satisfy the criteria of being positive and continuous, being invariant under 1-to-1 co-ordinate transformations, and of combining additively when A and X are independent, so that if p(A, X) = p(A)p(X), then

H
          
          
            α
          
        
        (
        A
        ,
        X
        )
        =
        
          
            H
          
          
            α
          
        
        (
        A
        )
        +
        
          
            H
          
          
            α
          
        
        (
        X
        )
        
      
    
    {\displaystyle \mathrm {H} _{\alpha }(A,X)=\mathrm {H} _{\alpha }(A)+\mathrm {H} _{\alpha }(X)\;}

and

D
          
            α
          
        
        (
        P
        (
        A
        )
        P
        (
        X
        )
        ‖
        Q
        (
        A
        )
        Q
        (
        X
        )
        )
        =
        
          D
          
            α
          
        
        (
        P
        (
        A
        )
        ‖
        Q
        (
        A
        )
        )
        +
        
          D
          
            α
          
        
        (
        P
        (
        X
        )
        ‖
        Q
        (
        X
        )
        )
        .
      
    
    {\displaystyle D_{\alpha }(P(A)P(X)\|Q(A)Q(X))=D_{\alpha }(P(A)\|Q(A))+D_{\alpha }(P(X)\|Q(X)).}

The stronger properties of the 
 
 
 
 α
 =
 1
 
 
 {\displaystyle \alpha =1}
 
 quantities allow the definition of <a href="/facts/Conditional_information/gHzPezGB">conditional information</a> and <a href="/facts/Mutual_information/HIUvsjvV">mutual information</a> from communication theory.

<h2 id="exponential-families">Exponential families</h2>
The Rényi entropies and divergences for an <a href="/facts/Exponential_family/1LkkqEIf">exponential family</a> admit simple expressions<a class="footnote-ref" id="fnref:19" href="#fn:19">19</a>

H
          
          
            α
          
        
        (
        
          p
          
            F
          
        
        (
        x
        ;
        θ
        )
        )
        =
        
          
            1
            
              1
              −
              α
            
          
        
        
          (
          
            F
            (
            α
            θ
            )
            −
            α
            F
            (
            θ
            )
            +
            log
            ⁡
            
              E
              
                p
              
            
            [
            
              e
              
                (
                α
                −
                1
                )
                k
                (
                x
                )
              
            
            ]
          
          )
        
      
    
    {\displaystyle \mathrm {H} _{\alpha }(p_{F}(x;\theta ))={\frac {1}{1-\alpha }}\left(F(\alpha \theta )-\alpha F(\theta )+\log E_{p}[e^{(\alpha -1)k(x)}]\right)}

and

D
          
            α
          
        
        (
        p
        :
        q
        )
        =
        
          
            
              
                J
                
                  F
                  ,
                  α
                
              
              (
              θ
              :
              
                θ
                ′
              
              )
            
            
              1
              −
              α
            
          
        
      
    
    {\displaystyle D_{\alpha }(p:q)={\frac {J_{F,\alpha }(\theta :\theta ')}{1-\alpha }}}

where

J
          
            F
            ,
            α
          
        
        (
        θ
        :
        
          θ
          ′
        
        )
        =
        α
        F
        (
        θ
        )
        +
        (
        1
        −
        α
        )
        F
        (
        
          θ
          ′
        
        )
        −
        F
        (
        α
        θ
        +
        (
        1
        −
        α
        )
        
          θ
          ′
        
        )
      
    
    {\displaystyle J_{F,\alpha }(\theta :\theta ')=\alpha F(\theta )+(1-\alpha )F(\theta ')-F(\alpha \theta +(1-\alpha )\theta ')}

is a Jensen difference divergence.

<h2 id="physical-meaning">Physical meaning</h2>
The Rényi entropy in quantum physics is not considered to be an <a href="/facts/Observable/fcBAsBjm">observable</a>, due to its nonlinear dependence on the density matrix. (This nonlinear dependence applies even in the special case of the Shannon entropy.) It can, however, be given an operational meaning through the two-time measurements (also known as full counting statistics) of energy transfers.
The limit of the quantum mechanical Rényi entropy as 
 
 
 
 α
 →
 1
 
 
 {\displaystyle \alpha \to 1}
 
 is the <a href="/facts/Von_Neumann_entropy/nzeJKowg">von Neumann entropy</a>.

<h2 id="see-also">See also</h2>
<ul><li><a href="/facts/Diversity_indices/E6NWQEry">Diversity indices</a></li>
<li><a href="/facts/Tsallis_entropy/Ikjxefzh">Tsallis entropy</a></li>
<li><a href="/facts/Generalized_entropy_index/82No2FYm">Generalized entropy index</a></li></ul>
<h2 id="notes">Notes</h2>

<ul><li>Beck, Christian; Schlögl, Friedrich (1993). Thermodynamics of chaotic systems: an introduction. Cambridge University Press. <a href="/facts/ISBN_(identifier)/15AdSPa9">ISBN</a> 0521433673.</li>
<li>Jizba, P.; Arimitsu, T. (2004). "The world according to Rényi: Thermodynamics of multifractal systems". Annals of Physics. 312 (1): 17–59. <a href="/facts/ArXiv_(identifier)/H6EtgnBe">arXiv</a>:<a href="https://arxiv.org/abs/cond-mat/0207707">cond-mat/0207707</a>. <a href="/facts/Bibcode_(identifier)/9HtdQSGB">Bibcode</a>:<a href="https://ui.adsabs.harvard.edu/abs/2004AnPhy.312...17J">2004AnPhy.312...17J</a>. <a href="/facts/Doi_(identifier)/muM9Etpq">doi</a>:<a href="https://doi.org/10.1016%2Fj.aop.2004.01.002">10.1016/j.aop.2004.01.002</a>. <a href="/facts/S2CID_(identifier)/ldJsHa2Y">S2CID</a> <a href="https://api.semanticscholar.org/CorpusID:119704502">119704502</a>.</li>
<li>Jizba, P.; Arimitsu, T. (2004). "On observability of Rényi's entropy". Physical Review E. 69 (2): 026128. <a href="/facts/ArXiv_(identifier)/H6EtgnBe">arXiv</a>:<a href="https://arxiv.org/abs/cond-mat/0307698">cond-mat/0307698</a>. <a href="/facts/Bibcode_(identifier)/9HtdQSGB">Bibcode</a>:<a href="https://ui.adsabs.harvard.edu/abs/2004PhRvE..69b6128J">2004PhRvE..69b6128J</a>. <a href="/facts/Doi_(identifier)/muM9Etpq">doi</a>:<a href="https://doi.org/10.1103%2FPhysRevE.69.026128">10.1103/PhysRevE.69.026128</a>. <a href="/facts/PMID_(identifier)/JlHAvMHt">PMID</a> <a href="https://pubmed.ncbi.nlm.nih.gov/14995541">14995541</a>. <a href="/facts/S2CID_(identifier)/ldJsHa2Y">S2CID</a> <a href="https://api.semanticscholar.org/CorpusID:39231939">39231939</a>.</li>
<li>Bromiley, P.A.; Thacker, N.A.; Bouhova-Thacker, E. (2004), Shannon Entropy, Rényi Entropy, and Information, <a href="/facts/CiteSeerX_(identifier)/SceDmd3c">CiteSeerX</a> <a href="https://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.330.9856">10.1.1.330.9856</a></li>
<li>Franchini, F.; Its, A. R.; Korepin, V. E. (2008). "Rényi entropy as a measure of entanglement in quantum spin chain". Journal of Physics A: Mathematical and Theoretical. 41 (25302): 025302. <a href="/facts/ArXiv_(identifier)/H6EtgnBe">arXiv</a>:<a href="https://arxiv.org/abs/0707.2534">0707.2534</a>. <a href="/facts/Bibcode_(identifier)/9HtdQSGB">Bibcode</a>:<a href="https://ui.adsabs.harvard.edu/abs/2008JPhA...41b5302F">2008JPhA...41b5302F</a>. <a href="/facts/Doi_(identifier)/muM9Etpq">doi</a>:<a href="https://doi.org/10.1088%2F1751-8113%2F41%2F2%2F025302">10.1088/1751-8113/41/2/025302</a>. <a href="/facts/S2CID_(identifier)/ldJsHa2Y">S2CID</a> <a href="https://api.semanticscholar.org/CorpusID:119672750">119672750</a>.</li>
<li><a href="https://www.encyclopediaofmath.org/index.php?title=Rényi_test">"Rényi test"</a>, <a href="/facts/Encyclopedia_of_Mathematics/WC6mGtPm">Encyclopedia of Mathematics</a>, <a href="/facts/European_Mathematical_Society/B3h7b672">EMS Press</a>, 2001 [1994]</li>
<li>Hero, A. O.; Michael, O.; Gorman, J. (2002). <a href="http://www.eecs.umich.edu/~hero/Preprints/cspl-328.pdf">Alpha-divergence for Classification, Indexing and Retrieval</a> (PDF) (Technical Report CSPL-328). Communications and Signal Processing Laboratory, University of Michigan. <a href="/facts/CiteSeerX_(identifier)/SceDmd3c">CiteSeerX</a> <a href="https://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.373.2763">10.1.1.373.2763</a>.</li>
<li>Its, A. R.; Korepin, V. E. (2010). "Generalized entropy of the Heisenberg spin chain". Theoretical and Mathematical Physics. 164 (3): 1136–1139. <a href="/facts/Bibcode_(identifier)/9HtdQSGB">Bibcode</a>:<a href="https://ui.adsabs.harvard.edu/abs/2010TMP...164.1136I">2010TMP...164.1136I</a>. <a href="/facts/Doi_(identifier)/muM9Etpq">doi</a>:<a href="https://doi.org/10.1007%2Fs11232-010-0091-6">10.1007/s11232-010-0091-6</a>. <a href="/facts/S2CID_(identifier)/ldJsHa2Y">S2CID</a> <a href="https://api.semanticscholar.org/CorpusID:119525704">119525704</a>.</li>
<li>Nielsen, F.; Boltz, S. (2010). "The Burbea-Rao and Bhattacharyya centroids". IEEE Transactions on Information Theory. 57 (8): 5455–5466. <a href="/facts/ArXiv_(identifier)/H6EtgnBe">arXiv</a>:<a href="https://arxiv.org/abs/1004.5049">1004.5049</a>. <a href="/facts/Doi_(identifier)/muM9Etpq">doi</a>:<a href="https://doi.org/10.1109%2FTIT.2011.2159046">10.1109/TIT.2011.2159046</a>. <a href="/facts/S2CID_(identifier)/ldJsHa2Y">S2CID</a> <a href="https://api.semanticscholar.org/CorpusID:14238708">14238708</a>.</li>
<li>Nielsen, Frank; Nock, Richard (2012). "A closed-form expression for the Sharma–Mittal entropy of exponential families". Journal of Physics A. 45 (3): 032003. <a href="/facts/ArXiv_(identifier)/H6EtgnBe">arXiv</a>:<a href="https://arxiv.org/abs/1112.4221">1112.4221</a>. <a href="/facts/Bibcode_(identifier)/9HtdQSGB">Bibcode</a>:<a href="https://ui.adsabs.harvard.edu/abs/2012JPhA...45c2003N">2012JPhA...45c2003N</a>. <a href="/facts/Doi_(identifier)/muM9Etpq">doi</a>:<a href="https://doi.org/10.1088%2F1751-8113%2F45%2F3%2F032003">10.1088/1751-8113/45/3/032003</a>. <a href="/facts/S2CID_(identifier)/ldJsHa2Y">S2CID</a> <a href="https://api.semanticscholar.org/CorpusID:8653096">8653096</a>.</li>
<li>Nielsen, Frank; Nock, Richard (2011). "On Rényi and Tsallis entropies and divergences for exponential families". Journal of Physics A. 45 (3): 032003. <a href="/facts/ArXiv_(identifier)/H6EtgnBe">arXiv</a>:<a href="https://arxiv.org/abs/1105.3259">1105.3259</a>. <a href="/facts/Bibcode_(identifier)/9HtdQSGB">Bibcode</a>:<a href="https://ui.adsabs.harvard.edu/abs/2012JPhA...45c2003N">2012JPhA...45c2003N</a>. <a href="/facts/Doi_(identifier)/muM9Etpq">doi</a>:<a href="https://doi.org/10.1088%2F1751-8113%2F45%2F3%2F032003">10.1088/1751-8113/45/3/032003</a>. <a href="/facts/S2CID_(identifier)/ldJsHa2Y">S2CID</a> <a href="https://api.semanticscholar.org/CorpusID:8653096">8653096</a>.</li>
<li><a href="/facts/Alfr%C3%A9d_R%C3%A9nyi/q7Dx7fgB">Rényi, Alfréd</a> (1961). <a href="http://digitalassets.lib.berkeley.edu/math/ucb/text/math_s4_v1_article-27.pdf">"On measures of information and entropy"</a> (PDF). Proceedings of the fourth Berkeley Symposium on Mathematics, Statistics and Probability 1960. pp. 547–561.</li>
<li>Rosso, O. A. (2006). "EEG analysis using wavelet-based information tools". Journal of Neuroscience Methods. 153 (2): 163–182. <a href="/facts/Doi_(identifier)/muM9Etpq">doi</a>:<a href="https://doi.org/10.1016%2Fj.jneumeth.2005.10.009">10.1016/j.jneumeth.2005.10.009</a>. <a href="/facts/PMID_(identifier)/JlHAvMHt">PMID</a> <a href="https://pubmed.ncbi.nlm.nih.gov/16675027">16675027</a>. <a href="/facts/S2CID_(identifier)/ldJsHa2Y">S2CID</a> <a href="https://api.semanticscholar.org/CorpusID:7134638">7134638</a>.</li>
<li>Zachos, C. K. (2007). "A classical bound on quantum entropy". Journal of Physics A. 40 (21): F407 – F412. <a href="/facts/ArXiv_(identifier)/H6EtgnBe">arXiv</a>:<a href="https://arxiv.org/abs/hep-th/0609148">hep-th/0609148</a>. <a href="/facts/Bibcode_(identifier)/9HtdQSGB">Bibcode</a>:<a href="https://ui.adsabs.harvard.edu/abs/2007JPhA...40..407Z">2007JPhA...40..407Z</a>. <a href="/facts/Doi_(identifier)/muM9Etpq">doi</a>:<a href="https://doi.org/10.1088%2F1751-8113%2F40%2F21%2FF02">10.1088/1751-8113/40/21/F02</a>. <a href="/facts/S2CID_(identifier)/ldJsHa2Y">S2CID</a> <a href="https://api.semanticscholar.org/CorpusID:1619604">1619604</a>.</li>
<li>Nazarov, Y. (2011). "Flows of Rényi entropies". Physical Review B. 84 (10): 205437. <a href="/facts/ArXiv_(identifier)/H6EtgnBe">arXiv</a>:<a href="https://arxiv.org/abs/1108.3537">1108.3537</a>. <a href="/facts/Bibcode_(identifier)/9HtdQSGB">Bibcode</a>:<a href="https://ui.adsabs.harvard.edu/abs/2015PhRvB..91j4303A">2015PhRvB..91j4303A</a>. <a href="/facts/Doi_(identifier)/muM9Etpq">doi</a>:<a href="https://doi.org/10.1103%2FPhysRevB.91.104303">10.1103/PhysRevB.91.104303</a>. <a href="/facts/S2CID_(identifier)/ldJsHa2Y">S2CID</a> <a href="https://api.semanticscholar.org/CorpusID:40312624">40312624</a>.</li>
<li>Ansari, Mohammad H.; Nazarov, Yuli V. (2015). "Rényi entropy flows from quantum heat engines". Physical Review B. 91 (10): 104303. <a href="/facts/ArXiv_(identifier)/H6EtgnBe">arXiv</a>:<a href="https://arxiv.org/abs/1408.3910">1408.3910</a>. <a href="/facts/Bibcode_(identifier)/9HtdQSGB">Bibcode</a>:<a href="https://ui.adsabs.harvard.edu/abs/2015PhRvB..91j4303A">2015PhRvB..91j4303A</a>. <a href="/facts/Doi_(identifier)/muM9Etpq">doi</a>:<a href="https://doi.org/10.1103%2FPhysRevB.91.104303">10.1103/PhysRevB.91.104303</a>. <a href="/facts/S2CID_(identifier)/ldJsHa2Y">S2CID</a> <a href="https://api.semanticscholar.org/CorpusID:40312624">40312624</a>.</li>
<li>Ansari, Mohammad H.; Nazarov, Yuli V. (2015). "Exact correspondence between Rényi entropy flows and physical flows". Physical Review B. 91 (17): 174307. <a href="/facts/ArXiv_(identifier)/H6EtgnBe">arXiv</a>:<a href="https://arxiv.org/abs/1502.08020">1502.08020</a>. <a href="/facts/Bibcode_(identifier)/9HtdQSGB">Bibcode</a>:<a href="https://ui.adsabs.harvard.edu/abs/2015PhRvB..91q4307A">2015PhRvB..91q4307A</a>. <a href="/facts/Doi_(identifier)/muM9Etpq">doi</a>:<a href="https://doi.org/10.1103%2FPhysRevB.91.174307">10.1103/PhysRevB.91.174307</a>. <a href="/facts/S2CID_(identifier)/ldJsHa2Y">S2CID</a> <a href="https://api.semanticscholar.org/CorpusID:36847902">36847902</a>.</li>
<li>Soklakov, A. N. (2020). <a href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7517462">"Economics of Disagreement—Financial Intuition for the Rényi Divergence"</a>. Entropy. 22 (8): 860. <a href="/facts/ArXiv_(identifier)/H6EtgnBe">arXiv</a>:<a href="https://arxiv.org/abs/1811.08308">1811.08308</a>. <a href="/facts/Bibcode_(identifier)/9HtdQSGB">Bibcode</a>:<a href="https://ui.adsabs.harvard.edu/abs/2020Entrp..22..860S">2020Entrp..22..860S</a>. <a href="/facts/Doi_(identifier)/muM9Etpq">doi</a>:<a href="https://doi.org/10.3390%2Fe22080860">10.3390/e22080860</a>. <a href="/facts/PMC_(identifier)/dX1zMt71">PMC</a> <a href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7517462">7517462</a>. <a href="/facts/PMID_(identifier)/JlHAvMHt">PMID</a> <a href="https://pubmed.ncbi.nlm.nih.gov/33286632">33286632</a>.</li>
<li>Ansari, Mohammad H.; van Steensel, Alwin; Nazarov, Yuli V. (2019). <a href="https://doi.org/10.3390%2Fe21090854">"Entropy Production in Quantum is Different"</a>. Entropy. 21 (9): 854. <a href="/facts/ArXiv_(identifier)/H6EtgnBe">arXiv</a>:<a href="https://arxiv.org/abs/1907.09241">1907.09241</a>. <a href="/facts/Doi_(identifier)/muM9Etpq">doi</a>:<a href="https://doi.org/10.3390%2Fe21090854">10.3390/e21090854</a>. <a href="/facts/S2CID_(identifier)/ldJsHa2Y">S2CID</a> <a href="https://api.semanticscholar.org/CorpusID:198148019">198148019</a>.</li>
<li>Rioul, Olivier (2021). <a href="https://hal.telecom-paris.fr/hal-02287963/file/201811rioul.pdf.pdf">"This is IT: A Primer on Shannon's Entropy and Information"</a> (PDF). Information Theory. Progress in Mathematical Physics. Vol. 78. Birkhäuser. pp. 49–86. <a href="/facts/Doi_(identifier)/muM9Etpq">doi</a>:<a href="https://doi.org/10.1007%2F978-3-030-81480-9_2">10.1007/978-3-030-81480-9_2</a>. <a href="/facts/ISBN_(identifier)/15AdSPa9">ISBN</a> 978-3-030-81479-3. <a href="/facts/S2CID_(identifier)/ldJsHa2Y">S2CID</a> <a href="https://api.semanticscholar.org/CorpusID:204783328">204783328</a>.</li></ul>

<h2 id="references">References</h2>

<ol>
<li id="fn:1">Rényi (1961) - Rényi, Alfréd (1961). "On measures of information and entropy" (PDF). Proceedings of the fourth Berkeley Symposium on Mathematics, Statistics and Probability 1960. pp. 547–561. <a href="http://digitalassets.lib.berkeley.edu/math/ucb/text/math_s4_v1_article-27.pdf" target="_blank">http://digitalassets.lib.berkeley.edu/math/ucb/text/math_s4_v1_article-27.pdf</a> <a href="#fnref:1" class="footnote-back-ref">↩</a></li>
<li id="fn:2">Rioul (2021) - Rioul, Olivier (2021). "This is IT: A Primer on Shannon's Entropy and Information" (PDF). Information Theory. Progress in Mathematical Physics. Vol. 78. Birkhäuser. pp. 49–86. doi:10.1007/978-3-030-81480-9_2. ISBN 978-3-030-81479-3. S2CID 204783328. <a href="https://hal.telecom-paris.fr/hal-02287963/file/201811rioul.pdf.pdf" target="_blank">https://hal.telecom-paris.fr/hal-02287963/file/201811rioul.pdf.pdf</a> <a href="#fnref:2" class="footnote-back-ref">↩</a></li>
<li id="fn:3">Barros, Vanessa; Rousseau, Jérôme (2021-06-01). "Shortest Distance Between Multiple Orbits and Generalized Fractal Dimensions". Annales Henri Poincaré. 22 (6): 1853–1885. arXiv:1912.07516. Bibcode:2021AnHP...22.1853B. doi:10.1007/s00023-021-01039-y. ISSN 1424-0661. S2CID 209376774. <a href="https://doi.org/10.1007/s00023-021-01039-y" target="_blank">https://doi.org/10.1007/s00023-021-01039-y</a> <a href="#fnref:3" class="footnote-back-ref">↩</a></li>
<li id="fn:4">Franchini, Its & Korepin (2008) - Franchini, F.; Its, A. R.; Korepin, V. E. (2008). "Rényi entropy as a measure of entanglement in quantum spin chain". Journal of Physics A: Mathematical and Theoretical. 41 (25302): 025302. arXiv:0707.2534. Bibcode:2008JPhA...41b5302F. doi:10.1088/1751-8113/41/2/025302. S2CID 119672750. <a href="https://arxiv.org/abs/0707.2534" target="_blank">https://arxiv.org/abs/0707.2534</a> <a href="#fnref:4" class="footnote-back-ref">↩</a></li>
<li id="fn:5">Its & Korepin (2010) - Its, A. R.; Korepin, V. E. (2010). "Generalized entropy of the Heisenberg spin chain". Theoretical and Mathematical Physics. 164 (3): 1136–1139. Bibcode:2010TMP...164.1136I. doi:10.1007/s11232-010-0091-6. S2CID 119525704. <a href="https://ui.adsabs.harvard.edu/abs/2010TMP...164.1136I" target="_blank">https://ui.adsabs.harvard.edu/abs/2010TMP...164.1136I</a> <a href="#fnref:5" class="footnote-back-ref">↩</a></li>
<li id="fn:6">Rényi (1961) - Rényi, Alfréd (1961). "On measures of information and entropy" (PDF). Proceedings of the fourth Berkeley Symposium on Mathematics, Statistics and Probability 1960. pp. 547–561. <a href="http://digitalassets.lib.berkeley.edu/math/ucb/text/math_s4_v1_article-27.pdf" target="_blank">http://digitalassets.lib.berkeley.edu/math/ucb/text/math_s4_v1_article-27.pdf</a> <a href="#fnref:6" class="footnote-back-ref">↩</a></li>
<li id="fn:7">RFC 4086, page 6 <a href="http://tools.ietf.org/html/rfc4086.html#page-6" target="_blank">http://tools.ietf.org/html/rfc4086.html#page-6</a> <a href="#fnref:7" class="footnote-back-ref">↩</a></li>
<li id="fn:8">Bromiley, Thacker & Bouhova-Thacker (2004) - Bromiley, P.A.; Thacker, N.A.; Bouhova-Thacker, E. (2004), Shannon Entropy, Rényi Entropy, and Information, CiteSeerX 10.1.1.330.9856 <a href="https://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.330.9856" target="_blank">https://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.330.9856</a> <a href="#fnref:8" class="footnote-back-ref">↩</a></li>
<li id="fn:9">Beck & Schlögl (1993) - Beck, Christian; Schlögl, Friedrich (1993). Thermodynamics of chaotic systems: an introduction. Cambridge University Press. ISBN 0521433673. <a href="#fnref:9" class="footnote-back-ref">↩</a></li>
<li id="fn:10">H
 
 
 1
 
 
 ≥
 
 
 H
 
 
 2
 
 
 
 
 
 {\displaystyle \textstyle \mathrm {H} _{1}\geq \mathrm {H} _{2}}
 
 holds because ⁠
 
 
 
 
 
 ∑
 
 i
 =
 1
 
 
 M
 
 
 
 
 p
 
 i
 
 
 log
 ⁡
 
 p
 
 i
 
 
 
 ≤
 log
 ⁡
 
 ∑
 
 i
 =
 1
 
 
 M
 
 
 
 
 p
 
 i
 
 
 2
 
 
 
 
 
 
 {\displaystyle \textstyle \sum \limits _{i=1}^{M}{p_{i}\log p_{i}}\leq \log \sum \limits _{i=1}^{M}{p_{i}^{2}}}
 
⁠. <a href="#fnref:10" class="footnote-back-ref">↩</a></li>
<li id="fn:11">H
 
 
 ∞
 
 
 ≤
 
 
 H
 
 
 2
 
 
 
 
 {\displaystyle \mathrm {H} _{\infty }\leq \mathrm {H} _{2}}
 
 holds because ⁠
 
 
 
 
 log
 ⁡
 
 ∑
 
 i
 =
 1
 
 
 n
 
 
 
 
 p
 
 i
 
 
 2
 
 
 
 ≤
 log
 ⁡
 
 sup
 
 i
 
 
 
 p
 
 i
 
 
 
 (
 
 
 ∑
 
 i
 =
 1
 
 
 n
 
 
 
 
 p
 
 i
 
 
 
 
 )
 
 =
 log
 ⁡
 
 sup
 
 i
 
 
 
 p
 
 i
 
 
 
 
 
 {\displaystyle \textstyle \log \sum \limits _{i=1}^{n}{p_{i}^{2}}\leq \log \sup _{i}p_{i}\left({\sum \limits _{i=1}^{n}{p_{i}}}\right)=\log \sup _{i}p_{i}}
 
⁠. <a href="#fnref:11" class="footnote-back-ref">↩</a></li>
<li id="fn:12">H
 
 
 2
 
 
 ≤
 2
 
 
 H
 
 
 ∞
 
 
 
 
 {\displaystyle \mathrm {H} _{2}\leq 2\mathrm {H} _{\infty }}
 
 holds because 
 
 
 
 
 log
 ⁡
 
 ∑
 
 i
 =
 1
 
 
 n
 
 
 
 
 p
 
 i
 
 
 2
 
 
 
 ≥
 log
 ⁡
 
 sup
 
 i
 
 
 
 p
 
 i
 
 
 2
 
 
 =
 2
 log
 ⁡
 
 sup
 
 i
 
 
 
 p
 
 i
 
 
 
 
 
 {\displaystyle \textstyle \log \sum \limits _{i=1}^{n}{p_{i}^{2}}\geq \log \sup _{i}p_{i}^{2}=2\log \sup _{i}p_{i}}
 <a href="#fnref:12" class="footnote-back-ref">↩</a></li>
<li id="fn:13">Devroye, Luc; Györfi, Laszlo; Lugosi, Gabor (1996-04-04). A Probabilistic Theory of Pattern Recognition (Corrected ed.). New York, NY: Springer. ISBN 978-0-387-94618-4. <a href="978-0-387-94618-4" target="_blank">978-0-387-94618-4</a> <a href="#fnref:13" class="footnote-back-ref">↩</a></li>
<li id="fn:14">Van Erven, Tim; Harremoës, Peter (2014). "Rényi Divergence and Kullback–Leibler Divergence". IEEE Transactions on Information Theory. 60 (7): 3797–3820. arXiv:1206.2459. doi:10.1109/TIT.2014.2320500. S2CID 17522805. <a href="/wiki/ArXiv_(identifier)" target="_blank">/wiki/ArXiv_(identifier)</a> <a href="#fnref:14" class="footnote-back-ref">↩</a></li>
<li id="fn:15">Van Erven, Tim; Harremoës, Peter (2014). "Rényi Divergence and Kullback–Leibler Divergence". IEEE Transactions on Information Theory. 60 (7): 3797–3820. arXiv:1206.2459. doi:10.1109/TIT.2014.2320500. S2CID 17522805. <a href="/wiki/ArXiv_(identifier)" target="_blank">/wiki/ArXiv_(identifier)</a> <a href="#fnref:15" class="footnote-back-ref">↩</a></li>
<li id="fn:16">Rényi (1961) - Rényi, Alfréd (1961). "On measures of information and entropy" (PDF). Proceedings of the fourth Berkeley Symposium on Mathematics, Statistics and Probability 1960. pp. 547–561. <a href="http://digitalassets.lib.berkeley.edu/math/ucb/text/math_s4_v1_article-27.pdf" target="_blank">http://digitalassets.lib.berkeley.edu/math/ucb/text/math_s4_v1_article-27.pdf</a> <a href="#fnref:16" class="footnote-back-ref">↩</a></li>
<li id="fn:17">Soklakov (2018) - Soklakov, A. N. (2020). "Economics of Disagreement—Financial Intuition for the Rényi Divergence". Entropy. 22 (8): 860. arXiv:1811.08308. Bibcode:2020Entrp..22..860S. doi:10.3390/e22080860. PMC 7517462. PMID 33286632. <a href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7517462" target="_blank">https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7517462</a> <a href="#fnref:17" class="footnote-back-ref">↩</a></li>
<li id="fn:18">Soklakov (2018) - Soklakov, A. N. (2020). "Economics of Disagreement—Financial Intuition for the Rényi Divergence". Entropy. 22 (8): 860. arXiv:1811.08308. Bibcode:2020Entrp..22..860S. doi:10.3390/e22080860. PMC 7517462. PMID 33286632. <a href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7517462" target="_blank">https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7517462</a> <a href="#fnref:18" class="footnote-back-ref">↩</a></li>
<li id="fn:19">Nielsen & Nock (2011) - Nielsen, Frank; Nock, Richard (2011). "On Rényi and Tsallis entropies and divergences for exponential families". Journal of Physics A. 45 (3): 032003. arXiv:1105.3259. Bibcode:2012JPhA...45c2003N. doi:10.1088/1751-8113/45/3/032003. S2CID 8653096. <a href="https://arxiv.org/abs/1105.3259" target="_blank">https://arxiv.org/abs/1105.3259</a> <a href="#fnref:19" class="footnote-back-ref">↩</a></li>
</ol>

Rényi entropy open-in-new

Rényi entropy