Transfer entropy

Transfer entropy is a <a href="/facts/Non-parametric_statistics/svTDJHTb">non-parametric statistic</a> measuring the amount of directed (time-asymmetric) transfer of <a href="/facts/Information/3LdrzXtJ">information</a> between two <a href="/facts/Random_process/Ng1n1dnB">random processes</a>. Transfer entropy from a process X to another process Y is the amount of uncertainty reduced in future values of Y by knowing the past values of X given past values of Y. More specifically, if 
 
 
 
 
 X
 
 t
 
 
 
 
 {\displaystyle X_{t}}
 
 and 
 
 
 
 
 Y
 
 t
 
 
 
 
 {\displaystyle Y_{t}}
 
 for 
 
 
 
 t
 ∈
 
 N
 
 
 
 {\displaystyle t\in \mathbb {N} }
 
 denote two random processes and the amount of information is measured using <a href="/facts/Shannon%2527s_entropy/NLg4NLvt">Shannon's entropy</a>, the transfer entropy can be written as:

T
          
            X
            →
            Y
          
        
        =
        H
        
          (
          
            
              Y
              
                t
              
            
            ∣
            
              Y
              
                t
                −
                1
                :
                t
                −
                L
              
            
          
          )
        
        −
        H
        
          (
          
            
              Y
              
                t
              
            
            ∣
            
              Y
              
                t
                −
                1
                :
                t
                −
                L
              
            
            ,
            
              X
              
                t
                −
                1
                :
                t
                −
                L
              
            
          
          )
        
        ,
      
    
    {\displaystyle T_{X\rightarrow Y}=H\left(Y_{t}\mid Y_{t-1:t-L}\right)-H\left(Y_{t}\mid Y_{t-1:t-L},X_{t-1:t-L}\right),}

where H(X) is Shannon's entropy of X. The above definition of transfer entropy has been extended by other types of <a href="/facts/Entropy_(information_theory)/NLg4NLvt">entropy</a> measures such as <a href="/facts/R%25C3%25A9nyi_entropy/SlrvqKNv">Rényi entropy</a>.
Transfer entropy is <a href="/facts/Conditional_mutual_information/MRsfVyhC">conditional mutual information</a>, with the history of the influenced variable 
 
 
 
 
 Y
 
 t
 −
 1
 :
 t
 −
 L
 
 
 
 
 {\displaystyle Y_{t-1:t-L}}
 
 in the condition:

T
          
            X
            →
            Y
          
        
        =
        I
        (
        
          Y
          
            t
          
        
        ;
        
          X
          
            t
            −
            1
            :
            t
            −
            L
          
        
        ∣
        
          Y
          
            t
            −
            1
            :
            t
            −
            L
          
        
        )
        .
      
    
    {\displaystyle T_{X\rightarrow Y}=I(Y_{t};X_{t-1:t-L}\mid Y_{t-1:t-L}).}

Transfer entropy reduces to <a href="/facts/Granger_causality/LAEvgA1A">Granger causality</a> for <a href="/facts/Autoregressive_model/LpboAQ0W">vector auto-regressive processes</a>. Hence, it is advantageous when the model assumption of Granger causality doesn't hold, for example, analysis of <a href="/facts/Non-linear_regression/8mMuJSe8">non-linear signals</a>. However, it usually requires more samples for accurate estimation.
The probabilities in the entropy formula can be estimated using different approaches (binning, nearest neighbors) or, in order to reduce complexity, using a non-uniform embedding.
While it was originally defined for <a href="/facts/Bivariate_analysis/skgknzMV">bivariate analysis</a>, transfer entropy has been extended to <a href="/facts/Multivariate_analysis/tDuNcWrL">multivariate</a> forms, either conditioning on other potential source variables or considering transfer from a collection of sources, although these forms require more samples again.
Transfer entropy has been used for estimation of <a href="/facts/Functional_connectivity/yq9Qk2Yg">functional connectivity</a> of <a href="/facts/Neurons/G7CyTVdH">neurons</a>, <a href="/facts/Social_influence/jR84Vghj">social influence</a> in <a href="/facts/Social_networks/3NkYaPIr">social networks</a> and <a href="/facts/Causal_inference/0D0zNs9b">statistical causality</a> between armed conflict events.
Transfer entropy is a finite version of the <a href="/facts/Directed_information/sYpdcvSA">directed information</a> which was defined in 1990 by <a href="/facts/James_Massey/qyfqKnQH">James Massey</a> as

I
 (
 
 X
 
 n
 
 
 →
 
 Y
 
 n
 
 
 )
 =
 
 ∑
 
 i
 =
 1
 
 
 n
 
 
 I
 (
 
 X
 
 i
 
 
 ;
 
 Y
 
 i
 
 
 
 |
 
 
 Y
 
 i
 −
 1
 
 
 )
 
 
 {\displaystyle I(X^{n}\to Y^{n})=\sum _{i=1}^{n}I(X^{i};Y_{i}|Y^{i-1})}
 
, where 
 
 
 
 
 X
 
 n
 
 
 
 
 {\displaystyle X^{n}}
 
 denotes the vector 
 
 
 
 
 X
 
 1
 
 
 ,
 
 X
 
 2
 
 
 ,
 .
 .
 .
 ,
 
 X
 
 n
 
 
 
 
 {\displaystyle X_{1},X_{2},...,X_{n}}
 
 and 
 
 
 
 
 Y
 
 n
 
 
 
 
 {\displaystyle Y^{n}}
 
 denotes 
 
 
 
 
 Y
 
 1
 
 
 ,
 
 Y
 
 2
 
 
 ,
 .
 .
 .
 ,
 
 Y
 
 n
 
 
 
 
 {\displaystyle Y_{1},Y_{2},...,Y_{n}}
 
. The <a href="/facts/Directed_information/sYpdcvSA">directed information</a> places an important role in characterizing the fundamental limits (<a href="/facts/Channel_capacity/55wJYsS6">channel capacity</a>) of communication channels with or without feedback and <a href="/facts/Gambling/Hgw4a9DH">gambling</a> with causal side information.

Transfer entropy open-in-new

Transfer entropy