Menu
Home
People
Places
Arts
History
Plants & Animals
Science
Life & Culture
Technology
Reference.org
Dual total correlation
open-in-new
<h2 id="definition">Definition</h2> <p>For a set of <i>n</i> <a href="/facts/Random_variable/TwTBXnLT">random variables</a> { X 1 , … , X n } {\displaystyle \{X_{1},\ldots ,X_{n}\}} , the dual total correlation D ( X 1 , … , X n ) {\displaystyle D(X_{1},\ldots ,X_{n})} is given by </p> D ( X 1 , … , X n ) = H ( X 1 , … , X n ) − ∑ i = 1 n H ( X i ∣ X 1 , … , X i − 1 , X i + 1 , … , X n ) , {\displaystyle D(X_{1},\ldots ,X_{n})=H\left(X_{1},\ldots ,X_{n}\right)-\sum _{i=1}^{n}H\left(X_{i}\mid X_{1},\ldots ,X_{i-1},X_{i+1},\ldots ,X_{n}\right),} <p>where H ( X 1 , … , X n ) {\displaystyle H(X_{1},\ldots ,X_{n})} is the <a href="/facts/Joint_entropy/V4WwuX6P">joint entropy</a> of the variable set { X 1 , … , X n } {\displaystyle \{X_{1},\ldots ,X_{n}\}} and H ( X i ∣ ⋯ ) {\displaystyle H(X_{i}\mid \cdots )} is the <a href="/facts/Conditional_entropy/gHzPezGB">conditional entropy</a> of variable X i {\displaystyle X_{i}} , given the rest. </p> <h2 id="normalized">Normalized</h2> <p>The dual total correlation normalized between [0,1] is simply the dual total correlation divided by its maximum value H ( X 1 , … , X n ) {\displaystyle H(X_{1},\ldots ,X_{n})} , </p> N D ( X 1 , … , X n ) = D ( X 1 , … , X n ) H ( X 1 , … , X n ) . {\displaystyle ND(X_{1},\ldots ,X_{n})={\frac {D(X_{1},\ldots ,X_{n})}{H(X_{1},\ldots ,X_{n})}}.} <h2 id="relationship-with-total-correlation">Relationship with Total Correlation</h2> <p>Dual total correlation is non-negative and bounded above by the joint entropy H ( X 1 , … , X n ) {\displaystyle H(X_{1},\ldots ,X_{n})} . </p> 0 ≤ D ( X 1 , … , X n ) ≤ H ( X 1 , … , X n ) . {\displaystyle 0\leq D(X_{1},\ldots ,X_{n})\leq H(X_{1},\ldots ,X_{n}).} <p>Secondly, Dual total correlation has a close relationship with total correlation, C ( X 1 , … , X n ) {\displaystyle C(X_{1},\ldots ,X_{n})} , and can be written in terms of differences between the total correlation of the whole, and all subsets of size N − 1 {\displaystyle N-1} :<a class="footnote-ref" id="fnref:7" href="#fn:7"><sup>7</sup></a> </p> D ( X ) = ( N − 1 ) C ( X ) − ∑ i = 1 N C ( X − i ) {\displaystyle D({\textbf {X}})=(N-1)C({\textbf {X}})-\sum _{i=1}^{N}C({\textbf {X}}^{-i})} <p>where X = { X 1 , … , X n } {\displaystyle {\textbf {X}}=\{X_{1},\ldots ,X_{n}\}} and X − i = { X 1 , … , X i − 1 , X i + 1 , … , X n } {\displaystyle {\textbf {X}}^{-i}=\{X_{1},\ldots ,X_{i-1},X_{i+1},\ldots ,X_{n}\}} </p><p>Furthermore, the total correlation and dual total correlation are related by the following bounds: </p> C ( X 1 , … , X n ) n − 1 ≤ D ( X 1 , … , X n ) ≤ ( n − 1 ) C ( X 1 , … , X n ) . {\displaystyle {\frac {C(X_{1},\ldots ,X_{n})}{n-1}}\leq D(X_{1},\ldots ,X_{n})\leq (n-1)\;C(X_{1},\ldots ,X_{n}).} <p>Finally, the difference between the total correlation and the dual total correlation defines a novel measure of higher-order information-sharing: the O-information:<a class="footnote-ref" id="fnref:8" href="#fn:8"><sup>8</sup></a> </p> Ω ( X ) = C ( X ) − D ( X ) {\displaystyle \Omega ({\textbf {X}})=C({\textbf {X}})-D({\textbf {X}})} . <p>The O-information (first introduced as the "enigmatic information" by James and Crutchfield<a class="footnote-ref" id="fnref:9" href="#fn:9"><sup>9</sup></a> is a signed measure that quantifies the extent to which the information in a multivariate random variable is dominated by synergistic interactions (in which case Ω ( X ) < 0 {\displaystyle \Omega ({\textbf {X}})<0} ) or redundant interactions (in which case Ω ( X ) > 0 {\displaystyle \Omega ({\textbf {X}})>0} . </p> <h2 id="history">History</h2> <p>Han (1978) originally defined the dual total correlation as, </p> D ( X 1 , … , X n ) ≡ [ ∑ i = 1 n H ( X 1 , … , X i − 1 , X i + 1 , … , X n ) ] − ( n − 1 ) H ( X 1 , … , X n ) . {\displaystyle {\begin{aligned}&D(X_{1},\ldots ,X_{n})\\[10pt]\equiv {}&\left[\sum _{i=1}^{n}H(X_{1},\ldots ,X_{i-1},X_{i+1},\ldots ,X_{n})\right]-(n-1)\;H(X_{1},\ldots ,X_{n})\;.\end{aligned}}} <p>However Abdallah and Plumbley (2010) showed its equivalence to the easier-to-understand form of the joint entropy minus the sum of conditional entropies via the following: </p> D ( X 1 , … , X n ) ≡ [ ∑ i = 1 n H ( X 1 , … , X i − 1 , X i + 1 , … , X n ) ] − ( n − 1 ) H ( X 1 , … , X n ) = [ ∑ i = 1 n H ( X 1 , … , X i − 1 , X i + 1 , … , X n ) ] + ( 1 − n ) H ( X 1 , … , X n ) = H ( X 1 , … , X n ) + [ ∑ i = 1 n H ( X 1 , … , X i − 1 , X i + 1 , … , X n ) − H ( X 1 , … , X n ) ] = H ( X 1 , … , X n ) − ∑ i = 1 n H ( X i ∣ X 1 , … , X i − 1 , X i + 1 , … , X n ) . {\displaystyle {\begin{aligned}&D(X_{1},\ldots ,X_{n})\\[10pt]\equiv {}&\left[\sum _{i=1}^{n}H(X_{1},\ldots ,X_{i-1},X_{i+1},\ldots ,X_{n})\right]-(n-1)\;H(X_{1},\ldots ,X_{n})\\={}&\left[\sum _{i=1}^{n}H(X_{1},\ldots ,X_{i-1},X_{i+1},\ldots ,X_{n})\right]+(1-n)\;H(X_{1},\ldots ,X_{n})\\={}&H(X_{1},\ldots ,X_{n})+\left[\sum _{i=1}^{n}H(X_{1},\ldots ,X_{i-1},X_{i+1},\ldots ,X_{n})-H(X_{1},\ldots ,X_{n})\right]\\={}&H\left(X_{1},\ldots ,X_{n}\right)-\sum _{i=1}^{n}H\left(X_{i}\mid X_{1},\ldots ,X_{i-1},X_{i+1},\ldots ,X_{n}\right)\;.\end{aligned}}} <h2 id="see-also">See also</h2> <ul><li><a href="/facts/Interaction_information/8FdPJcQX">Interaction information</a></li> <li><a href="/facts/Mutual_information/HIUvsjvV">Mutual information</a></li> <li><a href="/facts/Total_correlation/w8w3W59S">Total correlation</a></li></ul> <h2 id="bibliography">Bibliography</h2> <h3>Footnotes</h3> <h3>References</h3> <ul><li>Fujishige, Satoru (1978). <a href="https://doi.org/10.1016%2FS0019-9958%2878%2991063-X">"Polymatroidal dependence structure of a set of random variables"</a>. <i>Information and Control</i>. 39: 55–72. <a href="/facts/Doi_(identifier)/muM9Etpq">doi</a>:<a href="https://doi.org/10.1016%2FS0019-9958%2878%2991063-X">10.1016/S0019-9958(78)91063-X</a>.</li> <li>Varley, Thomas; Pope, Maria; Faskowitz, Joshua; Sporns, Olaf (2023). <a href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10125999">"Multivariate information theory uncovers synergistic subsystems of the human cerebral cortex"</a>. <i>Communications Biology</i>. 6: 451. <a href="/facts/Doi_(identifier)/muM9Etpq">doi</a>:<a href="https://doi.org/10.1038%2Fs42003-023-04843-w">10.1038/s42003-023-04843-w</a>. <a href="/facts/PMC_(identifier)/dX1zMt71">PMC</a> <a href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10125999">10125999</a>. <a href="/facts/PMID_(identifier)/JlHAvMHt">PMID</a> <a href="https://pubmed.ncbi.nlm.nih.gov/37095282">37095282</a>.</li></ul> <h2 id="references">References</h2> <ol> <li id="fn:1"><p>Han, Te Sun (1978). "Nonnegative entropy measures of multivariate symmetric correlations". Information and Control. 36 (2): 133–156. doi:10.1016/S0019-9958(78)90275-9. <a href="https://doi.org/10.1016%2FS0019-9958%2878%2990275-9" target="_blank">https://doi.org/10.1016%2FS0019-9958%2878%2990275-9</a> <a href="#fnref:1" class="footnote-back-ref">↩</a></p></li> <li id="fn:2"><p>Dubnov, Shlomo (2006). "Spectral Anticipations". Computer Music Journal. 30 (2): 63–83. doi:10.1162/comj.2006.30.2.63. S2CID 2202704. <a href="/wiki/Doi_(identifier)" target="_blank">/wiki/Doi_(identifier)</a> <a href="#fnref:2" class="footnote-back-ref">↩</a></p></li> <li id="fn:3"><p>Nihat Ay, E. Olbrich, N. Bertschinger (2001). A unifying framework for complexity measures of finite systems. European Conference on Complex Systems. pdf. <a href="http://www.cabdyn.ox.ac.uk/complexity_PDFs/ECCS06/Conference_Proceedings/PDF/p202.pdf" target="_blank">http://www.cabdyn.ox.ac.uk/complexity_PDFs/ECCS06/Conference_Proceedings/PDF/p202.pdf</a> <a href="#fnref:3" class="footnote-back-ref">↩</a></p></li> <li id="fn:4"><p>Olbrich, E.; Bertschinger, N.; Ay, N.; Jost, J. (2008). "How should complexity scale with system size?". The European Physical Journal B. 63 (3): 407–415. Bibcode:2008EPJB...63..407O. doi:10.1140/epjb/e2008-00134-9. S2CID 120391127. <a href="https://doi.org/10.1140%2Fepjb%2Fe2008-00134-9" target="_blank">https://doi.org/10.1140%2Fepjb%2Fe2008-00134-9</a> <a href="#fnref:4" class="footnote-back-ref">↩</a></p></li> <li id="fn:5"><p>Abdallah, Samer A.; Plumbley, Mark D. (2010). "A measure of statistical complexity based on predictive information". arXiv:1012.1890v1 [math.ST]. <a href="/wiki/ArXiv_(identifier)" target="_blank">/wiki/ArXiv_(identifier)</a> <a href="#fnref:5" class="footnote-back-ref">↩</a></p></li> <li id="fn:6"><p>Nihat Ay, E. Olbrich, N. Bertschinger (2001). A unifying framework for complexity measures of finite systems. European Conference on Complex Systems. pdf. <a href="http://www.cabdyn.ox.ac.uk/complexity_PDFs/ECCS06/Conference_Proceedings/PDF/p202.pdf" target="_blank">http://www.cabdyn.ox.ac.uk/complexity_PDFs/ECCS06/Conference_Proceedings/PDF/p202.pdf</a> <a href="#fnref:6" class="footnote-back-ref">↩</a></p></li> <li id="fn:7"><p>Varley, Thomas F.; Pope, Maria; Faskowitz, Joshua; Sporns, Olaf (24 April 2023). "Multivariate information theory uncovers synergistic subsystems of the human cerebral cortex". Communications Biology. 6 (1): 451. doi:10.1038/s42003-023-04843-w. PMC 10125999. PMID 37095282. <a href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10125999" target="_blank">https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10125999</a> <a href="#fnref:7" class="footnote-back-ref">↩</a></p></li> <li id="fn:8"><p>Rosas, Fernando E.; Mediano, Pedro A. M.; Gastpar, Michael; Jensen, Henrik J. (13 September 2019). "Quantifying high-order interdependencies via multivariate extensions of the mutual information". Physical Review E. 100 (3): 032305. arXiv:1902.11239. Bibcode:2019PhRvE.100c2305R. doi:10.1103/PhysRevE.100.032305. PMID 31640038. <a href="/wiki/ArXiv_(identifier)" target="_blank">/wiki/ArXiv_(identifier)</a> <a href="#fnref:8" class="footnote-back-ref">↩</a></p></li> <li id="fn:9"><p>James, Ryan G.; Ellison, Christopher J.; Crutchfield, James P. (1 September 2011). "Anatomy of a bit: Information in a time series observation". Chaos: An Interdisciplinary Journal of Nonlinear Science. 21 (3): 037109. arXiv:1105.2988. Bibcode:2011Chaos..21c7109J. doi:10.1063/1.3637494. PMID 21974672. <a href="/wiki/ArXiv_(identifier)" target="_blank">/wiki/ArXiv_(identifier)</a> <a href="#fnref:9" class="footnote-back-ref">↩</a></p></li> </ol>