Menu
Home
People
Places
Arts
History
Plants & Animals
Science
Life & Culture
Technology
Reference.org
FastICA
open-in-new
<h2 id="algorithm">Algorithm</h2> <h3><i>Prewhitening</i> the data</h3> <p>Let the X := ( x i j ) ∈ R N × M {\displaystyle \mathbf {X} :=(x_{ij})\in \mathbb {R} ^{N\times M}} denote the input data matrix, M {\displaystyle M} the number of columns corresponding with the number of samples of mixed signals and N {\displaystyle N} the number of rows corresponding with the number of independent source signals. The input data matrix X {\displaystyle \mathbf {X} } must be <i>prewhitened</i>, or centered and whitened, before applying the FastICA algorithm to it. </p> <ul><li>Centering the data entails demeaning each component of the input data X {\displaystyle \mathbf {X} } , that is,</li></ul> x i j ← x i j − 1 M ∑ j ′ x i j ′ {\displaystyle x_{ij}\leftarrow x_{ij}-{\frac {1}{M}}\sum _{j^{\prime }}x_{ij^{\prime }}} for each i = 1 , … , N {\displaystyle i=1,\ldots ,N} and j = 1 , … , M {\displaystyle j=1,\ldots ,M} . After centering, each row of X {\displaystyle \mathbf {X} } has an <a href="/facts/Expected_value/1XV0JKL8">expected value</a> of 0 {\displaystyle 0} . <ul><li><i>Whitening</i> the data requires a <a href="/facts/Linear_transformation/Q1PBKNVV">linear transformation</a> L : R N × M → R N × M {\displaystyle \mathbf {L} :\mathbb {R} ^{N\times M}\to \mathbb {R} ^{N\times M}} of the centered data so that the components of L ( X ) {\displaystyle \mathbf {L} (\mathbf {X} )} are uncorrelated and have variance one. More precisely, if X {\displaystyle \mathbf {X} } is a centered data matrix, the covariance of L x := L ( X ) {\displaystyle \mathbf {L} _{\mathbf {x} }:=\mathbf {L} (\mathbf {X} )} is the ( N × N ) {\displaystyle (N\times N)} -dimensional identity matrix, that is,</li></ul> E { L x L x T } = I N {\displaystyle \mathrm {E} \left\{\mathbf {L} _{\mathbf {x} }\mathbf {L} _{\mathbf {x} }^{T}\right\}=\mathbf {I} _{N}} A common method for whitening is by performing an <a href="/facts/Eigenvalue_decomposition/a2nfF7hJ">eigenvalue decomposition</a> on the <a href="/facts/Covariance_matrix/kLhgjHC6">covariance matrix</a> of the centered data X {\displaystyle \mathbf {X} } , E { X X T } = E D E T {\displaystyle E\left\{\mathbf {X} \mathbf {X} ^{T}\right\}=\mathbf {E} \mathbf {D} \mathbf {E} ^{T}} , where E {\displaystyle \mathbf {E} } is the matrix of eigenvectors and D {\displaystyle \mathbf {D} } is the diagonal matrix of eigenvalues. The whitened data matrix is defined thus by X ← D − 1 / 2 E T X . {\displaystyle \mathbf {X} \leftarrow \mathbf {D} ^{-1/2}\mathbf {E} ^{T}\mathbf {X} .} <h3>Single component extraction</h3> <p>The iterative algorithm finds the direction for the weight vector w ∈ R N {\displaystyle \mathbf {w} \in \mathbb {R} ^{N}} that maximizes a measure of non-Gaussianity of the projection w T X {\displaystyle \mathbf {w} ^{T}\mathbf {X} } , with X ∈ R N × M {\displaystyle \mathbf {X} \in \mathbb {R} ^{N\times M}} denoting a prewhitened data matrix as described above. Note that w {\displaystyle \mathbf {w} } is a column vector. To measure non-Gaussianity, FastICA relies on a nonquadratic <a href="/facts/Nonlinear/4aYItALC">nonlinear</a> <a href="/facts/Function_(mathematics)/CdReEKCh">function</a> f ( u ) {\displaystyle f(u)} , its first derivative g ( u ) {\displaystyle g(u)} , and its second derivative g ′ ( u ) {\displaystyle g^{\prime }(u)} . Hyvärinen states that the functions </p> f ( u ) = log cosh ( u ) , g ( u ) = tanh ( u ) , and g ′ ( u ) = 1 − tanh 2 ( u ) , {\displaystyle f(u)=\log \cosh(u),\quad g(u)=\tanh(u),\quad {\text{and}}\quad {g}'(u)=1-\tanh ^{2}(u),} <p>are useful for general purposes, while </p> f ( u ) = − e − u 2 / 2 , g ( u ) = u e − u 2 / 2 , and g ′ ( u ) = ( 1 − u 2 ) e − u 2 / 2 {\displaystyle f(u)=-e^{-u^{2}/2},\quad g(u)=ue^{-u^{2}/2},\quad {\text{and}}\quad {g}'(u)=(1-u^{2})e^{-u^{2}/2}} <p>may be highly robust.<a class="footnote-ref" id="fnref:3" href="#fn:3"><sup>3</sup></a> The steps for extracting the weight vector w {\displaystyle \mathbf {w} } for single component in FastICA are the following: </p> <ol><li>Randomize the initial weight vector w {\displaystyle \mathbf {w} } </li> <li>Let w + ← E { X g ( w T X ) T } − E { g ′ ( w T X ) } w {\displaystyle \mathbf {w} ^{+}\leftarrow E\left\{\mathbf {X} g(\mathbf {w} ^{T}\mathbf {X} )^{T}\right\}-E\left\{g'(\mathbf {w} ^{T}\mathbf {X} )\right\}\mathbf {w} } , where E { . . . } {\displaystyle E\left\{...\right\}} means averaging over all column-vectors of matrix X {\displaystyle \mathbf {X} } </li> <li>Let w ← w + / ‖ w + ‖ {\displaystyle \mathbf {w} \leftarrow \mathbf {w} ^{+}/\|\mathbf {w} ^{+}\|} </li> <li>If not converged, go back to 2</li></ol> <h3>Multiple component extraction</h3> <p>The single unit iterative algorithm estimates only one weight vector which extracts a single component. Estimating additional components that are mutually "independent" requires repeating the algorithm to obtain linearly independent projection vectors - note that the notion of <a href="/facts/Independence_(probability_theory)/NUzQtnUL">independence</a> here refers to maximizing non-Gaussianity in the estimated components. Hyvärinen provides several ways of extracting multiple components with the simplest being the following. Here, 1 M {\displaystyle \mathbf {1_{M}} } is a column vector of 1's of dimension M {\displaystyle M} . </p><p>Algorithm FastICA </p> Input: C {\displaystyle C} Number of desired components Input: X ∈ R N × M {\displaystyle \mathbf {X} \in \mathbb {R} ^{N\times M}} Prewhitened matrix, where each column represents an N {\displaystyle N} -dimensional sample, where C <= N {\displaystyle C<=N} Output: W ∈ R N × C {\displaystyle \mathbf {W} \in \mathbb {R} ^{N\times C}} Un-mixing matrix where each column projects X {\displaystyle \mathbf {X} } onto independent component. Output: S ∈ R C × M {\displaystyle \mathbf {S} \in \mathbb {R} ^{C\times M}} Independent components matrix, with M {\displaystyle M} columns representing a sample with C {\displaystyle C} dimensions. for p in 1 to C: <i> w p ← {\displaystyle \mathbf {w_{p}} \leftarrow } Random vector of length N</i> while w p {\displaystyle \mathbf {w_{p}} } changes w p ← 1 M X g ( w p T X ) T − 1 M g ′ ( w p T X ) 1 M w p {\displaystyle \mathbf {w_{p}} \leftarrow {\frac {1}{M}}\mathbf {X} g(\mathbf {w_{p}} ^{T}\mathbf {X} )^{T}-{\frac {1}{M}}g'(\mathbf {w_{p}} ^{T}\mathbf {X} )\mathbf {1_{M}} \mathbf {w_{p}} } w p ← w p − ∑ j = 1 p − 1 ( w p T w j ) w j {\displaystyle \mathbf {w_{p}} \leftarrow \mathbf {w_{p}} -\sum _{j=1}^{p-1}(\mathbf {w_{p}} ^{T}\mathbf {w_{j}} )\mathbf {w_{j}} } w p ← w p ‖ w p ‖ {\displaystyle \mathbf {w_{p}} \leftarrow {\frac {\mathbf {w_{p}} }{\|\mathbf {w_{p}} \|}}} output W ← [ w 1 , … , w C ] {\displaystyle \mathbf {W} \leftarrow {\begin{bmatrix}\mathbf {w_{1}} ,\dots ,\mathbf {w_{C}} \end{bmatrix}}} output S ← W T X {\displaystyle \mathbf {S} \leftarrow \mathbf {W^{T}} \mathbf {X} } <h2 id="see-also">See also</h2> <ul><li><a href="/facts/Unsupervised_learning/8a6f61MC">Unsupervised learning</a></li> <li><a href="/facts/Machine_learning/e0w0XJTu">Machine learning</a></li> <li>The <a href="/facts/IT%2B%2B/B4gdSCkR">IT++</a> library features a FastICA implementation in <a href="/facts/C%2B%2B/Et0F2qn9">C++</a></li> <li><a href="/facts/Infomax/6JCHoxdF">Infomax</a></li></ul> <h2 id="external-links">External links</h2> <ul><li><a href="https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.FastICA.html">FastICA in Python</a></li> <li><a href="http://www.cis.hut.fi/projects/ica/fastica/">FastICA package for Matlab or Octave</a></li> <li><a href="https://cran.r-project.org/web/packages/fastICA/index.html">fastICA package</a> in <a href="/facts/R_programming_language/LSrkr8K8">R programming language</a></li> <li><a href="http://sourceforge.net/projects/fastica">FastICA in Java</a> on <a href="/facts/SourceForge/lxYAgUAX">SourceForge</a></li> <li><a href="http://rapid-i.com/wiki/index.php?title=Independent_Component_Analysis">FastICA in Java</a> in <a href="/facts/RapidMiner/F7jGMflS">RapidMiner</a>.</li> <li><a href="https://github.com/derekharrison/FastICA">FastICA in Matlab</a></li> <li><a href="http://mdp-toolkit.sourceforge.net/index.html">FastICA in MDP</a></li> <li><a href="https://juliastats.org/MultivariateStats.jl/stable/ica/">FastICA in Julia</a></li></ul> <h2 id="references">References</h2> <ol> <li id="fn:1"><p>Hyvärinen, A.; Oja, E. (2000). "Independent component analysis: Algorithms and applications" (PDF). Neural Networks. 13 (4–5): 411–430. CiteSeerX 10.1.1.79.7003. doi:10.1016/S0893-6080(00)00026-5. PMID 10946390. <a href="http://www.cs.helsinki.fi/u/ahyvarin/papers/NN00new.pdf" target="_blank">http://www.cs.helsinki.fi/u/ahyvarin/papers/NN00new.pdf</a> <a href="#fnref:1" class="footnote-back-ref">↩</a></p></li> <li id="fn:2"><p>Hyvarinen, A. (1999). "Fast and robust fixed-point algorithms for independent component analysis" (PDF). IEEE Transactions on Neural Networks. 10 (3): 626–634. CiteSeerX 10.1.1.297.8229. doi:10.1109/72.761722. PMID 18252563. <a href="http://www.cs.helsinki.fi/u/ahyvarin/papers/TNN99new.pdf" target="_blank">http://www.cs.helsinki.fi/u/ahyvarin/papers/TNN99new.pdf</a> <a href="#fnref:2" class="footnote-back-ref">↩</a></p></li> <li id="fn:3"><p>Hyvärinen, A.; Oja, E. (2000). "Independent component analysis: Algorithms and applications" (PDF). Neural Networks. 13 (4–5): 411–430. CiteSeerX 10.1.1.79.7003. doi:10.1016/S0893-6080(00)00026-5. PMID 10946390. <a href="http://www.cs.helsinki.fi/u/ahyvarin/papers/NN00new.pdf" target="_blank">http://www.cs.helsinki.fi/u/ahyvarin/papers/NN00new.pdf</a> <a href="#fnref:3" class="footnote-back-ref">↩</a></p></li> </ol>