Manifold hypothesis

<h2 id="the-information-geometry-of-statistical-manifolds">The information geometry of statistical manifolds</h2>
<p>An empirically-motivated approach to the manifold hypothesis focuses on its correspondence with an effective theory for manifold learning under the assumption that robust machine learning requires encoding the dataset of interest using methods for data compression. This perspective gradually emerged using the tools of information geometry thanks to the coordinated effort of scientists working on the <a href="/facts/Efficient_coding_hypothesis/g4ohdBfq">efficient coding hypothesis</a>, <a href="/facts/Predictive_coding/DRRQQcTB">predictive coding</a> and <a href="/facts/Variational_Bayesian_methods/pFuMLBcU">variational Bayesian methods</a>. 
</p><p>The argument for reasoning about the information geometry on the latent space of distributions rests upon the existence and uniqueness of the <a href="/facts/Fisher_information_metric/zEeAWdII">Fisher information metric</a>.<a class="footnote-ref" id="fnref:6" href="#fn:6"><sup>6</sup></a> In this general setting, we are trying to find a stochastic embedding of a statistical manifold. From the perspective of dynamical systems, in the <a href="/facts/Big_data/XGtrVg4z">big data</a> regime this manifold generally exhibits certain properties such as homeostasis: 
</p>
<ol><li>We can sample large amounts of data from the underlying generative process.</li>
<li>Machine Learning experiments are reproducible, so the statistics of the generating process exhibit stationarity.</li></ol>
<p>In a sense made precise by theoretical neuroscientists working on the <a href="/facts/Free_energy_principle/Qgdkg134">free energy principle</a>, the statistical manifold in question possesses a <a href="/facts/Markov_blanket/h3Ns4Nhj">Markov blanket</a>.<a class="footnote-ref" id="fnref:7" href="#fn:7"><sup>7</sup></a>
</p>
<h2 id="see-also">See also</h2>
<ul><li><a href="/facts/Kolmogorov_complexity/L9ezq8pY">Kolmogorov complexity</a></li>
<li><a href="/facts/Minimum_description_length/em8tXqjf">Minimum description length</a></li>
<li><a href="/facts/Solomonoff%2527s_theory_of_inductive_inference/4kecWjSB">Solomonoff's theory of inductive inference</a></li></ul>

<h2 id="further-reading">Further reading</h2>

<ul><li>Brown, Bradley C. A.; Caterini, Anthony L.; Ross, Brendan Leigh; Cresswell, Jesse C.; Loaiza-Ganem, Gabriel (2023). <a href="https://openreview.net/pdf?id=Rvee9CAX4fi"><i>The Union of Manifolds Hypothesis and its Implications for Deep Generative Modelling</i></a>. The Eleventh International Conference on Learning Representations. <a href="/facts/ArXiv_(identifier)/H6EtgnBe">arXiv</a>:<a href="https://arxiv.org/abs/2207.02862">2207.02862</a>.</li>
<li>Lee, Yonghyeon (2023). <i>A Geometric Perspective on Autoencoders</i>. <a href="/facts/ArXiv_(identifier)/H6EtgnBe">arXiv</a>:<a href="https://arxiv.org/abs/2309.08247">2309.08247</a>.</li></ul>

<h2 id="references">References</h2>

<ol>
<li id="fn:1"><p>Gorban, A. N.; Tyukin, I. Y. (2018). "Blessing of dimensionality: mathematical foundations of the statistical physics of data". Phil. Trans. R. Soc. A. 15 (3): 20170237. Bibcode:2018RSPTA.37670237G. doi:10.1098/rsta.2017.0237. PMC 5869543. PMID 29555807. <a href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5869543" target="_blank">https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5869543</a> <a href="#fnref:1" class="footnote-back-ref">↩</a></p></li>
<li id="fn:2"><p>Cayton, L. (2005). Algorithms for manifold learning (PDF) (Technical report). University of California at San Diego. p. 1. 12(1–17). <a href="https://www.lcayton.com/resexam.pdf" target="_blank">https://www.lcayton.com/resexam.pdf</a> <a href="#fnref:2" class="footnote-back-ref">↩</a></p></li>
<li id="fn:3"><p>Fefferman, Charles; Mitter, Sanjoy; Narayanan, Hariharan (2016-02-09). "Testing the manifold hypothesis". Journal of the American Mathematical Society. 29 (4): 983–1049. arXiv:1310.0425. doi:10.1090/jams/852. S2CID 50258911. <a href="https://www.ams.org/jams/2016-29-04/S0894-0347-2016-00852-4/" target="_blank">https://www.ams.org/jams/2016-29-04/S0894-0347-2016-00852-4/</a> <a href="#fnref:3" class="footnote-back-ref">↩</a></p></li>
<li id="fn:4"><p>Olah, Christopher (2014). "Blog: Neural Networks, Manifolds, and Topology". <a href="https://colah.github.io/posts/2014-03-NN-Manifolds-Topology/" target="_blank">https://colah.github.io/posts/2014-03-NN-Manifolds-Topology/</a> <a href="#fnref:4" class="footnote-back-ref">↩</a></p></li>
<li id="fn:5"><p>Chollet, Francois (2021). Deep Learning with Python (2nd ed.). Manning. pp. 128–129. ISBN 9781617296864. <a href="9781617296864" target="_blank">9781617296864</a> <a href="#fnref:5" class="footnote-back-ref">↩</a></p></li>
<li id="fn:6"><p>Caticha, Ariel (2015). Geometry from Information Geometry. MaxEnt 2015, the 35th International Workshop on Bayesian Inference and Maximum Entropy Methods in Science and Engineering. arXiv:1512.09076. <a href="/wiki/ArXiv_(identifier)" target="_blank">/wiki/ArXiv_(identifier)</a> <a href="#fnref:6" class="footnote-back-ref">↩</a></p></li>
<li id="fn:7"><p>Kirchhoff, Michael; Parr, Thomas; Palacios, Ensor; Friston, Karl; Kiverstein, Julian (2018). "The Markov blankets of life: autonomy, active inference and the free energy principle". J. R. Soc. Interface. 15 (138): 20170792. doi:10.1098/rsif.2017.0792. PMC 5805980. PMID 29343629. <a href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5805980" target="_blank">https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5805980</a> <a href="#fnref:7" class="footnote-back-ref">↩</a></p></li>
</ol>

Manifold hypothesis open-in-new

Manifold hypothesis