Kernel embedding of distributions

In <a href="/facts/Machine_learning/e0w0XJTu">machine learning</a>, the kernel embedding of distributions (also called the kernel mean or mean map) comprises a class of <a href="/facts/Nonparametric/svTDJHTb">nonparametric</a> methods in which a <a href="/facts/Probability_distribution/EpsKKVRu">probability distribution</a> is represented as an element of a <a href="/facts/Reproducing_kernel_Hilbert_space/VegkYAXb">reproducing kernel Hilbert space</a> (RKHS). A generalization of the individual data-point feature mapping done in classical <a href="/facts/Kernel_methods/fYuURIPk">kernel methods</a>, the embedding of distributions into infinite-dimensional feature spaces can preserve all of the statistical features of arbitrary distributions, while allowing one to compare and manipulate distributions using Hilbert space operations such as <a href="/facts/Inner_product/HoyjElEy">inner products</a>, distances, <a href="/facts/Projection_(linear_algebra)/sElXGkxD">projections</a>, <a href="/facts/Linear_transformation/Q1PBKNVV">linear transformations</a>, and <a href="/facts/Spectral_theory/8sAEEIMM">spectral analysis</a>. This <a href="/facts/Machine_learning/e0w0XJTu">learning</a> framework is very general and can be applied to distributions over any space 
 
 
 
 Ω
 
 
 {\displaystyle \Omega }
 
 on which a sensible <a href="/facts/Kernel_function/Ws3zkxfl">kernel function</a> (measuring similarity between elements of 
 
 
 
 Ω
 
 
 {\displaystyle \Omega }
 
) may be defined. For example, various kernels have been proposed for learning from data which are: <a href="/facts/Vector_(mathematics_and_physics)/U8mM7A5F">vectors</a> in 
 
 
 
 
 
 R
 
 
 d
 
 
 
 
 {\displaystyle \mathbb {R} ^{d}}
 
, discrete classes/categories, <a href="/facts/String_(computer_science)/4ChJu5WS">strings</a>, <a href="/facts/Graph_(discrete_mathematics)/kw3eIBUe">graphs</a>/<a href="/facts/Network_theory/m6CQovwm">networks</a>, images, <a href="/facts/Time_series/fSXPR817">time series</a>, <a href="/facts/Manifold/rXPERJtu">manifolds</a>, <a href="/facts/Dynamical_systems/782gREl5">dynamical systems</a>, and other structured objects. The theory behind kernel embeddings of distributions has been primarily developed by <a href="http://alex.smola.org/">Alex Smola</a>, <a href="http://www.cc.gatech.edu/~lsong/">Le Song </a>, <a href="http://www.gatsby.ucl.ac.uk/~gretton/">Arthur Gretton</a>, and <a href="/facts/Bernhard_Sch%25C3%25B6lkopf/LCexnwav">Bernhard Schölkopf</a>. A review of recent works on kernel embedding of distributions can be found in.
The analysis of distributions is fundamental in <a href="/facts/Machine_learning/e0w0XJTu">machine learning</a> and <a href="/facts/Statistics/DNPsKGYU">statistics</a>, and many algorithms in these fields rely on information theoretic approaches such as <a href="/facts/Entropy/s52IXeFz">entropy</a>, <a href="/facts/Mutual_information/HIUvsjvV">mutual information</a>, or <a href="/facts/Kullback%25E2%2580%2593Leibler_divergence/nh7SjlPE">Kullback–Leibler divergence</a>. However, to estimate these quantities, one must first either perform density estimation, or employ sophisticated space-partitioning/bias-correction strategies which are typically infeasible for high-dimensional data. Commonly, methods for modeling complex distributions rely on parametric assumptions that may be unfounded or computationally challenging (e.g. <a href="/facts/Mixture_model/WWe6TJlg">Gaussian mixture models</a>), while nonparametric methods like <a href="/facts/Kernel_density_estimation/NmznyEW8">kernel density estimation</a> (Note: the smoothing kernels in this context have a different interpretation than the kernels discussed here) or <a href="/facts/Characteristic_function_(probability_theory)/rl3uAGKM">characteristic function</a> representation (via the <a href="/facts/Fourier_transform/XXhKJtc8">Fourier transform</a> of the distribution) break down in high-dimensional settings.
Methods based on the kernel embedding of distributions sidestep these problems and also possess the following advantages: 

<ol><li>Data may be modeled without restrictive assumptions about the form of the distributions and relationships between variables</li>
<li>Intermediate density estimation is not needed</li>
<li>Practitioners may specify the properties of a distribution most relevant for their problem (incorporating prior knowledge via choice of the kernel)</li>
<li>If a characteristic kernel is used, then the embedding can uniquely preserve all information about a distribution, while thanks to the <a href="/facts/Kernel_trick/fYuURIPk">kernel trick</a>, computations on the potentially infinite-dimensional RKHS can be implemented in practice as simple <a href="/facts/Gramian_matrix/SGb2VGTU">Gram</a> matrix operations</li>
<li>Dimensionality-independent rates of convergence for the empirical kernel mean (estimated using samples from the distribution) to the kernel embedding of the true underlying distribution can be proven.</li>
<li>Learning algorithms based on this framework exhibit good generalization ability and finite sample convergence, while often being simpler and more effective than information theoretic methods</li></ol>
Thus, learning via the kernel embedding of distributions offers a principled drop-in replacement for information theoretic approaches and is a framework which not only subsumes many popular methods in machine learning and statistics as special cases, but also can lead to entirely new learning algorithms.

Kernel embedding of distributions open-in-new

Kernel embedding of distributions