Empirical process

<h2 id="definition">Definition</h2>
For X1, X2, ... Xn <a href="/facts/Independent_and_identically-distributed_random_variables/othIRaWt">independent and identically-distributed random variables</a> in R with common <a href="/facts/Cumulative_distribution_function/WaKU8tp4">cumulative distribution function</a> F(x), the empirical distribution function is defined by

F
          
            n
          
        
        (
        x
        )
        =
        
          
            1
            n
          
        
        
          ∑
          
            i
            =
            1
          
          
            n
          
        
        
          I
          
            (
            −
            ∞
            ,
            x
            ]
          
        
        (
        
          X
          
            i
          
        
        )
        ,
      
    
    {\displaystyle F_{n}(x)={\frac {1}{n}}\sum _{i=1}^{n}I_{(-\infty ,x]}(X_{i}),}

where IC is the <a href="/facts/Indicator_function/QEuh04NM">indicator function</a> of the set C.
For every (fixed) x, Fn(x) is a sequence of random variables which converge to F(x) <a href="/facts/Almost_surely/fWaxzpBA">almost surely</a> by the strong <a href="/facts/Law_of_large_numbers/X3Bjcy3v">law of large numbers</a>. That is, Fn converges to F <a href="/facts/Pointwise_convergence/TXGpeyIK">pointwise</a>. Glivenko and Cantelli strengthened this result by proving <a href="/facts/Uniform_convergence/ecz9eNWn">uniform convergence</a> of Fn to F by the <a href="/facts/Glivenko%25E2%2580%2593Cantelli_theorem/o5E7Tjda">Glivenko–Cantelli theorem</a>.<a class="footnote-ref" id="fnref:2" href="#fn:2">2</a>
A centered and scaled version of the empirical measure is the <a href="/facts/Signed_measure/vukVf84A">signed measure</a>

G
          
            n
          
        
        (
        A
        )
        =
        
          
            n
          
        
        (
        
          P
          
            n
          
        
        (
        A
        )
        −
        P
        (
        A
        )
        )
      
    
    {\displaystyle G_{n}(A)={\sqrt {n}}(P_{n}(A)-P(A))}

It induces a map on measurable functions f given by

f
        ↦
        
          G
          
            n
          
        
        f
        =
        
          
            n
          
        
        (
        
          P
          
            n
          
        
        −
        P
        )
        f
        =
        
          
            n
          
        
        
          (
          
            
              
                1
                n
              
            
            
              ∑
              
                i
                =
                1
              
              
                n
              
            
            f
            (
            
              X
              
                i
              
            
            )
            −
            
              E
            
            f
          
          )
        
      
    
    {\displaystyle f\mapsto G_{n}f={\sqrt {n}}(P_{n}-P)f={\sqrt {n}}\left({\frac {1}{n}}\sum _{i=1}^{n}f(X_{i})-\mathbb {E} f\right)}

By the <a href="/facts/Central_limit_theorem/0gc2L0Wd">central limit theorem</a>, 
 
 
 
 
 G
 
 n
 
 
 (
 A
 )
 
 
 {\displaystyle G_{n}(A)}
 
 <a href="/facts/Converges_in_distribution/pWdatFlY">converges in distribution</a> to a <a href="/facts/Normal_distribution/UapjjPyQ">normal</a> random variable N(0, P(A)(1 − P(A))) for fixed measurable set A. Similarly, for a fixed function f, 
 
 
 
 
 G
 
 n
 
 
 f
 
 
 {\displaystyle G_{n}f}
 
 converges in distribution to a normal random variable 
 
 
 
 N
 (
 0
 ,
 
 E
 
 (
 f
 −
 
 E
 
 f
 
 )
 
 2
 
 
 )
 
 
 {\displaystyle N(0,\mathbb {E} (f-\mathbb {E} f)^{2})}
 
, provided that 
 
 
 
 
 E
 
 f
 
 
 {\displaystyle \mathbb {E} f}
 
 and 
 
 
 
 
 E
 
 
 f
 
 2
 
 
 
 
 {\displaystyle \mathbb {E} f^{2}}
 
 exist.
Definition

(
 
 
 
 G
 
 n
 
 
 (
 c
 )
 
 
 
 )
 
 
 
 c
 ∈
 
 
 C
 
 
 
 
 
 
 {\displaystyle {\bigl (}G_{n}(c){\bigr )}_{c\in {\mathcal {C}}}}
 
 is called an empirical process indexed by 
 
 
 
 
 
 C
 
 
 
 
 {\displaystyle {\mathcal {C}}}
 
, a collection of measurable subsets of S.

(
 
 
 
 G
 
 n
 
 
 f
 
 
 
 )
 
 
 
 f
 ∈
 
 
 F
 
 
 
 
 
 
 {\displaystyle {\bigl (}G_{n}f{\bigr )}_{f\in {\mathcal {F}}}}
 
 is called an empirical process indexed by 
 
 
 
 
 
 F
 
 
 
 
 {\displaystyle {\mathcal {F}}}
 
, a collection of measurable functions from S to 
 
 
 
 
 R
 
 
 
 {\displaystyle \mathbb {R} }
 
.
A significant result in the area of empirical processes is <a href="/facts/Donsker%2527s_theorem/7PdQrWrk">Donsker's theorem</a>. It has led to a study of Donsker classes: sets of functions with the useful property that empirical processes indexed by these classes <a href="/facts/Convergence_of_random_variables/pWdatFlY">converge weakly</a> to a certain <a href="/facts/Gaussian_process/MrBq7kYW">Gaussian process</a>. While it can be shown that Donsker classes are <a href="/facts/Glivenko%25E2%2580%2593Cantelli_class/o5E7Tjda">Glivenko–Cantelli classes</a>, the converse is not true in general.

<h2 id="example">Example</h2>
As an example, consider <a href="/facts/Empirical_distribution_function/frWIF9F9">empirical distribution functions</a>. For real-valued <a href="/facts/Iid/othIRaWt">iid</a> random variables X1, X2, ..., Xn they are given by

F
          
            n
          
        
        (
        x
        )
        =
        
          P
          
            n
          
        
        (
        (
        −
        ∞
        ,
        x
        ]
        )
        =
        
          P
          
            n
          
        
        
          I
          
            (
            −
            ∞
            ,
            x
            ]
          
        
        .
      
    
    {\displaystyle F_{n}(x)=P_{n}((-\infty ,x])=P_{n}I_{(-\infty ,x]}.}

In this case, empirical processes are indexed by a class 
 
 
 
 
 
 C
 
 
 =
 {
 (
 −
 ∞
 ,
 x
 ]
 :
 x
 ∈
 
 R
 
 }
 .
 
 
 {\displaystyle {\mathcal {C}}=\{(-\infty ,x]:x\in \mathbb {R} \}.}
 
 It has been shown that 
 
 
 
 
 
 C
 
 
 
 
 {\displaystyle {\mathcal {C}}}
 
 is a Donsker class, in particular,

n
 
 
 (
 
 F
 
 n
 
 
 (
 x
 )
 −
 F
 (
 x
 )
 )
 
 
 {\displaystyle {\sqrt {n}}(F_{n}(x)-F(x))}
 
 converges <a href="/facts/Weak_convergence_of_measures/fb0I0KAL">weakly</a> in 
 
 
 
 
 ℓ
 
 ∞
 
 
 (
 
 R
 
 )
 
 
 {\displaystyle \ell ^{\infty }(\mathbb {R} )}
 
 to a <a href="/facts/Brownian_bridge/vnStxc6h">Brownian bridge</a> B(F(x)) .
<h2 id="see-also">See also</h2>
<ul><li><a href="/facts/Khmaladze_transformation/cozJGDeT">Khmaladze transformation</a></li>
<li><a href="/facts/Weak_convergence_of_measures/fb0I0KAL">Weak convergence of measures</a></li>
<li><a href="/facts/Glivenko%25E2%2580%2593Cantelli_theorem/o5E7Tjda">Glivenko–Cantelli theorem</a></li></ul>

<h2 id="further-reading">Further reading</h2>
<ul><li>Billingsley, P. (1995). Probability and Measure (Third ed.). New York: John Wiley and Sons. <a href="/facts/ISBN_(identifier)/15AdSPa9">ISBN</a> 0471007102.</li>
<li>Donsker, M. D. (1952). <a href="https://doi.org/10.1214%2Faoms%2F1177729445">"Justification and Extension of Doob's Heuristic Approach to the Kolmogorov- Smirnov Theorems"</a>. The Annals of Mathematical Statistics. 23 (2): 277–281. <a href="/facts/Doi_(identifier)/muM9Etpq">doi</a>:<a href="https://doi.org/10.1214%2Faoms%2F1177729445">10.1214/aoms/1177729445</a>.</li>
<li>Dudley, R. M. (1978). <a href="https://doi.org/10.1214%2Faop%2F1176995384">"Central Limit Theorems for Empirical Measures"</a>. The Annals of Probability. 6 (6): 899–929. <a href="/facts/Doi_(identifier)/muM9Etpq">doi</a>:<a href="https://doi.org/10.1214%2Faop%2F1176995384">10.1214/aop/1176995384</a>.</li>
<li>Dudley, R. M. (1999). Uniform Central Limit Theorems. Cambridge Studies in Advanced Mathematics. Vol. 63. Cambridge, UK: Cambridge University Press.</li>
<li>Kosorok, M. R. (2008). Introduction to Empirical Processes and Semiparametric Inference. Springer Series in Statistics. <a href="/facts/Doi_(identifier)/muM9Etpq">doi</a>:<a href="https://doi.org/10.1007%2F978-0-387-74978-5">10.1007/978-0-387-74978-5</a>. <a href="/facts/ISBN_(identifier)/15AdSPa9">ISBN</a> 978-0-387-74977-8.</li>
<li><a href="/facts/Galen_Shorack/S0BQvl7j">Shorack, G. R.</a>; <a href="/facts/Jon_Wellner/VUsfkkVE">Wellner, J. A.</a> (2009). Empirical Processes with Applications to Statistics. <a href="/facts/Doi_(identifier)/muM9Etpq">doi</a>:<a href="https://doi.org/10.1137%2F1.9780898719017">10.1137/1.9780898719017</a>. <a href="/facts/ISBN_(identifier)/15AdSPa9">ISBN</a> 978-0-89871-684-9.</li>
<li><a href="/facts/Aad_van_der_Vaart/zEPpqEdW">van der Vaart, Aad W.</a>; Wellner, Jon A. (2000). Weak Convergence and Empirical Processes: With Applications to Statistics (2nd ed.). Springer. <a href="/facts/ISBN_(identifier)/15AdSPa9">ISBN</a> 978-0-387-94640-5.</li>
<li>Dzhaparidze, K. O.; Nikulin, M. S. (1982). "Probability distributions of the Kolmogorov and omega-square statistics for continuous distributions with shift and scale parameters". Journal of Soviet Mathematics. 20 (3): 2147. <a href="/facts/Doi_(identifier)/muM9Etpq">doi</a>:<a href="https://doi.org/10.1007%2FBF01239992">10.1007/BF01239992</a>. <a href="/facts/S2CID_(identifier)/ldJsHa2Y">S2CID</a> <a href="https://api.semanticscholar.org/CorpusID:123206522">123206522</a>.</li></ul>
<h2 id="external-links">External links</h2>
<ul><li><a href="http://www.stat.yale.edu/~pollard/Books/Iowa">Empirical Processes: Theory and Applications</a>, by David Pollard, a textbook available online.</li>
<li><a href="http://www.bios.unc.edu/~kosorok/current.pdf">Introduction to Empirical Processes and Semiparametric Inference</a>, by Michael Kosorok, another textbook available online.</li></ul>

<h2 id="references">References</h2>

<ol>
<li id="fn:1">Mojirsheibani, M. (2007). "Nonparametric curve estimation with missing data: A general empirical process approach". Journal of Statistical Planning and Inference. 137 (9): 2733–2758. doi:10.1016/j.jspi.2006.02.016. <a href="/wiki/Doi_(identifier)" target="_blank">/wiki/Doi_(identifier)</a> <a href="#fnref:1" class="footnote-back-ref">↩</a></li>
<li id="fn:2">Wolfowitz, J. (1954). "Generalization of the Theorem of Glivenko-Cantelli". The Annals of Mathematical Statistics. 25: 131–138. doi:10.1214/aoms/1177728852. <a href="https://doi.org/10.1214%2Faoms%2F1177728852" target="_blank">https://doi.org/10.1214%2Faoms%2F1177728852</a> <a href="#fnref:2" class="footnote-back-ref">↩</a></li>
</ol>

Empirical process open-in-new

Empirical process