Menu
Home
People
Places
Arts
History
Plants & Animals
Science
Life & Culture
Technology
Reference.org
Proofs of convergence of random variables
open-in-new
<h2 id="convergence-almost-surely-implies-convergence-in-probability"> Convergence almost surely implies convergence in probability</h2> X n → a s X ⇒ X n → p X {\displaystyle X_{n}\ {\overset {\mathrm {as} }{\rightarrow }}\ X\quad \Rightarrow \quad X_{n}\ {\overset {p}{\rightarrow }}\ X} <p>Proof: If { X n } {\displaystyle \{X_{n}\}} converges to X {\displaystyle X} almost surely, it means that the set of points O = { ω ∣ lim X n ( ω ) ≠ X ( ω ) } {\displaystyle O=\{\omega \mid \lim X_{n}(\omega )\neq X(\omega )\}} has measure zero. Now fix ε > 0 {\displaystyle \varepsilon >0} and consider a sequence of sets </p> A n = ⋃ m ≥ n { | X m − X | > ε } {\displaystyle A_{n}=\bigcup _{m\geq n}\left\{\left|X_{m}-X\right|>\varepsilon \right\}} <p>This sequence of sets is decreasing ( A n ⊇ A n + 1 ⊇ … {\displaystyle A_{n}\supseteq A_{n+1}\supseteq \ldots } ) towards the set </p> A ∞ = ⋂ n ≥ 1 A n . {\displaystyle A_{\infty }=\bigcap _{n\geq 1}A_{n}.} <p>The probabilities of this sequence are also decreasing, so lim Pr ( A n ) = Pr ( A ∞ ) {\displaystyle \lim \operatorname {Pr} (A_{n})=\operatorname {Pr} (A_{\infty })} ; we shall show now that this number is equal to zero. Now for any point ω {\displaystyle \omega } outside of O {\displaystyle O} we have lim X n ( ω ) = X ( ω ) {\displaystyle \lim X_{n}(\omega )=X(\omega )} , which implies that | X n ( ω ) − X ( ω ) | < ε {\displaystyle \left|X_{n}(\omega )-X(\omega )\right|<\varepsilon } for all n ≥ N {\displaystyle n\geq N} for some N {\displaystyle N} . In particular, for such n {\displaystyle n} the point ω {\displaystyle \omega } will not lie in A n {\displaystyle A_{n}} , and hence won't lie in A ∞ {\displaystyle A_{\infty }} . Therefore, A ∞ ⊆ O {\displaystyle A_{\infty }\subseteq O} and so Pr ( A ∞ ) = 0 {\displaystyle \operatorname {Pr} (A_{\infty })=0} . </p><p>Finally, by continuity from above, </p> Pr ( | X n − X | > ε ) ≤ Pr ( A n ) → n → ∞ 0 , {\displaystyle \operatorname {Pr} \left(|X_{n}-X|>\varepsilon \right)\leq \operatorname {Pr} (A_{n})\ {\underset {n\to \infty }{\rightarrow }}0,} <p>which by definition means that X n {\displaystyle X_{n}} converges in probability to X {\displaystyle X} . </p> <h2 id="convergence-in-probability-does-not-imply-almost-sure-convergence-in-the-discrete-case"> Convergence in probability does not imply almost sure convergence in the discrete case</h2> <p>If <i>Xn</i> are independent random variables assuming value one with probability 1/<i>n</i> and zero otherwise, then <i>Xn</i> converges to zero in probability but not almost surely. This can be verified using the <a href="/facts/Borel%25E2%2580%2593Cantelli_lemma/BX397zgr">Borel–Cantelli lemmas</a>. </p> <h2 id="convergence-in-probability-implies-convergence-in-distribution"> Convergence in probability implies convergence in distribution</h2> X n → p X ⇒ X n → d X , {\displaystyle X_{n}\ {\xrightarrow {p}}\ X\quad \Rightarrow \quad X_{n}\ {\xrightarrow {d}}\ X,} <h3>Proof for the case of scalar random variables</h3> <p>Lemma. Let <i>X</i>, <i>Y</i> be random variables, let <i>a</i> be a real number and ε > 0. Then </p> Pr ( Y ≤ a ) ≤ Pr ( X ≤ a + ε ) + Pr ( | Y − X | > ε ) . {\displaystyle \operatorname {Pr} (Y\leq a)\leq \operatorname {Pr} (X\leq a+\varepsilon )+\operatorname {Pr} (|Y-X|>\varepsilon ).} <p>Proof of lemma: </p> Pr ( Y ≤ a ) = Pr ( Y ≤ a , X ≤ a + ε ) + Pr ( Y ≤ a , X > a + ε ) ≤ Pr ( X ≤ a + ε ) + Pr ( Y − X ≤ a − X , a − X < − ε ) ≤ Pr ( X ≤ a + ε ) + Pr ( Y − X < − ε ) ≤ Pr ( X ≤ a + ε ) + Pr ( Y − X < − ε ) + Pr ( Y − X > ε ) = Pr ( X ≤ a + ε ) + Pr ( | Y − X | > ε ) {\displaystyle {\begin{aligned}\operatorname {Pr} (Y\leq a)&=\operatorname {Pr} (Y\leq a,\ X\leq a+\varepsilon )+\operatorname {Pr} (Y\leq a,\ X>a+\varepsilon )\\&\leq \operatorname {Pr} (X\leq a+\varepsilon )+\operatorname {Pr} (Y-X\leq a-X,\ a-X<-\varepsilon )\\&\leq \operatorname {Pr} (X\leq a+\varepsilon )+\operatorname {Pr} (Y-X<-\varepsilon )\\&\leq \operatorname {Pr} (X\leq a+\varepsilon )+\operatorname {Pr} (Y-X<-\varepsilon )+\operatorname {Pr} (Y-X>\varepsilon )\\&=\operatorname {Pr} (X\leq a+\varepsilon )+\operatorname {Pr} (|Y-X|>\varepsilon )\end{aligned}}} <p>Shorter proof of the lemma: </p><p>We have </p> { Y ≤ a } ⊂ { X ≤ a + ε } ∪ { | Y − X | > ε } {\displaystyle {\begin{aligned}\{Y\leq a\}\subset \{X\leq a+\varepsilon \}\cup \{|Y-X|>\varepsilon \}\end{aligned}}} <p>for if Y ≤ a {\displaystyle Y\leq a} and | Y − X | ≤ ε {\displaystyle |Y-X|\leq \varepsilon } , then X ≤ a + ε {\displaystyle X\leq a+\varepsilon } . Hence by the union bound, </p> Pr ( Y ≤ a ) ≤ Pr ( X ≤ a + ε ) + Pr ( | Y − X | > ε ) . {\displaystyle {\begin{aligned}\operatorname {Pr} (Y\leq a)\leq \operatorname {Pr} (X\leq a+\varepsilon )+\operatorname {Pr} (|Y-X|>\varepsilon ).\end{aligned}}} <p>Proof of the theorem: Recall that in order to prove convergence in distribution, one must show that the sequence of cumulative distribution functions converges to the <i>FX</i> at every point where <i>FX</i> is continuous. Let <i>a</i> be such a point. For every ε > 0, due to the preceding lemma, we have: </p> Pr ( X n ≤ a ) ≤ Pr ( X ≤ a + ε ) + Pr ( | X n − X | > ε ) Pr ( X ≤ a − ε ) ≤ Pr ( X n ≤ a ) + Pr ( | X n − X | > ε ) {\displaystyle {\begin{aligned}\operatorname {Pr} (X_{n}\leq a)&\leq \operatorname {Pr} (X\leq a+\varepsilon )+\operatorname {Pr} (|X_{n}-X|>\varepsilon )\\\operatorname {Pr} (X\leq a-\varepsilon )&\leq \operatorname {Pr} (X_{n}\leq a)+\operatorname {Pr} (|X_{n}-X|>\varepsilon )\end{aligned}}} <p>So, we have </p> Pr ( X ≤ a − ε ) − Pr ( | X n − X | > ε ) ≤ Pr ( X n ≤ a ) ≤ Pr ( X ≤ a + ε ) + Pr ( | X n − X | > ε ) . {\displaystyle \operatorname {Pr} (X\leq a-\varepsilon )-\operatorname {Pr} \left(\left|X_{n}-X\right|>\varepsilon \right)\leq \operatorname {Pr} (X_{n}\leq a)\leq \operatorname {Pr} (X\leq a+\varepsilon )+\operatorname {Pr} \left(\left|X_{n}-X\right|>\varepsilon \right).} <p>Taking the limit as <i>n</i> → ∞, we obtain: </p> F X ( a − ε ) ≤ lim n → ∞ Pr ( X n ≤ a ) ≤ F X ( a + ε ) , {\displaystyle F_{X}(a-\varepsilon )\leq \lim _{n\to \infty }\operatorname {Pr} (X_{n}\leq a)\leq F_{X}(a+\varepsilon ),} <p>where <i>FX</i>(<i>a</i>) = Pr(<i>X</i> ≤ <i>a</i>) is the <a href="/facts/Cumulative_distribution_function/WaKU8tp4">cumulative distribution function</a> of <i>X</i>. This function is continuous at <i>a</i> by assumption, and therefore both <i>FX</i>(<i>a</i>−ε) and <i>FX</i>(<i>a</i>+ε) converge to <i>FX</i>(<i>a</i>) as ε → 0+. Taking this limit, we obtain </p> lim n → ∞ Pr ( X n ≤ a ) = Pr ( X ≤ a ) , {\displaystyle \lim _{n\to \infty }\operatorname {Pr} (X_{n}\leq a)=\operatorname {Pr} (X\leq a),} <p>which means that {<i>Xn</i>} converges to <i>X</i> in distribution. </p> <h3>Proof for the generic case</h3> <p>The implication follows for when <i>Xn</i> is a random vector by using this property proved later on this page and by taking <i>Xn = X</i> in the statement of that property. </p> <h2 id="convergence-in-distribution-to-a-constant-implies-convergence-in-probability"> Convergence in distribution to a constant implies convergence in probability</h2> X n → d c ⇒ X n → p c , {\displaystyle X_{n}\ {\xrightarrow {d}}\ c\quad \Rightarrow \quad X_{n}\ {\xrightarrow {p}}\ c,} provided <i>c</i> is a constant. <p>Proof: Fix ε > 0. Let <i>B</i>ε(<i>c</i>) be the <a href="/facts/Ball_(mathematics)/f9vpMVMp">open ball</a> of radius ε around point <i>c</i>, and <i>B</i>ε(<i>c</i>)<i>c</i> its complement. Then </p> Pr ( | X n − c | ≥ ε ) = Pr ( X n ∈ B ε ( c ) c ) . {\displaystyle \operatorname {Pr} \left(|X_{n}-c|\geq \varepsilon \right)=\operatorname {Pr} \left(X_{n}\in B_{\varepsilon }(c)^{c}\right).} <p>By the portmanteau lemma (part C), if <i>Xn</i> converges in distribution to <i>c</i>, then the <a href="/facts/Limsup/QeAhlygs">limsup</a> of the latter probability must be less than or equal to Pr(<i>c</i> ∈ <i>B</i>ε(<i>c</i>)<i>c</i>), which is obviously equal to zero. Therefore, </p> lim n → ∞ Pr ( | X n − c | ≥ ε ) ≤ lim sup n → ∞ Pr ( | X n − c | ≥ ε ) = lim sup n → ∞ Pr ( X n ∈ B ε ( c ) c ) ≤ Pr ( c ∈ B ε ( c ) c ) = 0 {\displaystyle {\begin{aligned}\lim _{n\to \infty }\operatorname {Pr} \left(\left|X_{n}-c\right|\geq \varepsilon \right)&\leq \limsup _{n\to \infty }\operatorname {Pr} \left(\left|X_{n}-c\right|\geq \varepsilon \right)\\&=\limsup _{n\to \infty }\operatorname {Pr} \left(X_{n}\in B_{\varepsilon }(c)^{c}\right)\\&\leq \operatorname {Pr} \left(c\in B_{\varepsilon }(c)^{c}\right)=0\end{aligned}}} <p>which by definition means that <i>Xn</i> converges to <i>c</i> in probability. </p> <h2 id="convergence-in-probability-to-a-sequence-converging-in-distribution-implies-convergence-to-the-same-distribution"> Convergence in probability to a sequence converging in distribution implies convergence to the same distribution</h2> | Y n − X n | → p 0 , X n → d X ⇒ Y n → d X {\displaystyle |Y_{n}-X_{n}|\ {\xrightarrow {p}}\ 0,\ \ X_{n}\ {\xrightarrow {d}}\ X\ \quad \Rightarrow \quad Y_{n}\ {\xrightarrow {d}}\ X} <p>Proof: We will prove this theorem using the portmanteau lemma, part B. As required in that lemma, consider any bounded function <i>f</i> (i.e. |<i>f</i>(<i>x</i>)| ≤ <i>M</i>) which is also Lipschitz: </p> ∃ K > 0 , ∀ x , y : | f ( x ) − f ( y ) | ≤ K | x − y | . {\displaystyle \exists K>0,\forall x,y:\quad |f(x)-f(y)|\leq K|x-y|.} <p>Take some ε > 0 and majorize the expression |E[<i>f</i>(<i>Yn</i>)] − E[<i>f</i>(<i>Xn</i>)]| as </p> | E [ f ( Y n ) ] − E [ f ( X n ) ] | ≤ E [ | f ( Y n ) − f ( X n ) | ] = E [ | f ( Y n ) − f ( X n ) | 1 { | Y n − X n | < ε } ] + E [ | f ( Y n ) − f ( X n ) | 1 { | Y n − X n | ≥ ε } ] ≤ E [ K | Y n − X n | 1 { | Y n − X n | < ε } ] + E [ 2 M 1 { | Y n − X n | ≥ ε } ] ≤ K ε Pr ( | Y n − X n | < ε ) + 2 M Pr ( | Y n − X n | ≥ ε ) ≤ K ε + 2 M Pr ( | Y n − X n | ≥ ε ) {\displaystyle {\begin{aligned}\left|\operatorname {E} \left[f(Y_{n})\right]-\operatorname {E} \left[f(X_{n})\right]\right|&\leq \operatorname {E} \left[\left|f(Y_{n})-f(X_{n})\right|\right]\\&=\operatorname {E} \left[\left|f(Y_{n})-f(X_{n})\right|\mathbf {1} _{\left\{|Y_{n}-X_{n}|<\varepsilon \right\}}\right]+\operatorname {E} \left[\left|f(Y_{n})-f(X_{n})\right|\mathbf {1} _{\left\{|Y_{n}-X_{n}|\geq \varepsilon \right\}}\right]\\&\leq \operatorname {E} \left[K\left|Y_{n}-X_{n}\right|\mathbf {1} _{\left\{|Y_{n}-X_{n}|<\varepsilon \right\}}\right]+\operatorname {E} \left[2M\mathbf {1} _{\left\{|Y_{n}-X_{n}|\geq \varepsilon \right\}}\right]\\&\leq K\varepsilon \operatorname {Pr} \left(\left|Y_{n}-X_{n}\right|<\varepsilon \right)+2M\operatorname {Pr} \left(\left|Y_{n}-X_{n}\right|\geq \varepsilon \right)\\&\leq K\varepsilon +2M\operatorname {Pr} \left(\left|Y_{n}-X_{n}\right|\geq \varepsilon \right)\end{aligned}}} <p>(here 1{...} denotes the <a href="/facts/Indicator_function/QEuh04NM">indicator function</a>; the expectation of the indicator function is equal to the probability of corresponding event). Therefore, </p> | E [ f ( Y n ) ] − E [ f ( X ) ] | ≤ | E [ f ( Y n ) ] − E [ f ( X n ) ] | + | E [ f ( X n ) ] − E [ f ( X ) ] | ≤ K ε + 2 M Pr ( | Y n − X n | ≥ ε ) + | E [ f ( X n ) ] − E [ f ( X ) ] | . {\displaystyle {\begin{aligned}\left|\operatorname {E} \left[f(Y_{n})\right]-\operatorname {E} \left[f(X)\right]\right|&\leq \left|\operatorname {E} \left[f(Y_{n})\right]-\operatorname {E} \left[f(X_{n})\right]\right|+\left|\operatorname {E} \left[f(X_{n})\right]-\operatorname {E} \left[f(X)\right]\right|\\&\leq K\varepsilon +2M\operatorname {Pr} \left(|Y_{n}-X_{n}|\geq \varepsilon \right)+\left|\operatorname {E} \left[f(X_{n})\right]-\operatorname {E} \left[f(X)\right]\right|.\end{aligned}}} <p>If we take the limit in this expression as <i>n</i> → ∞, the second term will go to zero since {<i>Yn−Xn</i>} converges to zero in probability; and the third term will also converge to zero, by the portmanteau lemma and the fact that <i>Xn</i> converges to <i>X</i> in distribution. Thus </p> lim n → ∞ | E [ f ( Y n ) ] − E [ f ( X ) ] | ≤ K ε . {\displaystyle \lim _{n\to \infty }\left|\operatorname {E} \left[f(Y_{n})\right]-\operatorname {E} \left[f(X)\right]\right|\leq K\varepsilon .} <p>Since ε was arbitrary, we conclude that the limit must in fact be equal to zero, and therefore E[<i>f</i>(<i>Yn</i>)] → E[<i>f</i>(<i>X</i>)], which again by the portmanteau lemma implies that {<i>Yn</i>} converges to <i>X</i> in distribution. QED. </p> <h2 id="convergence-of-one-sequence-in-distribution-and-another-to-a-constant-implies-joint-convergence-in-distribution"> Convergence of one sequence in distribution and another to a constant implies joint convergence in distribution</h2> <p class="note">Main article: <a href="/facts/Slutsky%2527s_theorem/Fwh5f8Wt">Slutsky's theorem</a></p> X n → d X , Y n → p c ⇒ ( X n , Y n ) → d ( X , c ) {\displaystyle X_{n}\ {\xrightarrow {d}}\ X,\ \ Y_{n}\ {\xrightarrow {p}}\ c\ \quad \Rightarrow \quad (X_{n},Y_{n})\ {\xrightarrow {d}}\ (X,c)} provided <i>c</i> is a constant. <p>Proof: We will prove this statement using the portmanteau lemma, part A. </p><p>First we want to show that (<i>Xn</i>, <i>c</i>) converges in distribution to (<i>X</i>, <i>c</i>). By the portmanteau lemma this will be true if we can show that E[<i>f</i>(<i>Xn</i>, <i>c</i>)] → E[<i>f</i>(<i>X</i>, <i>c</i>)] for any bounded continuous function <i>f</i>(<i>x</i>, <i>y</i>). So let <i>f</i> be such arbitrary bounded continuous function. Now consider the function of a single variable <i>g</i>(<i>x</i>) := <i>f</i>(<i>x</i>, <i>c</i>). This will obviously be also bounded and continuous, and therefore by the portmanteau lemma for sequence {<i>Xn</i>} converging in distribution to <i>X</i>, we will have that E[<i>g</i>(<i>Xn</i>)] → E[<i>g</i>(<i>X</i>)]. However the latter expression is equivalent to “E[<i>f</i>(<i>Xn</i>, <i>c</i>)] → E[<i>f</i>(<i>X</i>, <i>c</i>)]”, and therefore we now know that (<i>Xn</i>, <i>c</i>) converges in distribution to (<i>X</i>, <i>c</i>). </p><p>Secondly, consider |(<i>Xn</i>, <i>Yn</i>) − (<i>Xn</i>, <i>c</i>)| = |<i>Yn</i> − <i>c</i>|. This expression converges in probability to zero because <i>Yn</i> converges in probability to <i>c</i>. Thus we have demonstrated two facts: </p> { | ( X n , Y n ) − ( X n , c ) | → p 0 , ( X n , c ) → d ( X , c ) . {\displaystyle {\begin{cases}\left|(X_{n},Y_{n})-(X_{n},c)\right|\ {\xrightarrow {p}}\ 0,\\(X_{n},c)\ {\xrightarrow {d}}\ (X,c).\end{cases}}} <p>By the property proved earlier, these two facts imply that (<i>Xn</i>, <i>Yn</i>) converge in distribution to (<i>X</i>, <i>c</i>). </p> <h2 id="convergence-of-two-sequences-in-probability-implies-joint-convergence-in-probability"> Convergence of two sequences in probability implies joint convergence in probability</h2> X n → p X , Y n → p Y ⇒ ( X n , Y n ) → p ( X , Y ) {\displaystyle X_{n}\ {\xrightarrow {p}}\ X,\ \ Y_{n}\ {\xrightarrow {p}}\ Y\ \quad \Rightarrow \quad (X_{n},Y_{n})\ {\xrightarrow {p}}\ (X,Y)} <p>Proof: </p> Pr ( | ( X n , Y n ) − ( X , Y ) | ≥ ε ) ≤ Pr ( | X n − X | + | Y n − Y | ≥ ε ) ≤ Pr ( | X n − X | ≥ ε / 2 ) + Pr ( | Y n − Y | ≥ ε / 2 ) {\displaystyle {\begin{aligned}\operatorname {Pr} \left(\left|(X_{n},Y_{n})-(X,Y)\right|\geq \varepsilon \right)&\leq \operatorname {Pr} \left(|X_{n}-X|+|Y_{n}-Y|\geq \varepsilon \right)\\&\leq \operatorname {Pr} \left(|X_{n}-X|\geq \varepsilon /2\right)+\operatorname {Pr} \left(|Y_{n}-Y|\geq \varepsilon /2\right)\end{aligned}}} <p>where the last step follows by the pigeonhole principle and the sub-additivity of the probability measure. Each of the probabilities on the right-hand side converge to zero as <i>n</i> → ∞ by definition of the convergence of {<i>Xn</i>} and {<i>Yn</i>} in probability to <i>X</i> and <i>Y</i> respectively. Taking the limit we conclude that the left-hand side also converges to zero, and therefore the sequence {(<i>Xn</i>, <i>Yn</i>)} converges in probability to {(<i>X</i>, <i>Y</i>)}. </p> <h2 id="see-also">See also</h2> <ul><li><a href="/facts/Convergence_of_random_variables/pWdatFlY">Convergence of random variables</a></li></ul> <ul><li><a href="/facts/Aad_van_der_Vaart/zEPpqEdW">van der Vaart, Aad W.</a> (1998). <i>Asymptotic statistics</i>. New York: Garrick Ardis. <a href="/facts/ISBN_(identifier)/15AdSPa9">ISBN</a> 978-0-521-49603-2.</li></ul>