Menu
Home
People
Places
Arts
History
Plants & Animals
Science
Life & Culture
Technology
Reference.org
Consistent estimator
open-in-new
<h2 id="definition">Definition</h2> <p>Formally speaking, an <a href="/facts/Estimator/CbkjcKvN">estimator</a> <i>Tn</i> of parameter <i>θ</i> is said to be weakly consistent, if it <a href="/facts/Convergence_in_probability/pWdatFlY">converges in probability</a> to the true value of the parameter:<a class="footnote-ref" id="fnref:1" href="#fn:1"><sup>1</sup></a> </p> plim n → ∞ T n = θ . {\displaystyle {\underset {n\to \infty }{\operatorname {plim} }}\;T_{n}=\theta .} <p>i.e. if, for all <i>ε</i> > 0 </p> lim n → ∞ Pr ( | T n − θ | > ε ) = 0. {\displaystyle \lim _{n\to \infty }\Pr {\big (}|T_{n}-\theta |>\varepsilon {\big )}=0.} <p>An <a href="/facts/Estimator/CbkjcKvN">estimator</a> <i>Tn</i> of parameter <i>θ</i> is said to be strongly consistent, if it converges almost surely to the true value of the parameter: </p> Pr ( lim n → ∞ T n = θ ) = 1. {\displaystyle \Pr {\big (}\lim _{n\to \infty }T_{n}=\theta {\big )}=1.} <p>A more rigorous definition takes into account the fact that <i>θ</i> is actually unknown, and thus, the convergence in probability must take place for every possible value of this parameter. Suppose {<i>pθ</i>: <i>θ</i> ∈ Θ} is a family of distributions (the <a href="/facts/Parametric_model/FeynAa0f">parametric model</a>), and <i>Xθ</i> = {<i>X</i>1, <i>X</i>2, … : <i>Xi</i> ~ <i>pθ</i>} is an infinite <a href="/facts/Statistical_sample/URCwKNge">sample</a> from the distribution <i>pθ</i>. Let { <i>Tn</i>(<i>Xθ</i>) } be a sequence of estimators for some parameter <i>g</i>(<i>θ</i>). Usually, <i>Tn</i> will be based on the first <i>n</i> observations of a sample. Then this sequence {<i>Tn</i>} is said to be (weakly) consistent if <a class="footnote-ref" id="fnref:2" href="#fn:2"><sup>2</sup></a> </p> plim n → ∞ T n ( X θ ) = g ( θ ) , for all θ ∈ Θ . {\displaystyle {\underset {n\to \infty }{\operatorname {plim} }}\;T_{n}(X^{\theta })=g(\theta ),\ \ {\text{for all}}\ \theta \in \Theta .} <p>This definition uses <i>g</i>(<i>θ</i>) instead of simply <i>θ</i>, because often one is interested in estimating a certain function or a sub-vector of the underlying parameter. In the next example, we estimate the location parameter of the model, but not the scale: </p> <h2 id="examples">Examples</h2> <h3>Sample mean of a normal random variable</h3> <p>Suppose one has a sequence of <a href="/facts/Independence_(probability_theory)/NUzQtnUL">statistically independent</a> observations {<i>X</i>1, <i>X</i>2, ...} from a <a href="/facts/Normal_distribution/UapjjPyQ">normal <i>N</i>(<i>μ</i>, <i>σ</i>2)</a> distribution. To estimate <i>μ</i> based on the first <i>n</i> observations, one can use the <a href="/facts/Sample_mean/Ah8VVVDT">sample mean</a>: <i>Tn</i> = (<i>X</i>1 + ... + <i>Xn</i>)/<i>n</i>. This defines a sequence of estimators, indexed by the sample size <i>n</i>. </p><p>From the properties of the normal distribution, we know the <a href="/facts/Sampling_distribution/50dJL1ja">sampling distribution</a> of this statistic: <i>T</i><i>n</i> is itself normally distributed, with mean <i>μ</i> and variance <i>σ</i>2/<i>n</i>. Equivalently, ( T n − μ ) / ( σ / n ) {\displaystyle \scriptstyle (T_{n}-\mu )/(\sigma /{\sqrt {n}})} has a standard normal distribution: </p> Pr [ | T n − μ | ≥ ε ] = Pr [ n | T n − μ | σ ≥ n ε / σ ] = 2 ( 1 − Φ ( n ε σ ) ) → 0 {\displaystyle \Pr \!\left[\,|T_{n}-\mu |\geq \varepsilon \,\right]=\Pr \!\left[{\frac {{\sqrt {n}}\,{\big |}T_{n}-\mu {\big |}}{\sigma }}\geq {\sqrt {n}}\varepsilon /\sigma \right]=2\left(1-\Phi \left({\frac {{\sqrt {n}}\,\varepsilon }{\sigma }}\right)\right)\to 0} <p>as <i>n</i> tends to infinity, for any fixed <i>ε</i> > 0. Therefore, the sequence <i>Tn</i> of sample means is consistent for the population mean <i>μ</i> (recalling that Φ {\displaystyle \Phi } is the <a href="/facts/Cumulative_distribution_function/WaKU8tp4">cumulative distribution</a> of the standard normal distribution). </p> <h2 id="establishing-consistency">Establishing consistency</h2> <p>The notion of asymptotic consistency is very close, almost synonymous to the notion of convergence in probability. As such, any theorem, lemma, or property which establishes convergence in probability may be used to prove the consistency. Many such tools exist: </p> <ul><li>In order to demonstrate consistency directly from the definition one can use the inequality <a class="footnote-ref" id="fnref:3" href="#fn:3"><sup>3</sup></a></li></ul> Pr [ h ( T n − θ ) ≥ ε ] ≤ E [ h ( T n − θ ) ] h ( ε ) , {\displaystyle \Pr \!{\big [}h(T_{n}-\theta )\geq \varepsilon {\big ]}\leq {\frac {\operatorname {E} {\big [}h(T_{n}-\theta ){\big ]}}{h(\varepsilon )}},} <p>the most common choice for function <i>h</i> being either the absolute value (in which case it is known as <a href="/facts/Markov_inequality/Re85kMQJ">Markov inequality</a>), or the quadratic function (respectively <a href="/facts/Chebyshev%27s_inequality/jf5ReDJv">Chebyshev's inequality</a>). </p> <ul><li>Another useful result is the <a href="/facts/Continuous_mapping_theorem/YEauUuWS">continuous mapping theorem</a>: if <i>Tn</i> is consistent for <i>θ</i> and <i>g</i>(·) is a real-valued function continuous at the point <i>θ</i>, then <i>g</i>(<i>Tn</i>) will be consistent for <i>g</i>(<i>θ</i>):<a class="footnote-ref" id="fnref:4" href="#fn:4"><sup>4</sup></a></li></ul> T n → p θ ⇒ g ( T n ) → p g ( θ ) {\displaystyle T_{n}\ {\xrightarrow {p}}\ \theta \ \quad \Rightarrow \quad g(T_{n})\ {\xrightarrow {p}}\ g(\theta )} <ul><li><a href="/facts/Slutsky%27s_theorem/Fwh5f8Wt">Slutsky's theorem</a> can be used to combine several different estimators, or an estimator with a non-random convergent sequence. If <i>Tn</i> →<i>d</i><i>α</i>, and <i>Sn</i> →<i>p</i><i>β</i>, then <a class="footnote-ref" id="fnref:5" href="#fn:5"><sup>5</sup></a></li></ul> T n + S n → d α + β , T n S n → d α β , T n / S n → d α / β , provided that β ≠ 0 {\displaystyle {\begin{aligned}&T_{n}+S_{n}\ {\xrightarrow {d}}\ \alpha +\beta ,\\&T_{n}S_{n}\ {\xrightarrow {d}}\ \alpha \beta ,\\&T_{n}/S_{n}\ {\xrightarrow {d}}\ \alpha /\beta ,{\text{ provided that }}\beta \neq 0\end{aligned}}} <ul><li>If estimator <i>Tn</i> is given by an explicit formula, then most likely the formula will employ sums of random variables, and then the <a href="/facts/Law_of_large_numbers/X3Bjcy3v">law of large numbers</a> can be used: for a sequence {<i>Xn</i>} of random variables and under suitable conditions,</li></ul> 1 n ∑ i = 1 n g ( X i ) → p E [ g ( X ) ] {\displaystyle {\frac {1}{n}}\sum _{i=1}^{n}g(X_{i})\ {\xrightarrow {p}}\ \operatorname {E} [\,g(X)\,]} <ul><li>If estimator <i>Tn</i> is defined implicitly, for example as a value that maximizes certain objective function (see <a href="/facts/Extremum_estimator/BuRVcBQc">extremum estimator</a>), then a more complicated argument involving <a href="/facts/Stochastic_equicontinuity/Nhrornn1">stochastic equicontinuity</a> has to be used.<a class="footnote-ref" id="fnref:6" href="#fn:6"><sup>6</sup></a></li></ul> <h2 id="bias-versus-consistency">Bias versus consistency</h2> <h3>Unbiased but not consistent</h3> <p>An estimator can be <a href="/facts/Biased_estimator/oxIvEgmd">unbiased</a> but not consistent. For example, for an <a href="/facts/Iid/othIRaWt">iid</a> sample {<i>x</i>1,..., <i>xn</i>} one can use <i>Tn</i>(<i>X</i>) = <i>x</i>n as the estimator of the mean E[<i>X</i>]. Note that here the sampling distribution of <i>Tn</i> is the same as the underlying distribution (for any <i>n,</i> as it ignores all points but the last). So E[<i>Tn</i>(<i>X</i>)] = E[<i>X</i>] for any <i>n,</i> hence it is unbiased, but it does not converge to any value. </p><p>However, if a sequence of estimators is unbiased <i>and</i> converges to a value, then it is consistent, as it must converge to the correct value. </p> <h3>Biased but consistent</h3> <p>Alternatively, an estimator can be biased but consistent. For example, if the mean is estimated by 1 n ∑ x i + 1 n {\displaystyle {1 \over n}\sum x_{i}+{1 \over n}} it is biased, but as n → ∞ {\displaystyle n\rightarrow \infty } , it approaches the correct value, and so it is consistent. </p><p>Important examples include the <a href="/facts/Sample_variance/ULBJKXD1">sample variance</a> and <a href="/facts/Sample_standard_deviation/qSug9BRl">sample standard deviation</a>. Without <a href="/facts/Bessel%27s_correction/VgVzNomA">Bessel's correction</a> (that is, when using the sample size n {\displaystyle n} instead of the <a href="/facts/Degrees_of_freedom_(statistics)/fvwKCU86">degrees of freedom</a> n − 1 {\displaystyle n-1} ), these are both negatively biased but consistent estimators. With the correction, the corrected sample variance is unbiased, while the corrected sample standard deviation is still biased, but less so, and both are still consistent: the correction factor converges to 1 as sample size grows. </p><p>Here is another example. Let T n {\displaystyle T_{n}} be a sequence of estimators for θ {\displaystyle \theta } . </p> Pr ( T n ) = { 1 − 1 / n , if T n = θ 1 / n , if T n = n δ + θ {\displaystyle \Pr(T_{n})={\begin{cases}1-1/n,&{\mbox{if }}\,T_{n}=\theta \\1/n,&{\mbox{if }}\,T_{n}=n\delta +\theta \end{cases}}} <p>We can see that T n → p θ {\displaystyle T_{n}{\xrightarrow {p}}\theta } , E [ T n ] = θ + δ {\displaystyle \operatorname {E} [T_{n}]=\theta +\delta } , and the bias does not converge to zero. </p> <h2 id="see-also">See also</h2> <ul><li><a href="/facts/Efficient_estimator/wTmgEqte">Efficient estimator</a></li> <li><a href="/facts/Fisher_consistency/yH7jWoF3">Fisher consistency</a> — alternative, although rarely used concept of consistency for the estimators</li> <li><a href="/facts/Regression_dilution/yshVFGzE">Regression dilution</a></li> <li><a href="/facts/Statistical_hypothesis_testing/yv9RPF6U">Statistical hypothesis testing</a></li> <li><a href="/facts/Instrumental_variables_estimation/cEQ1xwT6">Instrumental variables estimation</a></li></ul> <h2 id="notes">Notes</h2> <ul><li><a href="/facts/Takeshi_Amemiya/CVByAXGq">Amemiya, Takeshi</a> (1985). <a href="https://archive.org/details/advancedeconomet00amem"><i>Advanced Econometrics</i></a>. <a href="/facts/Harvard_University_Press/YluAnq74">Harvard University Press</a>. <a href="/facts/ISBN_(identifier)/15AdSPa9">ISBN</a> 0-674-00560-0.</li> <li><a href="/facts/Erich_Leo_Lehmann/hun7hqhM">Lehmann, E. L.</a>; <a href="/facts/George_Casella/CTjcaJCt">Casella, G.</a> (1998). <i>Theory of Point Estimation</i> (2nd ed.). Springer. <a href="/facts/ISBN_(identifier)/15AdSPa9">ISBN</a> 0-387-98502-6.</li> <li>Newey, W. K.; <a href="/facts/Daniel_McFadden/VfXVLVot">McFadden, D.</a> (1994). "Chapter 36: Large sample estimation and hypothesis testing". In Robert F. Engle; Daniel L. McFadden (eds.). <i>Handbook of Econometrics</i>. Vol. 4. Elsevier Science. <a href="/facts/ISBN_(identifier)/15AdSPa9">ISBN</a> 0-444-88766-0. <a href="/facts/S2CID_(identifier)/ldJsHa2Y">S2CID</a> <a href="https://api.semanticscholar.org/CorpusID:29436457">29436457</a>.</li> <li>Nikulin, M. S. (2001) [1994], <a href="https://www.encyclopediaofmath.org/index.php?title=Consistent_estimator">"Consistent estimator"</a>, <i><a href="/facts/Encyclopedia_of_Mathematics/WC6mGtPm">Encyclopedia of Mathematics</a></i>, <a href="/facts/European_Mathematical_Society/B3h7b672">EMS Press</a></li> <li><a href="/facts/Elliott_Sober/xQm61uC4">Sober, E.</a> (1988), "Likelihood and convergence", <i><a href="/facts/Philosophy_of_Science/aonsfhTY">Philosophy of Science</a></i>, 55 (2): 228–237, <a href="/facts/Doi_(identifier)/muM9Etpq">doi</a>:<a href="https://doi.org/10.1086%2F289429">10.1086/289429</a>.</li></ul> <h2 id="external-links">External links</h2> <ul><li><a href="https://www.youtube.com/watch?v=TfuqBxRgRTU&list=PLD15D38DC7AA3B737&index=3#t=32m00m">Econometrics lecture (topic: unbiased vs. consistent)</a> on <a href="/facts/YouTube_video_(identifier)/db276jst">YouTube</a> by <a href="/facts/Mark_Thoma/ThV29HB0">Mark Thoma</a></li></ul> <h2 id="references">References</h2> <ol> <li id="fn:1"><p>Amemiya 1985, Definition 3.4.2. - Amemiya, Takeshi (1985). Advanced Econometrics. Harvard University Press. ISBN 0-674-00560-0. <a href="https://archive.org/details/advancedeconomet00amem" target="_blank">https://archive.org/details/advancedeconomet00amem</a> <a href="#fnref:1" class="footnote-back-ref">↩</a></p></li> <li id="fn:2"><p>Lehman & Casella 1998, p. 332. - Lehmann, E. L.; Casella, G. (1998). Theory of Point Estimation (2nd ed.). Springer. ISBN 0-387-98502-6. <a href="#fnref:2" class="footnote-back-ref">↩</a></p></li> <li id="fn:3"><p>Amemiya 1985, equation (3.2.5). - Amemiya, Takeshi (1985). Advanced Econometrics. Harvard University Press. ISBN 0-674-00560-0. <a href="https://archive.org/details/advancedeconomet00amem" target="_blank">https://archive.org/details/advancedeconomet00amem</a> <a href="#fnref:3" class="footnote-back-ref">↩</a></p></li> <li id="fn:4"><p>Amemiya 1985, Theorem 3.2.6. - Amemiya, Takeshi (1985). Advanced Econometrics. Harvard University Press. ISBN 0-674-00560-0. <a href="https://archive.org/details/advancedeconomet00amem" target="_blank">https://archive.org/details/advancedeconomet00amem</a> <a href="#fnref:4" class="footnote-back-ref">↩</a></p></li> <li id="fn:5"><p>Amemiya 1985, Theorem 3.2.7. - Amemiya, Takeshi (1985). Advanced Econometrics. Harvard University Press. ISBN 0-674-00560-0. <a href="https://archive.org/details/advancedeconomet00amem" target="_blank">https://archive.org/details/advancedeconomet00amem</a> <a href="#fnref:5" class="footnote-back-ref">↩</a></p></li> <li id="fn:6"><p>Newey & McFadden 1994, Chapter 2. - Newey, W. K.; McFadden, D. (1994). "Chapter 36: Large sample estimation and hypothesis testing". In Robert F. Engle; Daniel L. McFadden (eds.). Handbook of Econometrics. Vol. 4. Elsevier Science. ISBN 0-444-88766-0. S2CID 29436457. <a href="https://api.semanticscholar.org/CorpusID:29436457" target="_blank">https://api.semanticscholar.org/CorpusID:29436457</a> <a href="#fnref:6" class="footnote-back-ref">↩</a></p></li> </ol>