• Images
• Videos
• Documents
• Books
• Articles

Single-parameter test

The statistic

Let L {\displaystyle L} be the likelihood function which depends on a univariate parameter θ {\displaystyle \theta } and let x {\displaystyle x} be the data. The score U ( θ ) {\displaystyle U(\theta )} is defined as

U ( θ ) = ∂ log ⁡ L ( θ ∣ x ) ∂ θ . {\displaystyle U(\theta )={\frac {\partial \log L(\theta \mid x)}{\partial \theta }}.}

The Fisher information is⁶

I ( θ ) = − E ⁡ [ ∂ 2 ∂ θ 2 log ⁡ f ( X ; θ ) | θ ] , {\displaystyle I(\theta )=-\operatorname {E} \left[\left.{\frac {\partial ^{2}}{\partial \theta ^{2}}}\log f(X;\theta )\,\right|\,\theta \right]\,,}

where ƒ is the probability density.

The statistic to test H 0 : θ = θ 0 {\displaystyle {\mathcal {H}}_{0}:\theta =\theta _{0}} is S ( θ 0 ) = U ( θ 0 ) 2 I ( θ 0 ) {\displaystyle S(\theta _{0})={\frac {U(\theta _{0})^{2}}{I(\theta _{0})}}}

which has an asymptotic distribution of χ 1 2 {\displaystyle \chi _{1}^{2}} , when H 0 {\displaystyle {\mathcal {H}}_{0}} is true. While asymptotically identical, calculating the LM statistic using the outer-gradient-product estimator of the Fisher information matrix can lead to bias in small samples.⁷

Note on notation

Note that some texts use an alternative notation, in which the statistic S ∗ ( θ ) = S ( θ ) {\displaystyle S^{*}(\theta )={\sqrt {S(\theta )}}} is tested against a normal distribution. This approach is equivalent and gives identical results.

As most powerful test for small deviations

( ∂ log ⁡ L ( θ ∣ x ) ∂ θ ) θ = θ 0 ≥ C {\displaystyle \left({\frac {\partial \log L(\theta \mid x)}{\partial \theta }}\right)_{\theta =\theta _{0}}\geq C}

where L {\displaystyle L} is the likelihood function, θ 0 {\displaystyle \theta _{0}} is the value of the parameter of interest under the null hypothesis, and C {\displaystyle C} is a constant set depending on the size of the test desired (i.e. the probability of rejecting H 0 {\displaystyle H_{0}} if H 0 {\displaystyle H_{0}} is true; see Type I error).

The score test is the most powerful test for small deviations from H 0 {\displaystyle H_{0}} . To see this, consider testing θ = θ 0 {\displaystyle \theta =\theta _{0}} versus θ = θ 0 + h {\displaystyle \theta =\theta _{0}+h} . By the Neyman–Pearson lemma, the most powerful test has the form

L ( θ 0 + h ∣ x ) L ( θ 0 ∣ x ) ≥ K ; {\displaystyle {\frac {L(\theta _{0}+h\mid x)}{L(\theta _{0}\mid x)}}\geq K;}

Taking the log of both sides yields

log ⁡ L ( θ 0 + h ∣ x ) − log ⁡ L ( θ 0 ∣ x ) ≥ log ⁡ K . {\displaystyle \log L(\theta _{0}+h\mid x)-\log L(\theta _{0}\mid x)\geq \log K.}

The score test follows making the substitution (by Taylor series expansion)

log ⁡ L ( θ 0 + h ∣ x ) ≈ log ⁡ L ( θ 0 ∣ x ) + h × ( ∂ log ⁡ L ( θ ∣ x ) ∂ θ ) θ = θ 0 {\displaystyle \log L(\theta _{0}+h\mid x)\approx \log L(\theta _{0}\mid x)+h\times \left({\frac {\partial \log L(\theta \mid x)}{\partial \theta }}\right)_{\theta =\theta _{0}}}

and identifying the C {\displaystyle C} above with log ⁡ ( K ) {\displaystyle \log(K)} .

Relationship with other hypothesis tests

If the null hypothesis is true, the likelihood ratio test, the Wald test, and the Score test are asymptotically equivalent tests of hypotheses.⁸⁹ When testing nested models, the statistics for each test then converge to a Chi-squared distribution with degrees of freedom equal to the difference in degrees of freedom in the two models. If the null hypothesis is not true, however, the statistics converge to a noncentral chi-squared distribution with possibly different noncentrality parameters.

Multiple parameters

A more general score test can be derived when there is more than one parameter. Suppose that θ ^ 0 {\displaystyle {\widehat {\theta }}_{0}} is the maximum likelihood estimate of θ {\displaystyle \theta } under the null hypothesis H 0 {\displaystyle H_{0}} while U {\displaystyle U} and I {\displaystyle I} are respectively, the score vector and the Fisher information matrix. Then

U T ( θ ^ 0 ) I − 1 ( θ ^ 0 ) U ( θ ^ 0 ) ∼ χ k 2 {\displaystyle U^{T}({\widehat {\theta }}_{0})I^{-1}({\widehat {\theta }}_{0})U({\widehat {\theta }}_{0})\sim \chi _{k}^{2}}

asymptotically under H 0 {\displaystyle H_{0}} , where k {\displaystyle k} is the number of constraints imposed by the null hypothesis and

U ( θ ^ 0 ) = ∂ log ⁡ L ( θ ^ 0 ∣ x ) ∂ θ {\displaystyle U({\widehat {\theta }}_{0})={\frac {\partial \log L({\widehat {\theta }}_{0}\mid x)}{\partial \theta }}}

and

I ( θ ^ 0 ) = − E ⁡ ( ∂ 2 log ⁡ L ( θ ^ 0 ∣ x ) ∂ θ ∂ θ ′ ) . {\displaystyle I({\widehat {\theta }}_{0})=-\operatorname {E} \left({\frac {\partial ^{2}\log L({\widehat {\theta }}_{0}\mid x)}{\partial \theta \,\partial \theta '}}\right).}

This can be used to test H 0 {\displaystyle H_{0}} .

The actual formula for the test statistic depends on which estimator of the Fisher information matrix is being used.¹⁰

Special cases

In many situations, the score statistic reduces to another commonly used statistic.¹¹

In linear regression, the Lagrange multiplier test can be expressed as a function of the F-test.¹²

When the data follows a normal distribution, the score statistic is the same as the t statistic.

When the data consists of binary observations, the score statistic is the same as the chi-squared statistic in the Pearson's chi-squared test.

See also

• Fisher information
• Uniformly most powerful test
• Score (statistics)
• Sup-LM test

Further reading

• Buse, A. (1982). "The Likelihood Ratio, Wald, and Lagrange Multiplier Tests: An Expository Note". The American Statistician. 36 (3a): 153–157. doi:10.1080/00031305.1982.10482817.
• Godfrey, L. G. (1988). "The Lagrange Multiplier Test and Testing for Misspecification : An Extended Analysis". Misspecification Tests in Econometrics. New York: Cambridge University Press. pp. 69–99. ISBN 0-521-26616-5.
• Ma, Jun; Nelson, Charles R. (2016). "The superiority of the LM test in a class of econometric models where the Wald test performs poorly". Unobserved Components and Time Series Econometrics. Oxford University Press. pp. 310–330. doi:10.1093/acprof:oso/9780199683666.003.0014. ISBN 978-0-19-968366-6.
• Rao, C. R. (2005). "Score Test: Historical Review and Recent Developments". Advances in Ranking and Selection, Multiple Comparisons, and Reliability. Boston: Birkhäuser. pp. 3–20. ISBN 978-0-8176-3232-8.

References

1. Rao, C. Radhakrishna (1948). "Large sample tests of statistical hypotheses concerning several parameters with applications to problems of estimation". Mathematical Proceedings of the Cambridge Philosophical Society. 44 (1): 50–57. Bibcode:1948PCPS...44...50R. doi:10.1017/S0305004100023987. /wiki/Mathematical_Proceedings_of_the_Cambridge_Philosophical_Society ↩

2. Silvey, S. D. (1959). "The Lagrangian Multiplier Test". Annals of Mathematical Statistics. 30 (2): 389–407. doi:10.1214/aoms/1177706259. JSTOR 2237089. https://doi.org/10.1214%2Faoms%2F1177706259 ↩

3. Breusch, T. S.; Pagan, A. R. (1980). "The Lagrange Multiplier Test and its Applications to Model Specification in Econometrics". Review of Economic Studies. 47 (1): 239–253. doi:10.2307/2297111. JSTOR 2297111. /wiki/Trevor_S._Breusch ↩

4. Fahrmeir, Ludwig; Kneib, Thomas; Lang, Stefan; Marx, Brian (2013). Regression : Models, Methods and Applications. Berlin: Springer. pp. 663–664. ISBN 978-3-642-34332-2. 978-3-642-34332-2 ↩

5. Kennedy, Peter (1998). A Guide to Econometrics (Fourth ed.). Cambridge: MIT Press. p. 68. ISBN 0-262-11235-3. 0-262-11235-3 ↩

6. Lehmann and Casella, eq. (2.5.16). ↩

7. Davidson, Russel; MacKinnon, James G. (1983). "Small sample properties of alternative forms of the Lagrange Multiplier test". Economics Letters. 12 (3–4): 269–275. doi:10.1016/0165-1765(83)90048-4. /wiki/Economics_Letters ↩

8. Engle, Robert F. (1983). "Wald, Likelihood Ratio, and Lagrange Multiplier Tests in Econometrics". In Intriligator, M. D.; Griliches, Z. (eds.). Handbook of Econometrics. Vol. II. Elsevier. pp. 796–801. ISBN 978-0-444-86185-6. 978-0-444-86185-6 ↩

9. Burzykowski, Andrzej Gałecki, Tomasz (2013). Linear mixed-effects models using R : a step-by-step approach. New York, NY: Springer. ISBN 978-1-4614-3899-1.{{cite book}}: CS1 maint: multiple names: authors list (link) 978-1-4614-3899-1 ↩

10. Taboga, Marco. "Lectures on Probability Theory and Mathematical Statistics". statlect.com. Retrieved 31 May 2022. https://www.statlect.com/fundamentals-of-statistics/score-test ↩

11. Cook, T. D.; DeMets, D. L., eds. (2007). Introduction to Statistical Methods for Clinical Trials. Chapman and Hall. pp. 296–297. ISBN 978-1-58488-027-1. 978-1-58488-027-1 ↩

12. Vandaele, Walter (1981). "Wald, likelihood ratio, and Lagrange multiplier tests as an F test". Economics Letters. 8 (4): 361–365. doi:10.1016/0165-1765(81)90026-4. /wiki/Economics_Letters ↩