The jackknife estimator of a parameter is found by systematically leaving out each observation from a dataset and calculating the parameter estimate over the remaining observations and then aggregating these calculations.
For example, if the parameter to be estimated is the population mean of random variable x {\displaystyle x} , then for a given set of i.i.d. observations x 1 , . . . , x n {\displaystyle x_{1},...,x_{n}} the natural estimator is the sample mean:
where the last sum used another way to indicate that the index i {\displaystyle i} runs over the set [ n ] = { 1 , … , n } {\displaystyle [n]=\{1,\ldots ,n\}} .
Then we proceed as follows: For each i ∈ [ n ] {\displaystyle i\in [n]} we compute the mean x ¯ ( i ) {\displaystyle {\bar {x}}_{(i)}} of the jackknife subsample consisting of all but the i {\displaystyle i} -th data point, and this is called the i {\displaystyle i} -th jackknife replicate:
It could help to think that these n {\displaystyle n} jackknife replicates x ¯ ( 1 ) , … , x ¯ ( n ) {\displaystyle {\bar {x}}_{(1)},\ldots ,{\bar {x}}_{(n)}} approximate the distribution of the sample mean x ¯ {\displaystyle {\bar {x}}} . A larger n {\displaystyle n} improves the approximation. Then finally to get the jackknife estimator, the n {\displaystyle n} jackknife replicates are averaged:
One may ask about the bias and the variance of x ¯ j a c k {\displaystyle {\bar {x}}_{\mathrm {jack} }} . From the definition of x ¯ j a c k {\displaystyle {\bar {x}}_{\mathrm {jack} }} as the average of the jackknife replicates one could try to calculate explicitly. The bias is a trivial calculation, but the variance of x ¯ j a c k {\displaystyle {\bar {x}}_{\mathrm {jack} }} is more involved since the jackknife replicates are not independent.
For the special case of the mean, one can show explicitly that the jackknife estimate equals the usual estimate:
This establishes the identity x ¯ j a c k = x ¯ {\displaystyle {\bar {x}}_{\mathrm {jack} }={\bar {x}}} . Then taking expectations we get E [ x ¯ j a c k ] = E [ x ¯ ] = E [ x ] {\displaystyle E[{\bar {x}}_{\mathrm {jack} }]=E[{\bar {x}}]=E[x]} , so x ¯ j a c k {\displaystyle {\bar {x}}_{\mathrm {jack} }} is unbiased, while taking variance we get V [ x ¯ j a c k ] = V [ x ¯ ] = V [ x ] / n {\displaystyle V[{\bar {x}}_{\mathrm {jack} }]=V[{\bar {x}}]=V[x]/n} . However, these properties do not generally hold for parameters other than the mean.
This simple example for the case of mean estimation is just to illustrate the construction of a jackknife estimator, while the real subtleties (and the usefulness) emerge for the case of estimating other parameters, such as higher moments than the mean or other functionals of the distribution.
x ¯ j a c k {\displaystyle {\bar {x}}_{\mathrm {jack} }} could be used to construct an empirical estimate of the bias of x ¯ {\displaystyle {\bar {x}}} , namely bias ^ ( x ¯ ) j a c k = c ( x ¯ j a c k − x ¯ ) {\displaystyle {\widehat {\operatorname {bias} }}({\bar {x}})_{\mathrm {jack} }=c({\bar {x}}_{\mathrm {jack} }-{\bar {x}})} with some suitable factor c > 0 {\displaystyle c>0} , although in this case we know that x ¯ j a c k = x ¯ {\displaystyle {\bar {x}}_{\mathrm {jack} }={\bar {x}}} so this construction does not add any meaningful knowledge, but it gives the correct estimation of the bias (which is zero).
A jackknife estimate of the variance of x ¯ {\displaystyle {\bar {x}}} can be calculated from the variance of the jackknife replicates x ¯ ( i ) {\displaystyle {\bar {x}}_{(i)}} :45
The left equality defines the estimator var ^ ( x ¯ ) j a c k {\displaystyle {\widehat {\operatorname {var} }}({\bar {x}})_{\mathrm {jack} }} and the right equality is an identity that can be verified directly. Then taking expectations we get E [ var ^ ( x ¯ ) j a c k ] = V [ x ] / n = V [ x ¯ ] {\displaystyle E[{\widehat {\operatorname {var} }}({\bar {x}})_{\mathrm {jack} }]=V[x]/n=V[{\bar {x}}]} , so this is an unbiased estimator of the variance of x ¯ {\displaystyle {\bar {x}}} .
The jackknife technique can be used to estimate (and correct) the bias of an estimator calculated over the entire sample.
Suppose θ {\displaystyle \theta } is the target parameter of interest, which is assumed to be some functional of the distribution of x {\displaystyle x} . Based on a finite set of observations x 1 , . . . , x n {\displaystyle x_{1},...,x_{n}} , which is assumed to consist of i.i.d. copies of x {\displaystyle x} , the estimator θ ^ {\displaystyle {\hat {\theta }}} is constructed:
The value of θ ^ {\displaystyle {\hat {\theta }}} is sample-dependent, so this value will change from one random sample to another.
By definition, the bias of θ ^ {\displaystyle {\hat {\theta }}} is as follows:
One may wish to compute several values of θ ^ {\displaystyle {\hat {\theta }}} from several samples, and average them, to calculate an empirical approximation of E [ θ ^ ] {\displaystyle E[{\hat {\theta }}]} , but this is impossible when there are no "other samples" when the entire set of available observations x 1 , . . . , x n {\displaystyle x_{1},...,x_{n}} was used to calculate θ ^ {\displaystyle {\hat {\theta }}} . In this kind of situation the jackknife resampling technique may be of help.
We construct the jackknife replicates:
where each replicate is a "leave-one-out" estimate based on the jackknife subsample consisting of all but one of the data points:
Then we define their average:
The jackknife estimate of the bias of θ ^ {\displaystyle {\hat {\theta }}} is given by:
and the resulting bias-corrected jackknife estimate of θ {\displaystyle \theta } is given by:
This removes the bias in the special case that the bias is O ( n − 1 ) {\displaystyle O(n^{-1})} and reduces it to O ( n − 2 ) {\displaystyle O(n^{-2})} in other cases.6
The jackknife technique can be also used to estimate the variance of an estimator calculated over the entire sample.
Efron 1982, p. 2. - Efron, Bradley (1982). The jackknife, the bootstrap, and other resampling plans. Philadelphia, PA: Society for Industrial and Applied Mathematics. ISBN 9781611970319. ↩
Cameron & Trivedi 2005, p. 375. - Cameron, Adrian; Trivedi, Pravin K. (2005). Microeconometrics : methods and applications. Cambridge New York: Cambridge University Press. ISBN 9780521848053. ↩
Efron 1982, p. 14. - Efron, Bradley (1982). The jackknife, the bootstrap, and other resampling plans. Philadelphia, PA: Society for Industrial and Applied Mathematics. ISBN 9781611970319. ↩
McIntosh, Avery I. "The Jackknife Estimation Method" (PDF). Boston University. Avery I. McIntosh. Archived from the original (PDF) on 2016-05-14. Retrieved 2016-04-30.: p. 3. https://web.archive.org/web/20160514022307/http://people.bu.edu/aimcinto/jackknife.pdf ↩