Ordinary least squares

<h2 id="linear-model">Linear model</h2>
Main article: <a href="/facts/Linear_regression_model/5n998IhK">Linear regression model</a>
Suppose the data consists of 
 
 
 
 n
 
 
 {\displaystyle n}
 
 <a href="/facts/Statistical_unit/oIW2uEBb">observations</a> 
 
 
 
 
 
 {
 
 
 
 x
 
 
 i
 
 
 ,
 
 y
 
 i
 
 
 
 }
 
 
 i
 =
 1
 
 
 n
 
 
 
 
 {\displaystyle \left\{\mathbf {x} _{i},y_{i}\right\}_{i=1}^{n}}
 
. Each observation 
 
 
 
 i
 
 
 {\displaystyle i}
 
 includes a scalar response 
 
 
 
 
 y
 
 i
 
 
 
 
 {\displaystyle y_{i}}
 
 and a column vector 
 
 
 
 
 
 x
 
 
 i
 
 
 
 
 {\displaystyle \mathbf {x} _{i}}
 
 of 
 
 
 
 p
 
 
 {\displaystyle p}
 
 parameters (regressors), i.e., 
 
 
 
 
 
 x
 
 
 i
 
 
 =
 
 
 [
 
 
 x
 
 i
 1
 
 
 ,
 
 x
 
 i
 2
 
 
 ,
 …
 ,
 
 x
 
 i
 p
 
 
 
 ]
 
 
 T
 
 
 
 
 {\displaystyle \mathbf {x} _{i}=\left[x_{i1},x_{i2},\dots ,x_{ip}\right]^{\operatorname {T} }}
 
. In a <a href="/facts/Linear_regression_model/5n998IhK">linear regression model</a>, the response variable, 
 
 
 
 
 y
 
 i
 
 
 
 
 {\displaystyle y_{i}}
 
, is a linear function of the regressors:

y
          
            i
          
        
        =
        
          β
          
            1
          
        
         
        
          x
          
            i
            1
          
        
        +
        
          β
          
            2
          
        
         
        
          x
          
            i
            2
          
        
        +
        ⋯
        +
        
          β
          
            p
          
        
         
        
          x
          
            i
            p
          
        
        +
        
          ε
          
            i
          
        
        ,
      
    
    {\displaystyle y_{i}=\beta _{1}\ x_{i1}+\beta _{2}\ x_{i2}+\cdots +\beta _{p}\ x_{ip}+\varepsilon _{i},}

or in <a href="/facts/Row_and_column_vectors/pPuRr9Xk">vector</a> form,

y
          
            i
          
        
        =
        
          
            x
          
          
            i
          
          
            T
          
        
        
          β
        
        +
        
          ε
          
            i
          
        
        ,
        
      
    
    {\displaystyle y_{i}=\mathbf {x} _{i}^{\operatorname {T} }{\boldsymbol {\beta }}+\varepsilon _{i},\,}

where 
 
 
 
 
 
 x
 
 
 i
 
 
 
 
 {\displaystyle \mathbf {x} _{i}}
 
, as introduced previously, is a column vector of the 
 
 
 
 i
 
 
 {\displaystyle i}
 
-th observation of all the explanatory variables; 
 
 
 
 
 β
 
 
 
 {\displaystyle {\boldsymbol {\beta }}}
 
 is a 
 
 
 
 p
 ×
 1
 
 
 {\displaystyle p\times 1}
 
 vector of unknown parameters; and the scalar 
 
 
 
 
 ε
 
 i
 
 
 
 
 {\displaystyle \varepsilon _{i}}
 
 represents unobserved random variables (<a href="/facts/Errors_and_residuals_in_statistics/Ef0LgAS4">errors</a>) of the 
 
 
 
 i
 
 
 {\displaystyle i}
 
-th observation. 
 
 
 
 
 ε
 
 i
 
 
 
 
 {\displaystyle \varepsilon _{i}}
 
 accounts for the influences upon the responses 
 
 
 
 
 y
 
 i
 
 
 
 
 {\displaystyle y_{i}}
 
 from sources other than the explanatory variables 
 
 
 
 
 
 x
 
 
 i
 
 
 
 
 {\displaystyle \mathbf {x} _{i}}
 
. This model can also be written in matrix notation as

y
        
        =
        
          X
        
        
          β
        
        +
        
          ε
        
        ,
        
      
    
    {\displaystyle \mathbf {y} =\mathbf {X} {\boldsymbol {\beta }}+{\boldsymbol {\varepsilon }},\,}

where 
 
 
 
 
 y
 
 
 
 {\displaystyle \mathbf {y} }
 
 and 
 
 
 
 
 ε
 
 
 
 {\displaystyle {\boldsymbol {\varepsilon }}}
 
 are 
 
 
 
 n
 ×
 1
 
 
 {\displaystyle n\times 1}
 
 vectors of the response variables and the errors of the 
 
 
 
 n
 
 
 {\displaystyle n}
 
 observations, and 
 
 
 
 
 X
 
 
 
 {\displaystyle \mathbf {X} }
 
 is an 
 
 
 
 n
 ×
 p
 
 
 {\displaystyle n\times p}
 
 matrix of regressors, also sometimes called the <a href="/facts/Design_matrix/yRTdMhf4">design matrix</a>, whose row 
 
 
 
 i
 
 
 {\displaystyle i}
 
 is 
 
 
 
 
 
 x
 
 
 i
 
 
 T
 
 
 
 
 {\displaystyle \mathbf {x} _{i}^{\operatorname {T} }}
 
 and contains the 
 
 
 
 i
 
 
 {\displaystyle i}
 
-th observations on all the explanatory variables.
Typically, a constant term is included in the set of regressors 
 
 
 
 
 X
 
 
 
 {\displaystyle \mathbf {X} }
 
, say, by taking 
 
 
 
 
 x
 
 i
 1
 
 
 =
 1
 
 
 {\displaystyle x_{i1}=1}
 
 for all 
 
 
 
 i
 =
 1
 ,
 …
 ,
 n
 
 
 {\displaystyle i=1,\dots ,n}
 
. The coefficient 
 
 
 
 
 β
 
 1
 
 
 
 
 {\displaystyle \beta _{1}}
 
 corresponding to this regressor is called the intercept. Without the intercept, the fitted line is forced to cross the origin when 
 
 
 
 
 x
 
 i
 
 
 =
 
 
 
 0
 →
 
 
 
 
 
 {\displaystyle x_{i}={\vec {0}}}
 
.
Regressors do not have to be independent for estimation to be consistent e.g. they may be non-linearly dependent. Short of perfect multicollinearity, parameter estimates may still be consistent; however, as multicollinearity rises the standard error around such estimates increases and reduces the precision of such estimates. When there is perfect multicollinearity, it is no longer possible to obtain unique estimates for the coefficients to the related regressors; estimation for these parameters cannot converge (thus, it cannot be consistent). 
As a concrete example where regressors are non-linearly dependent yet estimation may still be consistent, we might suspect the response depends linearly both on a value and its square; in which case we would include one regressor whose value is just the square of another regressor. In that case, the model would be quadratic in the second regressor, but none-the-less is still considered a linear model because the model is still linear in the parameters (
 
 
 
 
 β
 
 
 
 {\displaystyle {\boldsymbol {\beta }}}
 
).

<h3>Matrix/vector formulation</h3>
Consider an <a href="/facts/Overdetermined_system/DMWk4pQR">overdetermined system</a>

∑
          
            j
            =
            1
          
          
            p
          
        
        
          x
          
            i
            j
          
        
        
          β
          
            j
          
        
        =
        
          y
          
            i
          
        
        ,
         
        (
        i
        =
        1
        ,
        2
        ,
        …
        ,
        n
        )
        ,
      
    
    {\displaystyle \sum _{j=1}^{p}x_{ij}\beta _{j}=y_{i},\ (i=1,2,\dots ,n),}

of 
 
 
 
 n
 
 
 {\displaystyle n}
 
 <a href="/facts/Linear_equation/JfCgALwf">linear equations</a> in 
 
 
 
 p
 
 
 {\displaystyle p}
 
 unknown <a href="/facts/Coefficients/5Hwq2Wuq">coefficients</a>, 
 
 
 
 
 β
 
 1
 
 
 ,
 
 β
 
 2
 
 
 ,
 …
 ,
 
 β
 
 p
 
 
 
 
 {\displaystyle \beta _{1},\beta _{2},\dots ,\beta _{p}}
 
, with 
 
 
 
 n
 >
 p
 
 
 {\displaystyle n>p}
 
. This can be written in <a href="/facts/Matrix_(mathematics)/qa8FI4ko">matrix</a> form as

X
        
        
          β
        
        =
        
          y
        
        ,
      
    
    {\displaystyle \mathbf {X} {\boldsymbol {\beta }}=\mathbf {y} ,}

where

X
        
        =
        
          
            [
            
              
                
                  
                    X
                    
                      11
                    
                  
                
                
                  
                    X
                    
                      12
                    
                  
                
                
                  ⋯
                
                
                  
                    X
                    
                      1
                      p
                    
                  
                
              
              
                
                  
                    X
                    
                      21
                    
                  
                
                
                  
                    X
                    
                      22
                    
                  
                
                
                  ⋯
                
                
                  
                    X
                    
                      2
                      p
                    
                  
                
              
              
                
                  ⋮
                
                
                  ⋮
                
                
                  ⋱
                
                
                  ⋮
                
              
              
                
                  
                    X
                    
                      n
                      1
                    
                  
                
                
                  
                    X
                    
                      n
                      2
                    
                  
                
                
                  ⋯
                
                
                  
                    X
                    
                      n
                      p
                    
                  
                
              
            
            ]
          
        
        ,
        
        
          β
        
        =
        
          
            [
            
              
                
                  
                    β
                    
                      1
                    
                  
                
              
              
                
                  
                    β
                    
                      2
                    
                  
                
              
              
                
                  ⋮
                
              
              
                
                  
                    β
                    
                      p
                    
                  
                
              
            
            ]
          
        
        ,
        
        
          y
        
        =
        
          
            [
            
              
                
                  
                    y
                    
                      1
                    
                  
                
              
              
                
                  
                    y
                    
                      2
                    
                  
                
              
              
                
                  ⋮
                
              
              
                
                  
                    y
                    
                      n
                    
                  
                
              
            
            ]
          
        
        .
      
    
    {\displaystyle \mathbf {X} ={\begin{bmatrix}X_{11}&X_{12}&\cdots &X_{1p}\\X_{21}&X_{22}&\cdots &X_{2p}\\\vdots &\vdots &\ddots &\vdots \\X_{n1}&X_{n2}&\cdots &X_{np}\end{bmatrix}},\qquad {\boldsymbol {\beta }}={\begin{bmatrix}\beta _{1}\\\beta _{2}\\\vdots \\\beta _{p}\end{bmatrix}},\qquad \mathbf {y} ={\begin{bmatrix}y_{1}\\y_{2}\\\vdots \\y_{n}\end{bmatrix}}.}

(Note: for a linear model as above, not all elements in 
 
 
 
 
 X
 
 
 
 {\displaystyle \mathbf {X} }
 
 contains information on the data points. The first column is populated with ones, 
 
 
 
 
 X
 
 i
 1
 
 
 =
 1
 
 
 {\displaystyle X_{i1}=1}
 
. Only the other columns contain actual data. So here 
 
 
 
 p
 
 
 {\displaystyle p}
 
 is equal to the number of regressors plus one).
Such a system usually has no exact solution, so the goal is instead to find the coefficients 
 
 
 
 
 β
 
 
 
 {\displaystyle {\boldsymbol {\beta }}}
 
 which fit the equations "best", in the sense of solving the <a href="/facts/Quadratic_form_(statistics)/qH3sK2VG">quadratic</a> <a href="/facts/Mathematical_optimization/oRn8Iv5I">minimization</a> problem

β
              ^
            
          
        
        =
        
          
            
              a
              r
              g
              
              m
              i
              n
            
            β
          
        
        
        S
        (
        
          β
        
        )
        ,
      
    
    {\displaystyle {\hat {\boldsymbol {\beta }}}={\underset {\boldsymbol {\beta }}{\operatorname {arg\,min} }}\,S({\boldsymbol {\beta }}),}

where the objective function 
 
 
 
 S
 
 
 {\displaystyle S}
 
 is given by

S
        (
        
          β
        
        )
        =
        
          ∑
          
            i
            =
            1
          
          
            n
          
        
        
          
            |
            
              
                y
                
                  i
                
              
              −
              
                ∑
                
                  j
                  =
                  1
                
                
                  p
                
              
              
                X
                
                  i
                  j
                
              
              
                β
                
                  j
                
              
            
            |
          
          
            2
          
        
        =
        
          
            ‖
            
              
                y
              
              −
              
                X
              
              
                β
              
            
            ‖
          
          
            2
          
        
        .
      
    
    {\displaystyle S({\boldsymbol {\beta }})=\sum _{i=1}^{n}\left|y_{i}-\sum _{j=1}^{p}X_{ij}\beta _{j}\right|^{2}=\left\|\mathbf {y} -\mathbf {X} {\boldsymbol {\beta }}\right\|^{2}.}

A justification for choosing this criterion is given in Properties below. This minimization problem has a unique solution, provided that the 
 
 
 
 p
 
 
 {\displaystyle p}
 
 columns of the matrix 
 
 
 
 
 X
 
 
 
 {\displaystyle \mathbf {X} }
 
 are <a href="/facts/Linearly_independent/8JrHfIIa">linearly independent</a>, given by solving the so-called normal equations:

(
          
            
              
                X
              
              
                T
              
            
            
              X
            
          
          )
        
        
          
            
              β
              ^
            
          
        
        =
        
          
            X
          
          
            T
          
        
        
          y
        
         
        .
      
    
    {\displaystyle \left(\mathbf {X} ^{\operatorname {T} }\mathbf {X} \right){\hat {\boldsymbol {\beta }}}=\mathbf {X} ^{\operatorname {T} }\mathbf {y} \ .}

The matrix 
 
 
 
 
 
 X
 
 
 T
 
 
 
 X
 
 
 
 {\displaystyle \mathbf {X} ^{\operatorname {T} }\mathbf {X} }
 
 is known as the normal matrix or <a href="/facts/Gram_matrix/SGb2VGTU">Gram matrix</a> and the matrix 
 
 
 
 
 
 X
 
 
 T
 
 
 
 y
 
 
 
 {\displaystyle \mathbf {X} ^{\operatorname {T} }\mathbf {y} }
 
 is known as the <a href="/facts/Moment_matrix/ec467Q7g">moment matrix</a> of regressand by regressors.<a class="footnote-ref" id="fnref:3" href="#fn:3">3</a> Finally, 
 
 
 
 
 
 
 β
 ^
 
 
 
 
 
 {\displaystyle {\hat {\boldsymbol {\beta }}}}
 
 is the coefficient vector of the least-squares <a href="/facts/Hyperplane/2s2ycYYd">hyperplane</a>, expressed as

β
              ^
            
          
        
        =
        
          
            (
            
              
                
                  X
                
                
                  T
                
              
              
                X
              
            
            )
          
          
            −
            1
          
        
        
          
            X
          
          
            T
          
        
        
          y
        
        .
      
    
    {\displaystyle {\hat {\boldsymbol {\beta }}}=\left(\mathbf {X} ^{\operatorname {T} }\mathbf {X} \right)^{-1}\mathbf {X} ^{\operatorname {T} }\mathbf {y} .}

or

β
              ^
            
          
        
        =
        
          β
        
        +
        
          
            (
            
              
                
                  X
                
                
                  T
                
              
              
                X
              
            
            )
          
          
            −
            1
          
        
        
          
            X
          
          
            T
          
        
        
          ε
        
        .
      
    
    {\displaystyle {\hat {\boldsymbol {\beta }}}={\boldsymbol {\beta }}+\left(\mathbf {X} ^{\operatorname {T} }\mathbf {X} \right)^{-1}\mathbf {X} ^{\operatorname {T} }{\boldsymbol {\varepsilon }}.}

<h2 id="estimation">Estimation</h2>
Suppose b is a "candidate" value for the parameter vector β. The quantity yi − xiTb, called the <a href="/facts/Errors_and_residuals_in_statistics/Ef0LgAS4">residual</a> for the i-th observation, measures the vertical distance between the data point (xi, yi) and the hyperplane y = xTb, and thus assesses the degree of fit between the actual data and the model. The <a href="/facts/Sum_of_squared_residuals/vbGsrB3x">sum of squared residuals</a> (SSR) (also called the error sum of squares (ESS) or residual sum of squares (RSS))<a class="footnote-ref" id="fnref:4" href="#fn:4">4</a> is a measure of the overall model fit:

S
        (
        b
        )
        =
        
          ∑
          
            i
            =
            1
          
          
            n
          
        
        (
        
          y
          
            i
          
        
        −
        
          x
          
            i
          
          
            T
          
        
        b
        
          )
          
            2
          
        
        =
        (
        y
        −
        X
        b
        
          )
          
            T
          
        
        (
        y
        −
        X
        b
        )
        ,
      
    
    {\displaystyle S(b)=\sum _{i=1}^{n}(y_{i}-x_{i}^{\operatorname {T} }b)^{2}=(y-Xb)^{\operatorname {T} }(y-Xb),}

where T denotes the matrix <a href="/facts/Transpose/8wmsagGS">transpose</a>, and the rows of X, denoting the values of all the independent variables associated with a particular value of the dependent variable, are Xi = xiT. The value of b which minimizes this sum is called the OLS estimator for β. The function S(b) is quadratic in b with positive-definite <a href="/facts/Hessian_matrix/RguMNr3m">Hessian</a>, and therefore this function possesses a unique global minimum at 
 
 
 
 b
 =
 
 
 
 β
 ^
 
 
 
 
 
 {\displaystyle b={\hat {\beta }}}
 
, which can be given by the explicit formula<a class="footnote-ref" id="fnref:5" href="#fn:5">5</a><a href="/facts/Proofs_involving_ordinary_least_squares/3kMzc73I">[proof]</a>

β
              ^
            
          
        
        =
        
          argmin
          
            b
            ∈
            
              
                R
              
              
                p
              
            
          
        
        ⁡
        S
        (
        b
        )
        =
        (
        
          X
          
            T
          
        
        X
        
          )
          
            −
            1
          
        
        
          X
          
            T
          
        
        y
         
        .
      
    
    {\displaystyle {\hat {\beta }}=\operatorname {argmin} _{b\in \mathbb {R} ^{p}}S(b)=(X^{\operatorname {T} }X)^{-1}X^{\operatorname {T} }y\ .}

The product N = XT X is a <a href="/facts/Gram_matrix/SGb2VGTU">Gram matrix</a>, and its inverse, Q = N−1, is the cofactor matrix of β,<a class="footnote-ref" id="fnref:6" href="#fn:6">6</a><a class="footnote-ref" id="fnref:7" href="#fn:7">7</a><a class="footnote-ref" id="fnref:8" href="#fn:8">8</a> closely related to its covariance matrix, Cβ.
The matrix (XT X)−1 XT = Q XT is called the <a href="/facts/Moore%25E2%2580%2593Penrose_pseudoinverse/CU73tntq">Moore–Penrose pseudoinverse</a> matrix of X. This formulation highlights the point that estimation can be carried out if, and only if, there is no perfect <a href="/facts/Multicollinearity/bQXzB6sA">multicollinearity</a> between the explanatory variables (which would cause the Gram matrix to have no inverse).

<h2 id="prediction">Prediction</h2>
After we have estimated β, the <a href="/facts/Fitted_value/CGhyuxgz">fitted values</a> (or predicted values) from the regression will be

y
              ^
            
          
        
        =
        X
        
          
            
              β
              ^
            
          
        
        =
        P
        y
        ,
      
    
    {\displaystyle {\hat {y}}=X{\hat {\beta }}=Py,}

where P = X(XTX)−1XT is the <a href="/facts/Projection_matrix/5bLbzjy9">projection matrix</a> onto the space V spanned by the columns of X. This matrix P is also sometimes called the <a href="/facts/Hat_matrix/5bLbzjy9">hat matrix</a> because it "puts a hat" onto the variable y. Another matrix, closely related to P is the annihilator matrix M = In − P; this is a projection matrix onto the space orthogonal to V. Both matrices P and M are <a href="/facts/Symmetric_matrix/08CGwjNl">symmetric</a> and <a href="/facts/Idempotent_matrix/88Es7cEn">idempotent</a> (meaning that P2 = P and M2 = M), and relate to the data matrix X via identities PX = X and MX = 0.<a class="footnote-ref" id="fnref:9" href="#fn:9">9</a> Matrix M creates the residuals from the regression:

ε
              ^
            
          
        
        =
        y
        −
        
          
            
              y
              ^
            
          
        
        =
        y
        −
        X
        
          
            
              β
              ^
            
          
        
        =
        M
        y
        =
        M
        (
        X
        β
        +
        ε
        )
        =
        (
        M
        X
        )
        β
        +
        M
        ε
        =
        M
        ε
        .
      
    
    {\displaystyle {\hat {\varepsilon }}=y-{\hat {y}}=y-X{\hat {\beta }}=My=M(X\beta +\varepsilon )=(MX)\beta +M\varepsilon =M\varepsilon .}

The variances of the predicted values 
 
 
 
 
 s
 
 
 
 
 
 y
 ^
 
 
 
 
 i
 
 
 
 
 2
 
 
 
 
 {\displaystyle s_{{\hat {y}}_{i}}^{2}}
 
 are found in the main diagonal of the <a href="/facts/Variance-covariance_matrix/kLhgjHC6">variance-covariance matrix</a> of predicted values:

C
          
            
              
                y
                ^
              
            
          
        
        =
        
          s
          
            2
          
        
        P
        ,
      
    
    {\displaystyle C_{\hat {y}}=s^{2}P,}

where P is the projection matrix and s2 is the sample variance.<a class="footnote-ref" id="fnref:10" href="#fn:10">10</a>
The full matrix is very large; its diagonal elements can be calculated individually as:

s
          
            
              
                
                  
                    y
                    ^
                  
                
              
              
                i
              
            
          
          
            2
          
        
        =
        
          s
          
            2
          
        
        
          X
          
            i
          
        
        (
        
          X
          
            T
          
        
        X
        
          )
          
            −
            1
          
        
        
          X
          
            i
          
          
            T
          
        
        ,
      
    
    {\displaystyle s_{{\hat {y}}_{i}}^{2}=s^{2}X_{i}(X^{T}X)^{-1}X_{i}^{T},}

where Xi is the i-th row of matrix X.

<h2 id="sample-statistics">Sample statistics</h2>
Using these residuals we can estimate the sample variance s2 using the <a href="/facts/Reduced_chi-squared/EEkm83X0">reduced chi-squared</a> statistic:

s
          
            2
          
        
        =
        
          
            
              
                
                  
                    
                      ε
                      ^
                    
                  
                
                
                  
                    T
                  
                
              
              
                
                  
                    ε
                    ^
                  
                
              
            
            
              n
              −
              p
            
          
        
        =
        
          
            
              (
              M
              y
              
                )
                
                  
                    T
                  
                
              
              M
              y
            
            
              n
              −
              p
            
          
        
        =
        
          
            
              
                y
                
                  
                    T
                  
                
              
              
                M
                
                  
                    T
                  
                
              
              M
              y
            
            
              n
              −
              p
            
          
        
        =
        
          
            
              
                y
                
                  
                    T
                  
                
              
              M
              y
            
            
              n
              −
              p
            
          
        
        =
        
          
            
              S
              (
              
                
                  
                    β
                    ^
                  
                
              
              )
            
            
              n
              −
              p
            
          
        
        ,
        
        
          
            
              
                σ
                ^
              
            
          
          
            2
          
        
        =
        
          
            
              n
              −
              p
            
            n
          
        
        
        
          s
          
            2
          
        
      
    
    {\displaystyle s^{2}={\frac {{\hat {\varepsilon }}^{\mathrm {T} }{\hat {\varepsilon }}}{n-p}}={\frac {(My)^{\mathrm {T} }My}{n-p}}={\frac {y^{\mathrm {T} }M^{\mathrm {T} }My}{n-p}}={\frac {y^{\mathrm {T} }My}{n-p}}={\frac {S({\hat {\beta }})}{n-p}},\qquad {\hat {\sigma }}^{2}={\frac {n-p}{n}}\;s^{2}}

The denominator, n−p, is the <a href="/facts/Degrees_of_freedom_(statistics)/fvwKCU86">statistical degrees of freedom</a>. The first quantity, s2, is the OLS estimate for σ2, whereas the second, 
 
 
 
 
 
 
 
 
 σ
 ^
 
 
 
 
 2
 
 
 
 
 
 {\displaystyle \scriptstyle {\hat {\sigma }}^{2}}
 
, is the MLE estimate for σ2. The two estimators are quite similar in large samples; the first estimator is always <a href="/facts/Estimator_bias/oxIvEgmd">unbiased</a>, while the second estimator is biased but has a smaller <a href="/facts/Mean_squared_error/kz3TR7bv">mean squared error</a>. In practice s2 is used more often, since it is more convenient for the hypothesis testing. The square root of s2 is called the <a href="/facts/Regression_standard_error/EEkm83X0">regression standard error</a>,<a class="footnote-ref" id="fnref:11" href="#fn:11">11</a> standard error of the regression,<a class="footnote-ref" id="fnref:12" href="#fn:12">12</a><a class="footnote-ref" id="fnref:13" href="#fn:13">13</a> or standard error of the equation.<a class="footnote-ref" id="fnref:14" href="#fn:14">14</a>
It is common to assess the goodness-of-fit of the OLS regression by comparing how much the initial variation in the sample can be reduced by regressing onto X. The <a href="/facts/Coefficient_of_determination/acBlgnrv">coefficient of determination</a> R2 is defined as a ratio of "explained" variance to the "total" variance of the dependent variable y, in the cases where the regression sum of squares equals the sum of squares of residuals:<a class="footnote-ref" id="fnref:15" href="#fn:15">15</a>

R
          
            2
          
        
        =
        
          
            
              ∑
              (
              
                
                  
                    
                      y
                      ^
                    
                  
                
                
                  i
                
              
              −
              
                
                  y
                  ¯
                
              
              
                )
                
                  2
                
              
            
            
              ∑
              (
              
                y
                
                  i
                
              
              −
              
                
                  y
                  ¯
                
              
              
                )
                
                  2
                
              
            
          
        
        =
        
          
            
              
                y
                
                  
                    T
                  
                
              
              
                P
                
                  
                    T
                  
                
              
              L
              P
              y
            
            
              
                y
                
                  
                    T
                  
                
              
              L
              y
            
          
        
        =
        1
        −
        
          
            
              
                y
                
                  
                    T
                  
                
              
              M
              y
            
            
              
                y
                
                  
                    T
                  
                
              
              L
              y
            
          
        
        =
        1
        −
        
          
            
              R
              S
              S
            
            
              T
              S
              S
            
          
        
      
    
    {\displaystyle R^{2}={\frac {\sum ({\hat {y}}_{i}-{\overline {y}})^{2}}{\sum (y_{i}-{\overline {y}})^{2}}}={\frac {y^{\mathrm {T} }P^{\mathrm {T} }LPy}{y^{\mathrm {T} }Ly}}=1-{\frac {y^{\mathrm {T} }My}{y^{\mathrm {T} }Ly}}=1-{\frac {\rm {RSS}}{\rm {TSS}}}}

where TSS is the <a href="/facts/Total_sum_of_squares/brFhGeVT">total sum of squares</a> for the dependent variable, 
 
 
 
 L
 =
 
 I
 
 n
 
 
 −
 
 
 1
 n
 
 
 
 J
 
 n
 
 
 
 
 {\textstyle L=I_{n}-{\frac {1}{n}}J_{n}}
 
, and 
 
 
 
 
 J
 
 n
 
 
 
 
 {\textstyle J_{n}}
 
 is an n×n matrix of ones. (
 
 
 
 L
 
 
 {\displaystyle L}
 
 is a <a href="/facts/Centering_matrix/sRlKytJB">centering matrix</a> which is equivalent to regression on a constant; it simply subtracts the mean from a variable.) In order for R2 to be meaningful, the matrix X of data on regressors must contain a column vector of ones to represent the constant whose coefficient is the regression intercept. In that case, R2 will always be a number between 0 and 1, with values close to 1 indicating a good degree of fit.

<h3>Simple linear regression model</h3>
Main article: <a href="/facts/Simple_linear_regression/SNINCEFU">Simple linear regression</a>
If the data matrix X contains only two variables, a constant and a scalar regressor xi, then this is called the "simple regression model". This case is often considered in the beginner statistics classes, as it provides much simpler formulas even suitable for manual calculation. The parameters are commonly denoted as (α, β):

y
          
            i
          
        
        =
        α
        +
        β
        
          x
          
            i
          
        
        +
        
          ε
          
            i
          
        
        .
      
    
    {\displaystyle y_{i}=\alpha +\beta x_{i}+\varepsilon _{i}.}

The least squares estimates in this case are given by simple formulas

β
                      ^
                    
                  
                
              
              
                
                =
                
                  
                    
                      
                        ∑
                        
                          i
                          =
                          1
                        
                        
                          n
                        
                      
                      
                        (
                        
                          x
                          
                            i
                          
                        
                        −
                        
                          
                            
                              x
                              ¯
                            
                          
                        
                        )
                        (
                        
                          y
                          
                            i
                          
                        
                        −
                        
                          
                            
                              y
                              ¯
                            
                          
                        
                        )
                      
                    
                    
                      
                        ∑
                        
                          i
                          =
                          1
                        
                        
                          n
                        
                      
                      
                        (
                        
                          x
                          
                            i
                          
                        
                        −
                        
                          
                            
                              x
                              ¯
                            
                          
                        
                        
                          )
                          
                            2
                          
                        
                      
                    
                  
                
              
            
            
              
                
                  
                    
                      α
                      ^
                    
                  
                
              
              
                
                =
                
                  
                    
                      y
                      ¯
                    
                  
                
                −
                
                  
                    
                      β
                      ^
                    
                  
                
                
                
                  
                    
                      x
                      ¯
                    
                  
                
                 
                ,
              
            
          
        
      
    
    {\displaystyle {\begin{aligned}{\widehat {\beta }}&={\frac {\sum _{i=1}^{n}{(x_{i}-{\bar {x}})(y_{i}-{\bar {y}})}}{\sum _{i=1}^{n}{(x_{i}-{\bar {x}})^{2}}}}\\[2pt]{\widehat {\alpha }}&={\bar {y}}-{\widehat {\beta }}\,{\bar {x}}\ ,\end{aligned}}}

<h2 id="alternative-derivations">Alternative derivations</h2>
In the previous section the least squares estimator 
 
 
 
 
 
 
 β
 ^
 
 
 
 
 
 {\displaystyle {\hat {\beta }}}
 
 was obtained as a value that minimizes the sum of squared residuals of the model. However it is also possible to derive the same estimator from other approaches. In all cases the formula for OLS estimator remains the same: ^β = (XTX)−1XTy; the only difference is in how we interpret this result.

<h3>Projection</h3>

For mathematicians, OLS is an approximate solution to an overdetermined system of linear equations Xβ ≈ y, where β is the unknown. Assuming the system cannot be solved exactly (the number of equations n is much larger than the number of unknowns p), we are looking for a solution that could provide the smallest discrepancy between the right- and left- hand sides. In other words, we are looking for the solution that satisfies

β
              ^
            
          
        
        =
        
          
            a
            r
            g
          
        
        
          min
          
            β
          
        
        
        ‖
        
          y
        
        −
        
          X
        
        
          β
        
        ‖
        ,
      
    
    {\displaystyle {\hat {\beta }}={\rm {arg}}\min _{\beta }\,\lVert \mathbf {y} -\mathbf {X} {\boldsymbol {\beta }}\rVert ,}

where ‖·‖ is the standard <a href="/facts/Norm_(mathematics)/xIbR4uE1">L2 norm</a> in the n-dimensional <a href="/facts/Euclidean_space/R2UbzmzM">Euclidean space</a> Rn. The predicted quantity Xβ is just a certain linear combination of the vectors of regressors. Thus, the residual vector y − Xβ will have the smallest length when y is <a href="/facts/Projection_(linear_algebra)/sElXGkxD">projected orthogonally</a> onto the <a href="/facts/Linear_subspace/2wtKTgHL">linear subspace</a> <a href="/facts/Linear_span/vPjclEtb">spanned</a> by the columns of X. The OLS estimator 
 
 
 
 
 
 
 β
 ^
 
 
 
 
 
 {\displaystyle {\hat {\beta }}}
 
 in this case can be interpreted as the coefficients of <a href="/facts/Vector_decomposition/89IPoN6c">vector decomposition</a> of ^y = Py along the basis of X.
In other words, the gradient equations at the minimum can be written as:

(
        
          y
        
        −
        
          X
        
        
          
            
              β
              ^
            
          
        
        
          )
          
            ⊤
          
        
        
          X
        
        =
        0.
      
    
    {\displaystyle (\mathbf {y} -\mathbf {X} {\hat {\boldsymbol {\beta }}})^{\top }\mathbf {X} =0.}

A geometrical interpretation of these equations is that the vector of residuals, 
 
 
 
 
 y
 
 −
 X
 
 
 
 β
 ^
 
 
 
 
 
 {\displaystyle \mathbf {y} -X{\hat {\boldsymbol {\beta }}}}
 
 is orthogonal to the <a href="/facts/Column_space/INXbQVNI">column space</a> of X, since the <a href="/facts/Dot_product/tNz8MLjT">dot product</a> 
 
 
 
 (
 
 y
 
 −
 
 X
 
 
 
 
 β
 ^
 
 
 
 )
 ⋅
 
 X
 
 
 v
 
 
 
 {\displaystyle (\mathbf {y} -\mathbf {X} {\hat {\boldsymbol {\beta }}})\cdot \mathbf {X} \mathbf {v} }
 
 is equal to zero for any conformal vector, v. This means that 
 
 
 
 
 y
 
 −
 
 X
 
 
 
 
 β
 ^
 
 
 
 
 
 {\displaystyle \mathbf {y} -\mathbf {X} {\boldsymbol {\hat {\beta }}}}
 
 is the shortest of all possible vectors 
 
 
 
 
 y
 
 −
 
 X
 
 
 β
 
 
 
 {\displaystyle \mathbf {y} -\mathbf {X} {\boldsymbol {\beta }}}
 
, that is, the variance of the residuals is the minimum possible. This is illustrated at the right.
Introducing 
 
 
 
 
 
 
 γ
 ^
 
 
 
 
 
 {\displaystyle {\hat {\boldsymbol {\gamma }}}}
 
 and a matrix K with the assumption that a matrix 
 
 
 
 [
 
 X
 
  
 
 K
 
 ]
 
 
 {\displaystyle [\mathbf {X} \ \mathbf {K} ]}
 
 is non-singular and KT X = 0 (cf. <a href="/facts/Linear_projection/sElXGkxD">Orthogonal projections</a>), the residual vector should satisfy the following equation:

r
              
              ^
            
          
        
        :=
        
          y
        
        −
        
          X
        
        
          
            
              β
              ^
            
          
        
        =
        
          K
        
        
          
            
              γ
              ^
            
          
        
        .
      
    
    {\displaystyle {\hat {\mathbf {r} }}:=\mathbf {y} -\mathbf {X} {\hat {\boldsymbol {\beta }}}=\mathbf {K} {\hat {\boldsymbol {\gamma }}}.}

The equation and solution of linear least squares are thus described as follows:

y
                
              
              
                
                =
                
                  
                    [
                    
                      
                        
                          
                            X
                          
                        
                        
                          
                            K
                          
                        
                      
                    
                    ]
                  
                
                
                  
                    [
                    
                      
                        
                          
                            
                              
                                β
                                ^
                              
                            
                          
                        
                      
                      
                        
                          
                            
                              
                                γ
                                ^
                              
                            
                          
                        
                      
                    
                    ]
                  
                
                ,

⇒
                
                  
                    [
                    
                      
                        
                          
                            
                              
                                β
                                ^
                              
                            
                          
                        
                      
                      
                        
                          
                            
                              
                                γ
                                ^
                              
                            
                          
                        
                      
                    
                    ]
                  
                
              
              
                
                =
                
                  
                    
                      [
                      
                        
                          
                            
                              X
                            
                          
                          
                            
                              K
                            
                          
                        
                      
                      ]
                    
                  
                  
                    −
                    1
                  
                
                
                  y
                
                =
                
                  
                    [
                    
                      
                        
                          
                            
                              (
                              
                                
                                  
                                    X
                                  
                                  
                                    ⊤
                                  
                                
                                
                                  X
                                
                              
                              )
                            
                            
                              −
                              1
                            
                          
                          
                            
                              X
                            
                            
                              ⊤
                            
                          
                        
                      
                      
                        
                          
                            
                              (
                              
                                
                                  
                                    K
                                  
                                  
                                    ⊤
                                  
                                
                                
                                  K
                                
                              
                              )
                            
                            
                              −
                              1
                            
                          
                          
                            
                              K
                            
                            
                              ⊤
                            
                          
                        
                      
                    
                    ]
                  
                
                
                  y
                
                .
              
            
          
        
      
    
    {\displaystyle {\begin{aligned}\mathbf {y} &={\begin{bmatrix}\mathbf {X} &\mathbf {K} \end{bmatrix}}{\begin{bmatrix}{\hat {\boldsymbol {\beta }}}\\{\hat {\boldsymbol {\gamma }}}\end{bmatrix}},\\{}\Rightarrow {\begin{bmatrix}{\hat {\boldsymbol {\beta }}}\\{\hat {\boldsymbol {\gamma }}}\end{bmatrix}}&={\begin{bmatrix}\mathbf {X} &\mathbf {K} \end{bmatrix}}^{-1}\mathbf {y} ={\begin{bmatrix}\left(\mathbf {X} ^{\top }\mathbf {X} \right)^{-1}\mathbf {X} ^{\top }\\\left(\mathbf {K} ^{\top }\mathbf {K} \right)^{-1}\mathbf {K} ^{\top }\end{bmatrix}}\mathbf {y} .\end{aligned}}}

Another way of looking at it is to consider the regression line to be a weighted average of the lines passing through the combination of any two points in the dataset.<a class="footnote-ref" id="fnref:16" href="#fn:16">16</a> Although this way of calculation is more computationally expensive, it provides a better intuition on OLS.

<h3>Maximum likelihood</h3>
The OLS estimator is identical to the <a href="/facts/Maximum_likelihood_estimator/0Yq2dpQD">maximum likelihood estimator</a> (MLE) under the normality assumption for the error terms.<a class="footnote-ref" id="fnref:17" href="#fn:17">17</a><a href="/facts/Proofs_involving_ordinary_least_squares/3kMzc73I">[proof]</a> This normality assumption has historical importance, as it provided the basis for the early work in linear regression analysis by <a href="/facts/Udny_Yule/3MHsP37s">Yule</a> and <a href="/facts/Karl_Pearson/fXB7GT5Q">Pearson</a>. From the properties of MLE, we can infer that the OLS estimator is asymptotically efficient (in the sense of attaining the <a href="/facts/Cram%25C3%25A9r%25E2%2580%2593Rao_bound/Kvhqdpol">Cramér–Rao bound</a> for variance) if the normality assumption is satisfied.<a class="footnote-ref" id="fnref:18" href="#fn:18">18</a>

<h3>Generalized method of moments</h3>
In <a href="/facts/Iid/othIRaWt">iid</a> case the OLS estimator can also be viewed as a <a href="/facts/Generalized_method_of_moments/mNc3410V">GMM</a> estimator arising from the moment conditions

E
        
        
          
            [
          
        
        
        
          x
          
            i
          
        
        
          (
          
            
              y
              
                i
              
            
            −
            
              x
              
                i
              
              
                T
              
            
            β
          
          )
        
        
        
          
            ]
          
        
        =
        0.
      
    
    {\displaystyle \mathrm {E} {\big [}\,x_{i}\left(y_{i}-x_{i}^{\operatorname {T} }\beta \right)\,{\big ]}=0.}

These moment conditions state that the regressors should be uncorrelated with the errors. Since xi is a p-vector, the number of moment conditions is equal to the dimension of the parameter vector β, and thus the system is exactly identified. This is the so-called classical GMM case, when the estimator does not depend on the choice of the weighting matrix.
Note that the original strict exogeneity assumption E[εi | xi] = 0 implies a far richer set of moment conditions than stated above. In particular, this assumption implies that for any vector-function ƒ, the moment condition E[ƒ(xi)·εi] = 0 will hold. However it can be shown using the <a href="/facts/Gauss%25E2%2580%2593Markov_theorem/j85SGhrw">Gauss–Markov theorem</a> that the optimal choice of function ƒ is to take ƒ(x) = x, which results in the moment equation posted above.

<h2 id="properties">Properties</h2>
<h3>Assumptions</h3>
See also: <a href="/facts/Linear_regression/5n998IhK">Linear regression § Assumptions</a>
There are several different frameworks in which the <a href="/facts/Linear_regression_model/5n998IhK">linear regression model</a> can be cast in order to make the OLS technique applicable. Each of these settings produces the same formulas and same results. The only difference is the interpretation and the assumptions which have to be imposed in order for the method to give meaningful results. The choice of the applicable framework depends mostly on the nature of data in hand, and on the inference task which has to be performed.
One of the lines of difference in interpretation is whether to treat the regressors as random variables, or as predefined constants. In the first case (random design) the regressors xi are random and sampled together with the yi's from some <a href="/facts/Statistical_population/KAILs8E3">population</a>, as in an <a href="/facts/Observational_study/zEhxI4eh">observational study</a>. This approach allows for more natural study of the <a href="/facts/Asymptotic_theory_(statistics)/tw5Mj44v">asymptotic properties</a> of the estimators. In the other interpretation (fixed design), the regressors X are treated as known constants set by a <a href="/facts/Design_of_experiments/VMrNftbr">design</a>, and y is sampled conditionally on the values of X as in an <a href="/facts/Experiment/esltM54f">experiment</a>. For practical purposes, this distinction is often unimportant, since estimation and inference is carried out while conditioning on X. All results stated in this article are within the random design framework.

<h4>Classical linear regression model</h4>
The classical model focuses on the "finite sample" estimation and inference, meaning that the number of observations n is fixed. This contrasts with the other approaches, which study the <a href="/facts/Asymptotic_theory_(statistics)/tw5Mj44v">asymptotic behavior</a> of OLS, and in which the behavior at a large number of samples is studied.

<ul><li>Correct specification. The linear functional form must coincide with the form of the actual data-generating process.</li>
<li>Strict exogeneity. The errors in the regression should have <a href="/facts/Conditional_expectation/3GxG1Q3x">conditional mean</a> zero:<a class="footnote-ref" id="fnref:19" href="#fn:19">19</a>
 
 
 
 E
 ⁡
 [
 
 ε
 ∣
 X
 
 ]
 =
 0.
 
 
 {\displaystyle \operatorname {E} [\,\varepsilon \mid X\,]=0.}
 
 The immediate consequence of the exogeneity assumption is that the errors have mean zero: E[ε] = 0 (for the <a href="/facts/Law_of_total_expectation/zSqn8uVj">law of total expectation</a>), and that the regressors are uncorrelated with the errors: E[XTε] = 0. The exogeneity assumption is critical for the OLS theory. If it holds then the regressor variables are called exogenous. If it does not, then those regressors that are correlated with the error term are called <a href="/facts/Endogeneity_(econometrics)/NvtsxNYE">endogenous</a>,<a class="footnote-ref" id="fnref:20" href="#fn:20">20</a> and the OLS estimator becomes biased. In such case the <a href="/facts/Instrumental_variable/cEQ1xwT6">method of instrumental variables</a> may be used to carry out inference.</li>
<li>No linear dependence. The regressors in X must all be <a href="/facts/Linearly_independent/8JrHfIIa">linearly independent</a>. Mathematically, this means that the matrix X must have full <a href="/facts/Column_rank/Yp9b5LK3">column rank</a> almost surely:<a class="footnote-ref" id="fnref:21" href="#fn:21">21</a>
 
 
 
 Pr
 
 
 
 [
 
 
 
 rank
 ⁡
 (
 X
 )
 =
 p
 
 
 
 ]
 
 
 =
 1.
 
 
 {\displaystyle \Pr \!{\big [}\,\operatorname {rank} (X)=p\,{\big ]}=1.}
 
 Usually, it is also assumed that the regressors have finite moments up to at least the second moment. Then the matrix Qxx = E[XTX / n] is finite and positive semi-definite. When this assumption is violated the regressors are called linearly dependent or <a href="/facts/Multicollinearity/bQXzB6sA">perfectly multicollinear</a>. In such case the value of the regression coefficient β cannot be learned, although prediction of y values is still possible for new values of the regressors that lie in the same linearly dependent subspace.</li>
<li>Spherical errors:<a class="footnote-ref" id="fnref:22" href="#fn:22">22</a>
 
 
 
 Var
 ⁡
 [
 
 ε
 ∣
 X
 
 ]
 =
 
 σ
 
 2
 
 
 
 I
 
 n
 
 
 ,
 
 
 {\displaystyle \operatorname {Var} [\,\varepsilon \mid X\,]=\sigma ^{2}I_{n},}
 
 where In is the <a href="/facts/Identity_matrix/Glbcf6ol">identity matrix</a> in dimension n, and σ2 is a parameter which determines the variance of each observation. This σ2 is considered a <a href="/facts/Nuisance_parameter/acJWVqxj">nuisance parameter</a> in the model, although usually it is also estimated. If this assumption is violated then the OLS estimates are still valid, but no longer efficient. It is customary to split this assumption into two parts:
<ul><li><a href="/facts/Homoscedasticity/50v2qyEN">Homoscedasticity</a>: E[ εi2 | X ] = σ2, which means that the error term has the same variance σ2 in each observation. When this requirement is violated this is called <a href="/facts/Heteroscedasticity/msBQWI7G">heteroscedasticity</a>, in such case a more efficient estimator would be <a href="/facts/Weighted_least_squares/BpCgNF5m">weighted least squares</a>. If the errors have infinite variance then the OLS estimates will also have infinite variance (although by the <a href="/facts/Law_of_large_numbers/X3Bjcy3v">law of large numbers</a> they will nonetheless tend toward the true values so long as the errors have zero mean). In this case, <a href="/facts/Robust_regression/gGvbTtVC">robust estimation</a> techniques are recommended.</li>
<li>No <a href="/facts/Autocorrelation/dm4MqK0A">autocorrelation</a>: the errors are <a href="/facts/Correlation/egkluAEm">uncorrelated</a> between observations: E[ εiεj | X ] = 0 for i ≠ j. This assumption may be violated in the context of <a href="/facts/Time_series/fSXPR817">time series</a> data, <a href="/facts/Panel_data/5cHTwLe9">panel data</a>, cluster samples, hierarchical data, repeated measures data, longitudinal data, and other data with dependencies. In such cases <a href="/facts/Generalized_least_squares/NI2GL3RK">generalized least squares</a> provides a better alternative than the OLS. Another expression for autocorrelation is serial correlation.</li></ul></li>
<li>Normality. It is sometimes additionally assumed that the errors have <a href="/facts/Multivariate_normal_distribution/2Xfegqz2">normal distribution</a> conditional on the regressors:<a class="footnote-ref" id="fnref:23" href="#fn:23">23</a>
 
 
 
 ε
 ∣
 X
 ∼
 
 
 N
 
 
 (
 0
 ,
 
 σ
 
 2
 
 
 
 I
 
 n
 
 
 )
 .
 
 
 {\displaystyle \varepsilon \mid X\sim {\mathcal {N}}(0,\sigma ^{2}I_{n}).}
 
This assumption is not needed for the validity of the OLS method, although certain additional finite-sample properties can be established in case when it does (especially in the area of hypotheses testing). Also when the errors are normal, the OLS estimator is equivalent to the <a href="/facts/Maximum_likelihood_estimator/0Yq2dpQD">maximum likelihood estimator</a> (MLE), and therefore it is asymptotically efficient in the class of all <a href="/facts/Regular_estimator/LUl7R3qo">regular estimators</a>. Importantly, the normality assumption applies only to the error terms; contrary to a popular misconception, the response (dependent) variable is not required to be normally distributed.<a class="footnote-ref" id="fnref:24" href="#fn:24">24</a></li></ul>
<h4>Independent and identically distributed (iid)</h4>
In some applications, especially with <a href="/facts/Cross-sectional_data/lTtwU45w">cross-sectional data</a>, an additional assumption is imposed — that all observations are <a href="/facts/Independent_and_identically_distributed_random_variables/othIRaWt">independent and identically distributed</a>. This means that all observations are taken from a <a href="/facts/Random_sample/URCwKNge">random sample</a> which makes all the assumptions listed earlier simpler and easier to interpret. Also this framework allows one to state asymptotic results (as the sample size n → ∞), which are understood as a theoretical possibility of fetching new independent observations from the <a href="/facts/Data_generating_process/lwnBmTmo">data generating process</a>. The list of assumptions in this case is:

<ul><li>IID observations: (xi, yi) is <a href="/facts/Independent_random_variables/NUzQtnUL">independent</a> from, and has the same <a href="/facts/Probability_distribution/EpsKKVRu">distribution</a> as, (xj, yj) for all i ≠ j;</li>
<li>No perfect multicollinearity: Qxx = E[ xi xiT ] is a <a href="/facts/Positive-definite_matrix/Hbr6AuPS">positive-definite matrix</a>;</li>
<li>Exogeneity: E[ εi | xi ] = 0;</li>
<li>Homoscedasticity: Var[ εi | xi ] = σ2.</li></ul>
<h4>Time series model</h4>
<ul><li>The <a href="/facts/Stochastic_process/Ng1n1dnB">stochastic process</a> {xi, yi} is <a href="/facts/Stationary_process/vozdgfEX">stationary</a> and <a href="/facts/Ergodic_process/7qSfLtjC">ergodic</a>; if {xi, yi} is nonstationary, OLS results are often spurious unless {xi, yi} is <a href="/facts/Cointegration/na815pCW">co-integrating</a>.<a class="footnote-ref" id="fnref:25" href="#fn:25">25</a></li>
<li>The regressors are predetermined: E[xiεi] = 0 for all i = 1, ..., n;</li>
<li>The p×p matrix Qxx = E[ xi xiT ] is of full rank, and hence <a href="/facts/Positive-definite_matrix/Hbr6AuPS">positive-definite</a>;</li>
<li>{xiεi} is a <a href="/facts/Martingale_difference_sequence/F473ymYq">martingale difference sequence</a>, with a finite matrix of second moments Qxxε² = E[ εi2xi xiT ].</li></ul>
<h3>Finite sample properties</h3>
First of all, under the strict exogeneity assumption the OLS estimators 
 
 
 
 
 
 
 
 β
 ^
 
 
 
 
 
 
 {\displaystyle \scriptstyle {\hat {\beta }}}
 
 and s2 are <a href="/facts/Bias_of_an_estimator/oxIvEgmd">unbiased</a>, meaning that their expected values coincide with the true values of the parameters:<a class="footnote-ref" id="fnref:26" href="#fn:26">26</a><a href="/facts/Proofs_involving_ordinary_least_squares/3kMzc73I">[proof]</a>

E
        ⁡
        [
        
        
          
            
              β
              ^
            
          
        
        ∣
        X
        
        ]
        =
        β
        ,
        
        E
        ⁡
        [
        
        
          s
          
            2
          
        
        ∣
        X
        
        ]
        =
        
          σ
          
            2
          
        
        .
      
    
    {\displaystyle \operatorname {E} [\,{\hat {\beta }}\mid X\,]=\beta ,\quad \operatorname {E} [\,s^{2}\mid X\,]=\sigma ^{2}.}

If the strict exogeneity does not hold (as is the case with many <a href="/facts/Time_series/fSXPR817">time series</a> models, where exogeneity is assumed only with respect to the past shocks but not the future ones), then these estimators will be biased in finite samples.
The <a href="/facts/Variance-covariance_matrix/kLhgjHC6">variance-covariance matrix</a> (or simply covariance matrix) of 
 
 
 
 
 
 
 
 β
 ^
 
 
 
 
 
 
 {\displaystyle \scriptstyle {\hat {\beta }}}
 
 is equal to<a class="footnote-ref" id="fnref:27" href="#fn:27">27</a>

Var
        ⁡
        [
        
        
          
            
              β
              ^
            
          
        
        ∣
        X
        
        ]
        =
        
          σ
          
            2
          
        
        
          
            (
            
              
                X
                
                  T
                
              
              X
            
            )
          
          
            −
            1
          
        
        =
        
          σ
          
            2
          
        
        Q
        .
      
    
    {\displaystyle \operatorname {Var} [\,{\hat {\beta }}\mid X\,]=\sigma ^{2}\left(X^{\operatorname {T} }X\right)^{-1}=\sigma ^{2}Q.}

In particular, the standard error of each coefficient 
 
 
 
 
 
 
 
 
 β
 ^
 
 
 
 
 j
 
 
 
 
 
 {\displaystyle \scriptstyle {\hat {\beta }}_{j}}
 
 is equal to square root of the j-th diagonal element of this matrix. The estimate of this standard error is obtained by replacing the unknown quantity σ2 with its estimate s2. Thus,

s
                .
                
                e
                .
              
              ^
            
          
        
        (
        
          
            
              
                β
                ^
              
            
          
          
            j
          
        
        )
        =
        
          
            
              s
              
                2
              
            
            
              
                (
                
                  
                    X
                    
                      T
                    
                  
                  X
                
                )
              
              
                j
                j
              
              
                −
                1
              
            
          
        
      
    
    {\displaystyle {\widehat {\operatorname {s.\!e.} }}({\hat {\beta }}_{j})={\sqrt {s^{2}\left(X^{\operatorname {T} }X\right)_{jj}^{-1}}}}

It can also be easily shown that the estimator 
 
 
 
 
 
 
 
 β
 ^
 
 
 
 
 
 
 {\displaystyle \scriptstyle {\hat {\beta }}}
 
 is uncorrelated with the residuals from the model:<a class="footnote-ref" id="fnref:28" href="#fn:28">28</a>

Cov
        ⁡
        [
        
        
          
            
              β
              ^
            
          
        
        ,
        
          
            
              ε
              ^
            
          
        
        ∣
        X
        
        ]
        =
        0.
      
    
    {\displaystyle \operatorname {Cov} [\,{\hat {\beta }},{\hat {\varepsilon }}\mid X\,]=0.}

The <a href="/facts/Gauss%25E2%2580%2593Markov_theorem/j85SGhrw">Gauss–Markov theorem</a> states that under the spherical errors assumption (that is, the errors should be <a href="/facts/Uncorrelated/Ub2XK1WJ">uncorrelated</a> and <a href="/facts/Homoscedastic/50v2qyEN">homoscedastic</a>) the estimator 
 
 
 
 
 
 
 
 β
 ^
 
 
 
 
 
 
 {\displaystyle \scriptstyle {\hat {\beta }}}
 
 is efficient in the class of linear unbiased estimators. This is called the best linear unbiased estimator (BLUE). Efficiency should be understood as if we were to find some other estimator 
 
 
 
 
 
 
 
 β
 ~
 
 
 
 
 
 
 {\displaystyle \scriptstyle {\tilde {\beta }}}
 
 which would be linear in y and unbiased, then <a class="footnote-ref" id="fnref:29" href="#fn:29">29</a>

Var
        ⁡
        [
        
        
          
            
              β
              ~
            
          
        
        ∣
        X
        
        ]
        −
        Var
        ⁡
        [
        
        
          
            
              β
              ^
            
          
        
        ∣
        X
        
        ]
        ≥
        0
      
    
    {\displaystyle \operatorname {Var} [\,{\tilde {\beta }}\mid X\,]-\operatorname {Var} [\,{\hat {\beta }}\mid X\,]\geq 0}

in the sense that this is a <a href="/facts/Nonnegative-definite_matrix/Hbr6AuPS">nonnegative-definite matrix</a>. This theorem establishes optimality only in the class of linear unbiased estimators, which is quite restrictive. Depending on the distribution of the error terms ε, other, non-linear estimators may provide better results than OLS.

<h4>Assuming normality</h4>
The properties listed so far are all valid regardless of the underlying distribution of the error terms. However, if you are willing to assume that the normality assumption holds (that is, that ε ~ N(0, σ2In)), then additional properties of the OLS estimators can be stated.
The estimator 
 
 
 
 
 
 
 
 β
 ^
 
 
 
 
 
 
 {\displaystyle \scriptstyle {\hat {\beta }}}
 
 is normally distributed, with mean and variance as given before:<a class="footnote-ref" id="fnref:30" href="#fn:30">30</a>

β
              ^
            
          
        
         
        ∼
         
        
          
            N
          
        
        
          
            (
          
        
        β
        ,
         
        
          σ
          
            2
          
        
        (
        
          X
          
            
              T
            
          
        
        X
        
          )
          
            −
            1
          
        
        
          
            )
          
        
        .
      
    
    {\displaystyle {\hat {\beta }}\ \sim \ {\mathcal {N}}{\big (}\beta ,\ \sigma ^{2}(X^{\mathrm {T} }X)^{-1}{\big )}.}

This estimator reaches the <a href="/facts/Cram%25C3%25A9r%25E2%2580%2593Rao_bound/Kvhqdpol">Cramér–Rao bound</a> for the model, and thus is optimal in the class of all unbiased estimators.<a class="footnote-ref" id="fnref:31" href="#fn:31">31</a> Note that unlike the <a href="/facts/Gauss%25E2%2580%2593Markov_theorem/j85SGhrw">Gauss–Markov theorem</a>, this result establishes optimality among both linear and non-linear estimators, but only in the case of normally distributed error terms.
The estimator s2 will be proportional to the <a href="/facts/Chi-squared_distribution/wYnWWgUe">chi-squared distribution</a>:<a class="footnote-ref" id="fnref:32" href="#fn:32">32</a>

s
          
            2
          
        
         
        ∼
         
        
          
            
              σ
              
                2
              
            
            
              n
              −
              p
            
          
        
        ⋅
        
          χ
          
            n
            −
            p
          
          
            2
          
        
      
    
    {\displaystyle s^{2}\ \sim \ {\frac {\sigma ^{2}}{n-p}}\cdot \chi _{n-p}^{2}}

The variance of this estimator is equal to 2σ4/(n − p), which does not attain the <a href="/facts/Cram%25C3%25A9r%25E2%2580%2593Rao_bound/Kvhqdpol">Cramér–Rao bound</a> of 2σ4/n. However it was shown that there are no unbiased estimators of σ2 with variance smaller than that of the estimator s2.<a class="footnote-ref" id="fnref:33" href="#fn:33">33</a> If we are willing to allow biased estimators, and consider the class of estimators that are proportional to the sum of squared residuals (SSR) of the model, then the best (in the sense of the <a href="/facts/Mean_squared_error/kz3TR7bv">mean squared error</a>) estimator in this class will be ~σ2 = SSR / (n − p + 2), which even beats the Cramér–Rao bound in case when there is only one regressor (p = 1).<a class="footnote-ref" id="fnref:34" href="#fn:34">34</a>
Moreover, the estimators 
 
 
 
 
 
 
 
 β
 ^
 
 
 
 
 
 
 {\displaystyle \scriptstyle {\hat {\beta }}}
 
 and s2 are <a href="/facts/Independent_random_variables/NUzQtnUL">independent</a>,<a class="footnote-ref" id="fnref:35" href="#fn:35">35</a> the fact which comes in useful when constructing the t- and F-tests for the regression.

<h4>Influential observations</h4>
Main article: <a href="/facts/Influential_observation/HLc4TDXz">Influential observation</a>
See also: <a href="/facts/Leverage_(statistics)/5lkfBFWo">Leverage (statistics)</a>
As was mentioned before, the estimator 
 
 
 
 
 
 
 β
 ^
 
 
 
 
 
 {\displaystyle {\hat {\beta }}}
 
 is linear in y, meaning that it represents a linear combination of the dependent variables yi. The weights in this linear combination are functions of the regressors X, and generally are unequal. The observations with high weights are called influential because they have a more pronounced effect on the value of the estimator.
To analyze which observations are influential we remove a specific j-th observation and consider how much the estimated quantities are going to change (similarly to the <a href="/facts/Jackknife_method/CmJ29chQ">jackknife method</a>). It can be shown that the change in the OLS estimator for β will be equal to <a class="footnote-ref" id="fnref:36" href="#fn:36">36</a>

β
                ^
              
            
          
          
            (
            j
            )
          
        
        −
        
          
            
              β
              ^
            
          
        
        =
        −
        
          
            1
            
              1
              −
              
                h
                
                  j
                
              
            
          
        
        (
        
          X
          
            
              T
            
          
        
        X
        
          )
          
            −
            1
          
        
        
          x
          
            j
          
          
            
              T
            
          
        
        
          
            
              
                ε
                ^
              
            
          
          
            j
          
        
        
        ,
      
    
    {\displaystyle {\hat {\beta }}^{(j)}-{\hat {\beta }}=-{\frac {1}{1-h_{j}}}(X^{\mathrm {T} }X)^{-1}x_{j}^{\mathrm {T} }{\hat {\varepsilon }}_{j}\,,}

where hj = xjT (XTX)−1xj is the j-th diagonal element of the hat matrix P, and xj is the vector of regressors corresponding to the j-th observation. Similarly, the change in the predicted value for j-th observation resulting from omitting that observation from the dataset will be equal to <a class="footnote-ref" id="fnref:37" href="#fn:37">37</a>

y
                ^
              
            
          
          
            j
          
          
            (
            j
            )
          
        
        −
        
          
            
              
                y
                ^
              
            
          
          
            j
          
        
        =
        
          x
          
            j
          
          
            
              T
            
          
        
        
          
            
              
                β
                ^
              
            
          
          
            (
            j
            )
          
        
        −
        
          x
          
            j
          
          
            T
          
        
        
          
            
              β
              ^
            
          
        
        =
        −
        
          
            
              h
              
                j
              
            
            
              1
              −
              
                h
                
                  j
                
              
            
          
        
        
        
          
            
              
                ε
                ^
              
            
          
          
            j
          
        
      
    
    {\displaystyle {\hat {y}}_{j}^{(j)}-{\hat {y}}_{j}=x_{j}^{\mathrm {T} }{\hat {\beta }}^{(j)}-x_{j}^{\operatorname {T} }{\hat {\beta }}=-{\frac {h_{j}}{1-h_{j}}}\,{\hat {\varepsilon }}_{j}}

From the properties of the hat matrix, 0 ≤ hj ≤ 1, and they sum up to p, so that on average hj ≈ p/n. These quantities hj are called the leverages, and observations with high hj are called leverage points.<a class="footnote-ref" id="fnref:38" href="#fn:38">38</a> Usually the observations with high leverage ought to be scrutinized more carefully, in case they are erroneous, or outliers, or in some other way atypical of the rest of the dataset.

<h4>Partitioned regression</h4>
Sometimes the variables and corresponding parameters in the regression can be logically split into two groups, so that the regression takes form

y
        =
        
          X
          
            1
          
        
        
          β
          
            1
          
        
        +
        
          X
          
            2
          
        
        
          β
          
            2
          
        
        +
        ε
        ,
      
    
    {\displaystyle y=X_{1}\beta _{1}+X_{2}\beta _{2}+\varepsilon ,}

where X1 and X2 have dimensions n×p1, n×p2, and β1, β2 are p1×1 and p2×1 vectors, with p1 + p2 = p.
The <a href="/facts/Frisch%25E2%2580%2593Waugh%25E2%2580%2593Lovell_theorem/I48IywQC">Frisch–Waugh–Lovell theorem</a> states that in this regression the residuals 
 
 
 
 
 
 
 ε
 ^
 
 
 
 
 
 {\displaystyle {\hat {\varepsilon }}}
 
 and the OLS estimate 
 
 
 
 
 
 
 
 
 β
 ^
 
 
 
 
 2
 
 
 
 
 
 {\displaystyle \scriptstyle {\hat {\beta }}_{2}}
 
 will be numerically identical to the residuals and the OLS estimate for β2 in the following regression:<a class="footnote-ref" id="fnref:39" href="#fn:39">39</a>

M
          
            1
          
        
        y
        =
        
          M
          
            1
          
        
        
          X
          
            2
          
        
        
          β
          
            2
          
        
        +
        η
        
        ,
      
    
    {\displaystyle M_{1}y=M_{1}X_{2}\beta _{2}+\eta \,,}

where M1 is the <a href="/facts/Annihilator_matrix/5bLbzjy9">annihilator matrix</a> for regressors X1.
The theorem can be used to establish a number of theoretical results. For example, having a regression with a constant and another regressor is equivalent to subtracting the means from the dependent variable and the regressor and then running the regression for the de-meaned variables but without the constant term.

<h4>Constrained estimation</h4>
Main article: <a href="/facts/Ridge_regression/Le6KJgJ1">Ridge regression</a>
Suppose it is known that the coefficients in the regression satisfy a system of linear equations

A
        :
        
        
          Q
          
            T
          
        
        β
        =
        c
        ,
        
      
    
    {\displaystyle A\colon \quad Q^{\operatorname {T} }\beta =c,\,}

where Q is a p×q matrix of full rank, and c is a q×1 vector of known constants, where q < p. In this case least squares estimation is equivalent to minimizing the sum of squared residuals of the model subject to the constraint A. The constrained least squares (CLS) estimator can be given by an explicit formula:<a class="footnote-ref" id="fnref:40" href="#fn:40">40</a>

β
                ^
              
            
          
          
            c
          
        
        =
        
          
            
              β
              ^
            
          
        
        −
        (
        
          X
          
            T
          
        
        X
        
          )
          
            −
            1
          
        
        Q
        
          
            (
          
        
        
          Q
          
            T
          
        
        (
        
          X
          
            T
          
        
        X
        
          )
          
            −
            1
          
        
        Q
        
          
            
              )
            
          
          
            −
            1
          
        
        (
        
          Q
          
            T
          
        
        
          
            
              β
              ^
            
          
        
        −
        c
        )
        .
      
    
    {\displaystyle {\hat {\beta }}^{c}={\hat {\beta }}-(X^{\operatorname {T} }X)^{-1}Q{\Big (}Q^{\operatorname {T} }(X^{\operatorname {T} }X)^{-1}Q{\Big )}^{-1}(Q^{\operatorname {T} }{\hat {\beta }}-c).}

This expression for the constrained estimator is valid as long as the matrix XTX is invertible. It was assumed from the beginning of this article that this matrix is of full rank, and it was noted that when the rank condition fails, β will not be identifiable. However it may happen that adding the restriction A makes β identifiable, in which case one would like to find the formula for the estimator. The estimator is equal to <a class="footnote-ref" id="fnref:41" href="#fn:41">41</a>

β
                ^
              
            
          
          
            c
          
        
        =
        R
        (
        
          R
          
            T
          
        
        
          X
          
            T
          
        
        X
        R
        
          )
          
            −
            1
          
        
        
          R
          
            T
          
        
        
          X
          
            T
          
        
        y
        +
        
          
            (
          
        
        
          I
          
            p
          
        
        −
        R
        (
        
          R
          
            T
          
        
        
          X
          
            T
          
        
        X
        R
        
          )
          
            −
            1
          
        
        
          R
          
            T
          
        
        
          X
          
            T
          
        
        X
        
          
            )
          
        
        Q
        (
        
          Q
          
            T
          
        
        Q
        
          )
          
            −
            1
          
        
        c
        ,
      
    
    {\displaystyle {\hat {\beta }}^{c}=R(R^{\operatorname {T} }X^{\operatorname {T} }XR)^{-1}R^{\operatorname {T} }X^{\operatorname {T} }y+{\Big (}I_{p}-R(R^{\operatorname {T} }X^{\operatorname {T} }XR)^{-1}R^{\operatorname {T} }X^{\operatorname {T} }X{\Big )}Q(Q^{\operatorname {T} }Q)^{-1}c,}

where R is a p×(p − q) matrix such that the matrix [Q R] is non-singular, and RTQ = 0. Such a matrix can always be found, although generally it is not unique. The second formula coincides with the first in case when XTX is invertible.<a class="footnote-ref" id="fnref:42" href="#fn:42">42</a>

<h3>Large sample properties</h3>
The least squares estimators are <a href="/facts/Point_estimate/AYsuptFp">point estimates</a> of the linear regression model parameters β. However, generally we also want to know how close those estimates might be to the true values of parameters. In other words, we want to construct the <a href="/facts/Interval_estimate/7G6VxM7I">interval estimates</a>.
Since we have not made any assumption about the distribution of error term εi, it is impossible to infer the distribution of the estimators 
 
 
 
 
 
 
 β
 ^
 
 
 
 
 
 {\displaystyle {\hat {\beta }}}
 
 and 
 
 
 
 
 
 
 
 σ
 ^
 
 
 
 
 2
 
 
 
 
 {\displaystyle {\hat {\sigma }}^{2}}
 
. Nevertheless, we can apply the <a href="/facts/Central_limit_theorem/0gc2L0Wd">central limit theorem</a> to derive their asymptotic properties as sample size n goes to infinity. While the sample size is necessarily finite, it is customary to assume that n is "large enough" so that the true distribution of the OLS estimator is close to its asymptotic limit.
We can show that under the model assumptions, the least squares estimator for β is <a href="/facts/Consistent_estimator/HAKDWYEM">consistent</a> (that is 
 
 
 
 
 
 
 β
 ^
 
 
 
 
 
 {\displaystyle {\hat {\beta }}}
 
 <a href="/facts/Convergence_of_random_variables/pWdatFlY">converges in probability</a> to β) and asymptotically normal:<a href="/facts/Proofs_involving_ordinary_least_squares/3kMzc73I">[proof]</a>

(
        
          
            
              β
              ^
            
          
        
        −
        β
        )
         
        
          
            →
            
              d
            
          
        
         
        
          
            N
          
        
        
          
            (
          
        
        0
        ,
        
        
          σ
          
            2
          
        
        
          Q
          
            x
            x
          
          
            −
            1
          
        
        
          
            )
          
        
        ,
      
    
    {\displaystyle ({\hat {\beta }}-\beta )\ {\xrightarrow {d}}\ {\mathcal {N}}{\big (}0,\;\sigma ^{2}Q_{xx}^{-1}{\big )},}

where 
 
 
 
 
 Q
 
 x
 x
 
 
 =
 
 X
 
 T
 
 
 X
 .
 
 
 {\displaystyle Q_{xx}=X^{\operatorname {T} }X.}

<h4>Intervals</h4>
Main articles: <a href="/facts/Confidence_interval/NS8mG5UE">Confidence interval</a> and <a href="/facts/Prediction_interval/0twP0IFI">Prediction interval</a>
Using this asymptotic distribution, approximate two-sided confidence intervals for the j-th component of the vector 
 
 
 
 
 
 
 β
 ^
 
 
 
 
 
 {\displaystyle {\hat {\beta }}}
 
 can be constructed as

β
 
 j
 
 
 ∈
 
 
 [
 
 
  
 
 
 
 
 β
 ^
 
 
 
 
 j
 
 
 ±
 
 q
 
 1
 −
 
 
 α
 2
 
 
 
 
 
 
 N
 
 
 (
 0
 ,
 1
 )
 
 
 
 
 
 
 
 
 
 σ
 ^
 
 
 
 
 2
 
 
 
 
 [
 
 Q
 
 x
 x
 
 
 −
 1
 
 
 ]
 
 
 j
 j
 
 
 
 
  
 
 
 ]
 
 
 
 
 {\displaystyle \beta _{j}\in {\bigg [}\ {\hat {\beta }}_{j}\pm q_{1-{\frac {\alpha }{2}}}^{{\mathcal {N}}(0,1)}\!{\sqrt {{\hat {\sigma }}^{2}\left[Q_{xx}^{-1}\right]_{jj}}}\ {\bigg ]}}
 
   at the 1 − α confidence level,
where q denotes the <a href="/facts/Quantile_function/sf0jbCXh">quantile function</a> of standard normal distribution, and [·]jj is the j-th diagonal element of a matrix.
Similarly, the least squares estimator for σ2 is also consistent and asymptotically normal (provided that the fourth moment of εi exists) with limiting distribution

(
        
          
            
              
                σ
                ^
              
            
          
          
            2
          
        
        −
        
          σ
          
            2
          
        
        )
         
        
          
            →
            
              d
            
          
        
         
        
          
            N
          
        
        
          (
          
            0
            ,
            
            E
            ⁡
            
              [
              
                ε
                
                  i
                
                
                  4
                
              
              ]
            
            −
            
              σ
              
                4
              
            
          
          )
        
        .
      
    
    {\displaystyle ({\hat {\sigma }}^{2}-\sigma ^{2})\ {\xrightarrow {d}}\ {\mathcal {N}}\left(0,\;\operatorname {E} \left[\varepsilon _{i}^{4}\right]-\sigma ^{4}\right).}

These asymptotic distributions can be used for prediction, testing hypotheses, constructing other estimators, etc.. As an example consider the problem of prediction. Suppose 
 
 
 
 
 x
 
 0
 
 
 
 
 {\displaystyle x_{0}}
 
 is some point within the domain of distribution of the regressors, and one wants to know what the response variable would have been at that point. The <a href="/facts/Mean_response/UCEhkAbG">mean response</a> is the quantity 
 
 
 
 
 y
 
 0
 
 
 =
 
 x
 
 0
 
 
 
 T
 
 
 
 β
 
 
 {\displaystyle y_{0}=x_{0}^{\mathrm {T} }\beta }
 
, whereas the <a href="/facts/Predicted_response/UCEhkAbG">predicted response</a> is 
 
 
 
 
 
 
 
 y
 ^
 
 
 
 
 0
 
 
 =
 
 x
 
 0
 
 
 
 T
 
 
 
 
 
 
 β
 ^
 
 
 
 
 
 {\displaystyle {\hat {y}}_{0}=x_{0}^{\mathrm {T} }{\hat {\beta }}}
 
. Clearly the predicted response is a random variable, its distribution can be derived from that of 
 
 
 
 
 
 
 β
 ^
 
 
 
 
 
 {\displaystyle {\hat {\beta }}}
 
:

(
          
            
              
                
                  
                    y
                    ^
                  
                
              
              
                0
              
            
            −
            
              y
              
                0
              
            
          
          )
        
         
        
          
            →
            
              d
            
          
        
         
        
          
            N
          
        
        
          (
          
            0
            ,
            
            
              σ
              
                2
              
            
            
              x
              
                0
              
              
                
                  T
                
              
            
            
              Q
              
                x
                x
              
              
                −
                1
              
            
            
              x
              
                0
              
            
          
          )
        
        ,
      
    
    {\displaystyle \left({\hat {y}}_{0}-y_{0}\right)\ {\xrightarrow {d}}\ {\mathcal {N}}\left(0,\;\sigma ^{2}x_{0}^{\mathrm {T} }Q_{xx}^{-1}x_{0}\right),}

which allows construct confidence intervals for mean response 
 
 
 
 
 y
 
 0
 
 
 
 
 {\displaystyle y_{0}}
 
 to be constructed:

y
 
 0
 
 
 ∈
 
 [
 
  
 
 x
 
 0
 
 
 
 T
 
 
 
 
 
 
 β
 ^
 
 
 
 ±
 
 q
 
 1
 −
 
 
 α
 2
 
 
 
 
 
 
 N
 
 
 (
 0
 ,
 1
 )
 
 
 
 
 
 
 
 
 
 σ
 ^
 
 
 
 
 2
 
 
 
 x
 
 0
 
 
 
 T
 
 
 
 
 Q
 
 x
 x
 
 
 −
 1
 
 
 
 x
 
 0
 
 
 
 
  
 
 ]
 
 
 
 {\displaystyle y_{0}\in \left[\ x_{0}^{\mathrm {T} }{\hat {\beta }}\pm q_{1-{\frac {\alpha }{2}}}^{{\mathcal {N}}(0,1)}\!{\sqrt {{\hat {\sigma }}^{2}x_{0}^{\mathrm {T} }Q_{xx}^{-1}x_{0}}}\ \right]}
 
   at the 1 − α confidence level.
<h4>Hypothesis testing</h4>
Main article: <a href="/facts/Hypothesis_testing/yv9RPF6U">Hypothesis testing</a>

Two hypothesis tests are particularly widely used. First, one wants to know if the estimated regression equation is any better than simply predicting that all values of the response variable equal its sample mean (if not, it is said to have no explanatory power). The <a href="/facts/Null_hypothesis/C8NwSE9J">null hypothesis</a> of no explanatory value of the estimated regression is tested using an <a href="/facts/F-test/Kq8lzNqY">F-test</a>. If the calculated F-value is found to be large enough to exceed its critical value for the pre-chosen level of significance, the null hypothesis is rejected and the <a href="/facts/Alternative_hypothesis/5cSH78Fm">alternative hypothesis</a>, that the regression has explanatory power, is accepted. Otherwise, the null hypothesis of no explanatory power is accepted.
Second, for each explanatory variable of interest, one wants to know whether its estimated coefficient differs significantly from zero—that is, whether this particular explanatory variable in fact has explanatory power in predicting the response variable. Here the null hypothesis is that the true coefficient is zero. This hypothesis is tested by computing the coefficient's <a href="/facts/T-statistic/eTlyuoqG">t-statistic</a>, as the ratio of the coefficient estimate to its <a href="/facts/Standard_error/Iv96AyCD">standard error</a>. If the t-statistic is larger than a predetermined value, the null hypothesis is rejected and the variable is found to have explanatory power, with its coefficient significantly different from zero. Otherwise, the null hypothesis of a zero value of the true coefficient is accepted.
In addition, the <a href="/facts/Chow_test/QYGUdETS">Chow test</a> is used to test whether two subsamples both have the same underlying true coefficient values. The sum of squared residuals of regressions on each of the subsets and on the combined data set are compared by computing an F-statistic; if this exceeds a critical value, the null hypothesis of no difference between the two subsets is rejected; otherwise, it is accepted.

<h2 id="example-with-real-data">Example with real data </h2>
See also: <a href="/facts/Simple_linear_regression/SNINCEFU">Simple linear regression § Example</a>, and <a href="/facts/Linear_least_squares/dKctFUCV">Linear least squares § Example</a>
The following data set gives average heights and weights for American women aged 30–39 (source: The World Almanac and Book of Facts, 1975).

<table><tbody><tr><th>Height (m)</th><td>1.47</td><td>1.50</td><td>1.52</td><td>1.55</td><td>1.57</td><td rowspan="6"></td></tr><tr><th>Weight (kg)</th><td>52.21</td><td>53.12</td><td>54.48</td><td>55.84</td><td>57.20</td></tr><tr><th>Height (m)</th><td>1.60</td><td>1.63</td><td>1.65</td><td>1.68</td><td>1.70</td></tr><tr><th>Weight (kg)</th><td>58.57</td><td>59.93</td><td>61.29</td><td>63.11</td><td>64.47</td></tr><tr><th>Height (m)</th><td>1.73</td><td>1.75</td><td>1.78</td><td>1.80</td><td>1.83</td></tr><tr><th>Weight (kg)</th><td>66.28</td><td>68.10</td><td>69.92</td><td>72.19</td><td>74.46</td></tr></tbody></table>
When only one dependent variable is being modeled, a <a href="/facts/Scatterplot/n03xBtBL">scatterplot</a> will suggest the form and strength of the relationship between the dependent variable and regressors. It might also reveal outliers, heteroscedasticity, and other aspects of the data that may complicate the interpretation of a fitted regression model. The scatterplot suggests that the relationship is strong and can be approximated as a quadratic function. OLS can handle non-linear relationships by introducing the regressor HEIGHT2. The regression model then becomes a multiple linear model:

w
          
            i
          
        
        =
        
          β
          
            1
          
        
        +
        
          β
          
            2
          
        
        
          h
          
            i
          
        
        +
        
          β
          
            3
          
        
        
          h
          
            i
          
          
            2
          
        
        +
        
          ε
          
            i
          
        
        .
      
    
    {\displaystyle w_{i}=\beta _{1}+\beta _{2}h_{i}+\beta _{3}h_{i}^{2}+\varepsilon _{i}.}

The output from most popular <a href="/facts/List_of_statistical_packages/QFqfrzWo">statistical packages</a> will look similar to this:

<table><tbody><tr><td>Method</td><td colspan="4">Least squares</td></tr><tr><td>Dependent variable</td><td colspan="4">WEIGHT</td></tr><tr><td>Observations</td><td colspan="4">15</td></tr><tr><td colspan="5"></td></tr><tr><th>Parameter</th><th>Value</th><th><a href="/facts/Standard_error/Iv96AyCD">Std error</a></th><th><a href="/facts/T-statistic/eTlyuoqG">t-statistic</a></th><th><a href="/facts/P-value/InnYd1bz">p-value</a></th></tr><tr><td colspan="5"></td></tr><tr><td> β 1 {\displaystyle \beta _{1}} </td><td>128.8128</td><td>16.3083</td><td>7.8986</td><td>0.0000</td></tr><tr><td> β 2 {\displaystyle \beta _{2}} </td><td>–143.1620</td><td>19.8332</td><td>–7.2183</td><td>0.0000</td></tr><tr><td> β 3 {\displaystyle \beta _{3}} </td><td>61.9603</td><td>6.0084</td><td>10.3122</td><td>0.0000</td></tr><tr><td colspan="5"></td></tr><tr><td><a href="/facts/Coefficient_of_determination/acBlgnrv">R2</a></td><td>0.9989</td><td colspan="2">S.E. of regression</td><td>0.2516</td></tr><tr><td>Adjusted R2</td><td>0.9987</td><td colspan="2">Model sum-of-sq.</td><td>692.61</td></tr><tr><td>Log-likelihood</td><td>1.0890</td><td colspan="2">Residual sum-of-sq.</td><td>0.7595</td></tr><tr><td><a href="/facts/Durbin%25E2%2580%2593Watson_statistic/3m4kYa5d">Durbin–Watson stat.</a></td><td>2.1013</td><td colspan="2">Total sum-of-sq.</td><td>693.37</td></tr><tr><td><a href="/facts/Akaike_information_criterion/RjKKgcnF">Akaike criterion</a></td><td>0.2548</td><td colspan="2">F-statistic</td><td>5471.2</td></tr><tr><td><a href="/facts/Schwarz_criterion/NHspnLrM">Schwarz criterion</a></td><td>0.3964</td><td colspan="2">p-value (F-stat)</td><td>0.0000</td></tr></tbody></table>
In this table:

<ul><li>The Value column gives the least squares estimates of parameters βj</li>
<li>The Std error column shows <a href="/facts/Standard_error_(statistics)/Iv96AyCD">standard errors</a> of each coefficient estimate: 
 
 
 
 
 
 
 
 σ
 ^
 
 
 
 
 j
 
 
 =
 
 
 (
 
 
 
 
 
 σ
 ^
 
 
 
 
 2
 
 
 
 
 [
 
 Q
 
 x
 x
 
 
 −
 1
 
 
 ]
 
 
 j
 j
 
 
 
 )
 
 
 
 1
 2
 
 
 
 
 
 {\displaystyle {\hat {\sigma }}_{j}=\left({\hat {\sigma }}^{2}\left[Q_{xx}^{-1}\right]_{jj}\right)^{\frac {1}{2}}}
 
</li>
<li>The <a href="/facts/T-statistic/eTlyuoqG">t-statistic</a> and p-value columns are testing whether any of the coefficients might be equal to zero. The t-statistic is calculated simply as 
 
 
 
 t
 =
 
 
 
 
 β
 ^
 
 
 
 
 j
 
 
 
 /
 
 
 
 
 
 σ
 ^
 
 
 
 
 j
 
 
 
 
 {\displaystyle t={\hat {\beta }}_{j}/{\hat {\sigma }}_{j}}
 
. If the errors ε follow a normal distribution, t follows a Student-t distribution. Under weaker conditions, t is asymptotically normal. Large values of t indicate that the null hypothesis can be rejected and that the corresponding coefficient is not zero. The second column, <a href="/facts/P-value/InnYd1bz">p-value</a>, expresses the results of the hypothesis test as a <a href="/facts/Statistical_significance/PMxjdlAu">significance level</a>. Conventionally, p-values smaller than 0.05 are taken as evidence that the population coefficient is nonzero.</li>
<li>R-squared is the <a href="/facts/Coefficient_of_determination/acBlgnrv">coefficient of determination</a> indicating goodness-of-fit of the regression. This statistic will be equal to one if fit is perfect, and to zero when regressors X have no explanatory power whatsoever. This is a biased estimate of the population R-squared, and will never decrease if additional regressors are added, even if they are irrelevant.</li>
<li>Adjusted R-squared is a slightly modified version of 
 
 
 
 
 R
 
 2
 
 
 
 
 {\displaystyle R^{2}}
 
, designed to penalize for the excess number of regressors which do not add to the explanatory power of the regression. This statistic is always smaller than 
 
 
 
 
 R
 
 2
 
 
 
 
 {\displaystyle R^{2}}
 
, can decrease as new regressors are added, and even be negative for poorly fitting models:</li></ul>

R
              ¯
            
          
          
            2
          
        
        =
        1
        −
        
          
            
              n
              −
              1
            
            
              n
              −
              p
            
          
        
        (
        1
        −
        
          R
          
            2
          
        
        )
      
    
    {\displaystyle {\overline {R}}^{2}=1-{\frac {n-1}{n-p}}(1-R^{2})}

<ul><li>Log-likelihood is calculated under the assumption that errors follow normal distribution. Even though the assumption is not very reasonable, this statistic may still find its use in conducting LR tests.</li>
<li><a href="/facts/Durbin%25E2%2580%2593Watson_statistic/3m4kYa5d">Durbin–Watson statistic</a> tests whether there is any evidence of serial correlation between the residuals. As a rule of thumb, the value smaller than 2 will be an evidence of positive correlation.</li>
<li><a href="/facts/Akaike_information_criterion/RjKKgcnF">Akaike information criterion</a> and <a href="/facts/Schwarz_criterion/NHspnLrM">Schwarz criterion</a> are both used for model selection. Generally when comparing two alternative models, smaller values of one of these criteria will indicate a better model.<a class="footnote-ref" id="fnref:43" href="#fn:43">43</a></li>
<li>Standard error of regression is an estimate of σ, standard error of the error term.</li>
<li>Total sum of squares, model sum of squared, and residual sum of squares tell us how much of the initial variation in the sample were explained by the regression.</li>
<li>F-statistic tries to test the hypothesis that all coefficients (except the intercept) are equal to zero. This statistic has F(p–1,n–p) distribution under the null hypothesis and normality assumption, and its p-value indicates probability that the hypothesis is indeed true. Note that when errors are not normal this statistic becomes invalid, and other tests such as <a href="/facts/Wald_test/bVs5k173">Wald test</a> or <a href="/facts/Likelihood_ratio_test/oFqLh3R0">LR test</a> should be used.</li></ul>

Ordinary least squares analysis often includes the use of diagnostic plots designed to detect departures of the data from the assumed form of the model. These are some of the common diagnostic plots:

<ul><li>Residuals against the explanatory variables in the model. A non-linear relation between these variables suggests that the linearity of the conditional mean function may not hold. Different levels of variability in the residuals for different levels of the explanatory variables suggests possible heteroscedasticity.</li>
<li>Residuals against explanatory variables not in the model. Any relation of the residuals to these variables would suggest considering these variables for inclusion in the model.</li>
<li>Residuals against the fitted values, 
 
 
 
 
 
 
 y
 ^
 
 
 
 
 
 {\displaystyle {\hat {y}}}
 
.</li>
<li>Residuals against the preceding residual. This plot may identify serial correlations in the residuals.</li></ul>
An important consideration when carrying out statistical inference using regression models is how the data were sampled. In this example, the data are averages rather than measurements on individual women. The fit of the model is very good, but this does not imply that the weight of an individual woman can be predicted with high accuracy based only on her height.

<h3>Sensitivity to rounding</h3>
Main article: <a href="/facts/Errors-in-variables_models/PRDddry0">Errors-in-variables models</a>
See also: <a href="/facts/Quantization_error_model/fddS3BE2">Quantization error model</a>
This example also demonstrates that coefficients determined by these calculations are sensitive to how the data is prepared. The heights were originally given rounded to the nearest inch and have been converted and rounded to the nearest centimetre. Since the conversion factor is one inch to 2.54 cm this is not an exact conversion. The original inches can be recovered by Round(x/0.0254) and then re-converted to metric without rounding. If this is done the results become:

<table><tbody><tr><th></th><th>Const</th><th>Height</th><th>Height2</th></tr><tr><td>Converted to metric with rounding.</td><td>128.8128</td><td>−143.162</td><td>61.96033</td></tr><tr><td>Converted to metric without rounding.</td><td>119.0205</td><td>−131.5076</td><td>58.5046</td></tr></tbody></table>

Using either of these equations to predict the weight of a 5' 6" (1.6764 m) woman gives similar values: 62.94 kg with rounding vs. 62.98 kg without rounding. Thus a seemingly small variation in the data has a real effect on the coefficients but a small effect on the results of the equation.
While this may look innocuous in the middle of the data range it could become significant at the extremes or in the case where the fitted model is used to project outside the data range (<a href="/facts/Extrapolation/wggUrqjF">extrapolation</a>).
This highlights a common error: this example is an abuse of OLS which inherently requires that the errors in the independent variable (in this case height) are zero or at least negligible. The initial rounding to nearest inch plus any actual measurement errors constitute a finite and non-negligible error. As a result, the fitted parameters are not the best estimates they are presumed to be. Though not totally spurious the error in the estimation will depend upon relative size of the x and y errors.

<h2 id="another-example-with-less-real-data">Another example with less real data</h2>
<h3>Problem statement</h3>
We can use the least square mechanism to figure out the equation of a two body orbit in polar base co-ordinates. The equation typically used is 
 
 
 
 r
 (
 θ
 )
 =
 
 
 p
 
 1
 −
 e
 cos
 ⁡
 (
 θ
 )
 
 
 
 
 
 {\displaystyle r(\theta )={\frac {p}{1-e\cos(\theta )}}}
 
 where 
 
 
 
 r
 (
 θ
 )
 
 
 {\displaystyle r(\theta )}
 
 is the radius of how far the object is from one of the bodies. In the equation the parameters 
 
 
 
 p
 
 
 {\displaystyle p}
 
 and 
 
 
 
 e
 
 
 {\displaystyle e}
 
 are used to determine the path of the orbit. We have measured the following data. 

<table><tbody><tr><th> θ {\displaystyle \theta } (in degrees)</th><td>43</td><td>45</td><td>52</td><td>93</td><td>108</td><td>116</td></tr><tr><th> r ( θ ) {\displaystyle r(\theta )} </th><td>4.7126</td><td>4.5542</td><td>4.0419</td><td>2.2187</td><td>1.8910</td><td>1.7599</td></tr></tbody></table>
We need to find the least-squares approximation of 
 
 
 
 e
 
 
 {\displaystyle e}
 
 and 
 
 
 
 p
 
 
 {\displaystyle p}
 
 for the given data.

<h3>Solution</h3>
First we need to represent e and p in a linear form. So we are going to rewrite the equation 
 
 
 
 r
 (
 θ
 )
 
 
 {\displaystyle r(\theta )}
 
 as 
 
 
 
 
 
 1
 
 r
 (
 θ
 )
 
 
 
 =
 
 
 1
 p
 
 
 −
 
 
 e
 p
 
 
 cos
 ⁡
 (
 θ
 )
 
 
 {\displaystyle {\frac {1}{r(\theta )}}={\frac {1}{p}}-{\frac {e}{p}}\cos(\theta )}
 
. Furthermore, one could fit for <a href="/facts/Apsides/VaL6ow9w">apsides</a> by expanding 
 
 
 
 cos
 ⁡
 (
 θ
 )
 
 
 {\displaystyle \cos(\theta )}
 
 with an extra parameter as 
 
 
 
 cos
 ⁡
 (
 θ
 −
 
 θ
 
 0
 
 
 )
 =
 cos
 ⁡
 (
 θ
 )
 cos
 ⁡
 (
 
 θ
 
 0
 
 
 )
 +
 sin
 ⁡
 (
 θ
 )
 sin
 ⁡
 (
 
 θ
 
 0
 
 
 )
 
 
 {\displaystyle \cos(\theta -\theta _{0})=\cos(\theta )\cos(\theta _{0})+\sin(\theta )\sin(\theta _{0})}
 
, which is linear in both 
 
 
 
 cos
 ⁡
 (
 θ
 )
 
 
 {\displaystyle \cos(\theta )}
 
 and in the extra basis function 
 
 
 
 sin
 ⁡
 (
 θ
 )
 
 
 {\displaystyle \sin(\theta )}
 
, used to extra 
 
 
 
 tan
 ⁡
 
 θ
 
 0
 
 
 =
 sin
 ⁡
 (
 
 θ
 
 0
 
 
 )
 
 /
 
 cos
 ⁡
 (
 
 θ
 
 0
 
 
 )
 
 
 {\displaystyle \tan \theta _{0}=\sin(\theta _{0})/\cos(\theta _{0})}
 
. We use the original two-parameter form to represent our observational data as:

 
 
 
 
 A
 
 T
 
 
 A
 
 
 
 (
 
 
 x
 y
 
 
 )
 
 
 
 =
 
 A
 
 T
 
 
 b
 
 
 {\displaystyle A^{T}A{\binom {x}{y}}=A^{T}b}
 
 where 
 
 
 
 x
 
 
 {\displaystyle x}
 
 is 
 
 
 
 
 
 1
 p
 
 
 
 
 {\displaystyle {\frac {1}{p}}}
 
 and 
 
 
 
 y
 
 
 {\displaystyle y}
 
 is 
 
 
 
 
 
 e
 p
 
 
 
 
 {\displaystyle {\frac {e}{p}}}
 
 and 
 
 
 
 A
 
 
 {\displaystyle A}
 
 is constructed by the first column being the coefficient of 
 
 
 
 
 
 1
 p
 
 
 
 
 {\displaystyle {\frac {1}{p}}}
 
 and the second column being the coefficient of 
 
 
 
 
 
 e
 p
 
 
 
 
 {\displaystyle {\frac {e}{p}}}
 
 and 
 
 
 
 b
 
 
 {\displaystyle b}
 
 is the values for the respective 
 
 
 
 
 
 1
 
 r
 (
 θ
 )
 
 
 
 
 
 {\displaystyle {\frac {1}{r(\theta )}}}
 
 so 
 
 
 
 A
 =
 
 
 [
 
 
 
 1
 
 
 −
 0.731354
 
 
 
 
 1
 
 
 −
 0.707107
 
 
 
 
 1
 
 
 −
 0.615661
 
 
 
 
 1
 
 
  
 0.052336
 
 
 
 
 1
 
 
 0.309017
 
 
 
 
 1
 
 
 0.438371
 
 
 
 ]
 
 
 
 
 {\displaystyle A={\begin{bmatrix}1&-0.731354\\1&-0.707107\\1&-0.615661\\1&\ 0.052336\\1&0.309017\\1&0.438371\end{bmatrix}}}
 
 and 
 
 
 
 b
 =
 
 
 [
 
 
 
 0.21220
 
 
 
 
 0.21958
 
 
 
 
 0.24741
 
 
 
 
 0.45071
 
 
 
 
 0.52883
 
 
 
 
 0.56820
 
 
 
 ]
 
 
 .
 
 
 {\displaystyle b={\begin{bmatrix}0.21220\\0.21958\\0.24741\\0.45071\\0.52883\\0.56820\end{bmatrix}}.}

On solving we get 
 
 
 
 
 
 
 (
 
 
 x
 y
 
 
 )
 
 
 
 =
 
 
 
 (
 
 
 0.43478
 0.30435
 
 
 )
 
 
 
 
 
 {\displaystyle {\binom {x}{y}}={\binom {0.43478}{0.30435}}}

so 
 
 
 
 p
 =
 
 
 1
 x
 
 
 =
 2.3000
 
 
 {\displaystyle p={\frac {1}{x}}=2.3000}
 
 and 
 
 
 
 e
 =
 p
 ⋅
 y
 =
 0.70001
 
 
 {\displaystyle e=p\cdot y=0.70001}
 


<h2 id="see-also">See also</h2>
<ul><li><a href="/facts/Minimum_mean_square_error/hI2lqEMh">Bayesian least squares</a></li>
<li><a href="/facts/Fama%25E2%2580%2593MacBeth_regression/y5cfWSl1">Fama–MacBeth regression</a></li>
<li><a href="/facts/Non-linear_least_squares/3K5Nmz0S">Nonlinear least squares</a></li>
<li><a href="/facts/Numerical_methods_for_linear_least_squares/JQmo6jj6">Numerical methods for linear least squares</a></li>
<li><a href="/facts/Nonlinear_system_identification/XnBWbGdF">Nonlinear system identification</a></li></ul>

<h2 id="further-reading">Further reading</h2>
<ul><li>Dougherty, Christopher (2002). Introduction to Econometrics (2nd ed.). New York: Oxford University Press. pp. 48–113. <a href="/facts/ISBN_(identifier)/15AdSPa9">ISBN</a> 0-19-877643-8.</li>
<li><a href="/facts/Damodar_N._Gujarati/ls400zNK">Gujarati, Damodar N.</a>; <a href="/facts/Dawn_C._Porter/Py48nLE6">Porter, Dawn C.</a> (2009). Basic Econometics (Fifth ed.). Boston: McGraw-Hill Irwin. pp. 55–96. <a href="/facts/ISBN_(identifier)/15AdSPa9">ISBN</a> 978-0-07-337577-9.</li>
<li>Heij, Christiaan; Boer, Paul; Franses, Philip H.; Kloek, Teun; van Dijk, Herman K. (2004). Econometric Methods with Applications in Business and Economics (1st ed.). Oxford: Oxford University Press. pp. 76–115. <a href="/facts/ISBN_(identifier)/15AdSPa9">ISBN</a> 978-0-19-926801-6.</li>
<li>Hill, R. Carter; Griffiths, William E.; Lim, Guay C. (2008). Principles of Econometrics (3rd ed.). Hoboken, NJ: John Wiley & Sons. pp. 8–47. <a href="/facts/ISBN_(identifier)/15AdSPa9">ISBN</a> 978-0-471-72360-8.</li>
<li><a href="/facts/Jeffrey_Wooldridge/XdwAoA1m">Wooldridge, Jeffrey</a> (2008). <a href="https://books.google.com/books?id=64vt5TDBNLwC&pg=PA22">"The Simple Regression Model"</a>. Introductory Econometrics: A Modern Approach (4th ed.). Mason, OH: Cengage Learning. pp. 22–67. <a href="/facts/ISBN_(identifier)/15AdSPa9">ISBN</a> 978-0-324-58162-1.</li></ul>

<h2 id="references">References</h2>

<ol>
<li id="fn:1">"The Origins of Ordinary Least Squares Assumptions". Feature Column. 2022-03-01. Retrieved 2024-05-16. <a href="https://mathvoices.ams.org/featurecolumn/2022/03/01/ordinary-least-squares/" target="_blank">https://mathvoices.ams.org/featurecolumn/2022/03/01/ordinary-least-squares/</a> <a href="#fnref:1" class="footnote-back-ref">↩</a></li>
<li id="fn:2">"What is a complete list of the usual assumptions for linear regression?". Cross Validated. Retrieved 2022-09-28. <a href="https://stats.stackexchange.com/q/16381" target="_blank">https://stats.stackexchange.com/q/16381</a> <a href="#fnref:2" class="footnote-back-ref">↩</a></li>
<li id="fn:3">Goldberger, Arthur S. (1964). "Classical Linear Regression". Econometric Theory. New York: John Wiley & Sons. pp. 158. ISBN 0-471-31101-4. {{cite book}}: ISBN / Date incompatibility (help) <a href="0-471-31101-4" target="_blank">0-471-31101-4</a> <a href="#fnref:3" class="footnote-back-ref">↩</a></li>
<li id="fn:4">Hayashi, Fumio (2000). Econometrics. Princeton University Press. p. 15. <a href="/wiki/Fumio_Hayashi" target="_blank">/wiki/Fumio_Hayashi</a> <a href="#fnref:4" class="footnote-back-ref">↩</a></li>
<li id="fn:5">Hayashi (2000, page 18). - Hayashi, Fumio (2000). Econometrics. Princeton University Press. p. 15. <a href="#fnref:5" class="footnote-back-ref">↩</a></li>
<li id="fn:6">Ghilani, Charles D.; Paul r. Wolf, Ph. D. (12 June 2006). Adjustment Computations: Spatial Data Analysis. John Wiley & Sons. ISBN 9780471697282. <a href="9780471697282" target="_blank">9780471697282</a> <a href="#fnref:6" class="footnote-back-ref">↩</a></li>
<li id="fn:7">Hofmann-Wellenhof, Bernhard; Lichtenegger, Herbert; Wasle, Elmar (20 November 2007). GNSS – Global Navigation Satellite Systems: GPS, GLONASS, Galileo, and more. Springer. ISBN 9783211730171. <a href="9783211730171" target="_blank">9783211730171</a> <a href="#fnref:7" class="footnote-back-ref">↩</a></li>
<li id="fn:8">Xu, Guochang (5 October 2007). GPS: Theory, Algorithms and Applications. Springer. ISBN 9783540727156. <a href="9783540727156" target="_blank">9783540727156</a> <a href="#fnref:8" class="footnote-back-ref">↩</a></li>
<li id="fn:9">Hayashi (2000, page 19) - Hayashi, Fumio (2000). Econometrics. Princeton University Press. p. 15. <a href="#fnref:9" class="footnote-back-ref">↩</a></li>
<li id="fn:10">Hoaglin, David C.; Welsch, Roy E. (1978). "The Hat Matrix in Regression and ANOVA". The American Statistician. 32 (1): 17–22. doi:10.1080/00031305.1978.10479237. hdl:1721.1/1920. ISSN 0003-1305. <a href="https://doi.org/10.1080%2F00031305.1978.10479237" target="_blank">https://doi.org/10.1080%2F00031305.1978.10479237</a> <a href="#fnref:10" class="footnote-back-ref">↩</a></li>
<li id="fn:11">Julian Faraway (2000), Practical Regression and Anova using R <a href="https://cran.r-project.org/doc/contrib/Faraway-PRA.pdf" target="_blank">https://cran.r-project.org/doc/contrib/Faraway-PRA.pdf</a> <a href="#fnref:11" class="footnote-back-ref">↩</a></li>
<li id="fn:12">Kenney, J.; Keeping, E. S. (1963). Mathematics of Statistics. van Nostrand. p. 187. <a href="#fnref:12" class="footnote-back-ref">↩</a></li>
<li id="fn:13">Zwillinger, D. (1995). Standard Mathematical Tables and Formulae. Chapman&Hall/CRC. p. 626. ISBN 0-8493-2479-3. <a href="0-8493-2479-3" target="_blank">0-8493-2479-3</a> <a href="#fnref:13" class="footnote-back-ref">↩</a></li>
<li id="fn:14">Hayashi (2000, page 19) - Hayashi, Fumio (2000). Econometrics. Princeton University Press. p. 15. <a href="#fnref:14" class="footnote-back-ref">↩</a></li>
<li id="fn:15">Hayashi (2000, page 20) - Hayashi, Fumio (2000). Econometrics. Princeton University Press. p. 15. <a href="#fnref:15" class="footnote-back-ref">↩</a></li>
<li id="fn:16">Akbarzadeh, Vahab (7 May 2014). "Line Estimation". <a href="https://mlmadesimple.wordpress.com/2014/05/07/line-estimation/" target="_blank">https://mlmadesimple.wordpress.com/2014/05/07/line-estimation/</a> <a href="#fnref:16" class="footnote-back-ref">↩</a></li>
<li id="fn:17">Hayashi (2000, page 49) - Hayashi, Fumio (2000). Econometrics. Princeton University Press. p. 15. <a href="#fnref:17" class="footnote-back-ref">↩</a></li>
<li id="fn:18">Hayashi (2000, page 52) - Hayashi, Fumio (2000). Econometrics. Princeton University Press. p. 15. <a href="#fnref:18" class="footnote-back-ref">↩</a></li>
<li id="fn:19">Hayashi (2000, page 7) - Hayashi, Fumio (2000). Econometrics. Princeton University Press. p. 15. <a href="#fnref:19" class="footnote-back-ref">↩</a></li>
<li id="fn:20">Hayashi (2000, page 187) - Hayashi, Fumio (2000). Econometrics. Princeton University Press. p. 15. <a href="#fnref:20" class="footnote-back-ref">↩</a></li>
<li id="fn:21">Hayashi (2000, page 10) - Hayashi, Fumio (2000). Econometrics. Princeton University Press. p. 15. <a href="#fnref:21" class="footnote-back-ref">↩</a></li>
<li id="fn:22">Hayashi (2000, page 10) - Hayashi, Fumio (2000). Econometrics. Princeton University Press. p. 15. <a href="#fnref:22" class="footnote-back-ref">↩</a></li>
<li id="fn:23">Hayashi (2000, page 34) - Hayashi, Fumio (2000). Econometrics. Princeton University Press. p. 15. <a href="#fnref:23" class="footnote-back-ref">↩</a></li>
<li id="fn:24">Williams, M. N; Grajales, C. A. G; Kurkiewicz, D (2013). "Assumptions of multiple regression: Correcting two misconceptions". Practical Assessment, Research & Evaluation. 18 (11). <a href="https://scholarworks.umass.edu/cgi/viewcontent.cgi?article=1308&context=pare" target="_blank">https://scholarworks.umass.edu/cgi/viewcontent.cgi?article=1308&context=pare</a> <a href="#fnref:24" class="footnote-back-ref">↩</a></li>
<li id="fn:25">"Memento on EViews Output" (PDF). Retrieved 28 December 2020. <a href="https://scholar.harvard.edu/files/jbenchimol/files/memento-eviews.pdf" target="_blank">https://scholar.harvard.edu/files/jbenchimol/files/memento-eviews.pdf</a> <a href="#fnref:25" class="footnote-back-ref">↩</a></li>
<li id="fn:26">Hayashi (2000, pages 27, 30) - Hayashi, Fumio (2000). Econometrics. Princeton University Press. p. 15. <a href="#fnref:26" class="footnote-back-ref">↩</a></li>
<li id="fn:27">Hayashi (2000, page 27) - Hayashi, Fumio (2000). Econometrics. Princeton University Press. p. 15. <a href="#fnref:27" class="footnote-back-ref">↩</a></li>
<li id="fn:28">Hayashi (2000, page 27) - Hayashi, Fumio (2000). Econometrics. Princeton University Press. p. 15. <a href="#fnref:28" class="footnote-back-ref">↩</a></li>
<li id="fn:29">Hayashi (2000, page 27) - Hayashi, Fumio (2000). Econometrics. Princeton University Press. p. 15. <a href="#fnref:29" class="footnote-back-ref">↩</a></li>
<li id="fn:30">Amemiya, Takeshi (1985). Advanced Econometrics. Harvard University Press. p. 13. ISBN 9780674005600. <a href="9780674005600" target="_blank">9780674005600</a> <a href="#fnref:30" class="footnote-back-ref">↩</a></li>
<li id="fn:31">Hayashi (2000, page 52) - Hayashi, Fumio (2000). Econometrics. Princeton University Press. p. 15. <a href="#fnref:31" class="footnote-back-ref">↩</a></li>
<li id="fn:32">Amemiya (1985, page 14) - Amemiya, Takeshi (1985). Advanced Econometrics. Harvard University Press. p. 13. ISBN 9780674005600. <a href="https://archive.org/details/advancedeconomet00amem" target="_blank">https://archive.org/details/advancedeconomet00amem</a> <a href="#fnref:32" class="footnote-back-ref">↩</a></li>
<li id="fn:33">Rao, C. R. (1973). Linear Statistical Inference and its Applications (Second ed.). New York: J. Wiley & Sons. p. 319. ISBN 0-471-70823-2. <a href="0-471-70823-2" target="_blank">0-471-70823-2</a> <a href="#fnref:33" class="footnote-back-ref">↩</a></li>
<li id="fn:34">Amemiya (1985, page 20) - Amemiya, Takeshi (1985). Advanced Econometrics. Harvard University Press. p. 13. ISBN 9780674005600. <a href="https://archive.org/details/advancedeconomet00amem" target="_blank">https://archive.org/details/advancedeconomet00amem</a> <a href="#fnref:34" class="footnote-back-ref">↩</a></li>
<li id="fn:35">Amemiya (1985, page 27) - Amemiya, Takeshi (1985). Advanced Econometrics. Harvard University Press. p. 13. ISBN 9780674005600. <a href="https://archive.org/details/advancedeconomet00amem" target="_blank">https://archive.org/details/advancedeconomet00amem</a> <a href="#fnref:35" class="footnote-back-ref">↩</a></li>
<li id="fn:36">Davidson, Russell; MacKinnon, James G. (1993). Estimation and Inference in Econometrics. New York: Oxford University Press. p. 33. ISBN 0-19-506011-3. <a href="0-19-506011-3" target="_blank">0-19-506011-3</a> <a href="#fnref:36" class="footnote-back-ref">↩</a></li>
<li id="fn:37">Davidson, Russell; MacKinnon, James G. (1993). Estimation and Inference in Econometrics. New York: Oxford University Press. p. 33. ISBN 0-19-506011-3. <a href="0-19-506011-3" target="_blank">0-19-506011-3</a> <a href="#fnref:37" class="footnote-back-ref">↩</a></li>
<li id="fn:38">Davidson & MacKinnon (1993, page 36) - Davidson, Russell; MacKinnon, James G. (1993). Estimation and Inference in Econometrics. New York: Oxford University Press. p. 33. ISBN 0-19-506011-3. <a href="#fnref:38" class="footnote-back-ref">↩</a></li>
<li id="fn:39">Davidson & MacKinnon (1993, page 20) - Davidson, Russell; MacKinnon, James G. (1993). Estimation and Inference in Econometrics. New York: Oxford University Press. p. 33. ISBN 0-19-506011-3. <a href="#fnref:39" class="footnote-back-ref">↩</a></li>
<li id="fn:40">Amemiya (1985, page 21) - Amemiya, Takeshi (1985). Advanced Econometrics. Harvard University Press. p. 13. ISBN 9780674005600. <a href="https://archive.org/details/advancedeconomet00amem" target="_blank">https://archive.org/details/advancedeconomet00amem</a> <a href="#fnref:40" class="footnote-back-ref">↩</a></li>
<li id="fn:41">Amemiya (1985, page 22) - Amemiya, Takeshi (1985). Advanced Econometrics. Harvard University Press. p. 13. ISBN 9780674005600. <a href="https://archive.org/details/advancedeconomet00amem" target="_blank">https://archive.org/details/advancedeconomet00amem</a> <a href="#fnref:41" class="footnote-back-ref">↩</a></li>
<li id="fn:42">Amemiya (1985, page 22) - Amemiya, Takeshi (1985). Advanced Econometrics. Harvard University Press. p. 13. ISBN 9780674005600. <a href="https://archive.org/details/advancedeconomet00amem" target="_blank">https://archive.org/details/advancedeconomet00amem</a> <a href="#fnref:42" class="footnote-back-ref">↩</a></li>
<li id="fn:43">Burnham, Kenneth P.; David Anderson (2002). Model Selection and Multi-Model Inference (2nd ed.). Springer. ISBN 0-387-95364-7. <a href="0-387-95364-7" target="_blank">0-387-95364-7</a> <a href="#fnref:43" class="footnote-back-ref">↩</a></li>
</ol>

Ordinary least squares open-in-new

Ordinary least squares