Semiparametric regression

<h2 id="methods">Methods</h2>
<p>Many different semiparametric regression methods have been proposed and developed. The most popular methods are the partially linear, index and varying coefficient models.
</p>
<h3>Partially linear models</h3>
<p>A <a href="/facts/Partially_linear_model/TPUl6spH">partially linear model</a> is given by
</p>

Y
          
            i
          
        
        =
        
          X
          
            i
          
          ′
        
        β
        +
        g
        
          (
          
            Z
            
              i
            
          
          )
        
        +
        
          u
          
            i
          
        
        ,
        
        
        i
        =
        1
        ,
        …
        ,
        n
        ,
        
      
    
    {\displaystyle Y_{i}=X'_{i}\beta +g\left(Z_{i}\right)+u_{i},\,\quad i=1,\ldots ,n,\,}

<p>where 
  
    
      
        
          Y
          
            i
          
        
      
    
    {\displaystyle Y_{i}}
  
 is the dependent variable, 
  
    
      
        
          X
          
            i
          
        
      
    
    {\displaystyle X_{i}}
  
 is a 
  
    
      
        p
        ×
        1
      
    
    {\displaystyle p\times 1}
  
 vector of explanatory variables, 
  
    
      
        β
      
    
    {\displaystyle \beta }
  
 is a 
  
    
      
        p
        ×
        1
      
    
    {\displaystyle p\times 1}
  
 vector of unknown parameters and 
  
    
      
        
          Z
          
            i
          
        
        ∈
        
          R
          
            q
          
        
      
    
    {\displaystyle Z_{i}\in \operatorname {R} ^{q}}
  
. The parametric part of the partially linear model is given by the parameter vector 
  
    
      
        β
      
    
    {\displaystyle \beta }
  
 while the nonparametric part is the unknown function 
  
    
      
        g
        
          (
          
            Z
            
              i
            
          
          )
        
      
    
    {\displaystyle g\left(Z_{i}\right)}
  
. The data is assumed to be i.i.d. with 
  
    
      
        E
        
          (
          
            
              u
              
                i
              
            
            
              |
            
            
              X
              
                i
              
            
            ,
            
              Z
              
                i
              
            
          
          )
        
        =
        0
      
    
    {\displaystyle E\left(u_{i}|X_{i},Z_{i}\right)=0}
  
 and the model allows for a conditionally <a href="/facts/Heteroskedastic/msBQWI7G">heteroskedastic</a> error process 
  
    
      
        E
        
          (
          
            
              u
              
                i
              
              
                2
              
            
            
              |
            
            x
            ,
            z
          
          )
        
        =
        
          σ
          
            2
          
        
        
          (
          
            x
            ,
            z
          
          )
        
      
    
    {\displaystyle E\left(u_{i}^{2}|x,z\right)=\sigma ^{2}\left(x,z\right)}
  
 of unknown form. This type of model was proposed by Robinson (1988) and extended to handle categorical covariates by Racine and Li (2007).
</p><p>This method is implemented by obtaining a 
  
    
      
        
          
            n
          
        
      
    
    {\displaystyle {\sqrt {n}}}
  
 consistent estimator of 
  
    
      
        β
      
    
    {\displaystyle \beta }
  
 and then deriving an estimator of 
  
    
      
        g
        
          (
          
            Z
            
              i
            
          
          )
        
      
    
    {\displaystyle g\left(Z_{i}\right)}
  
 from the <a href="/facts/Kernel_regression/7Txccjd4">nonparametric regression</a> of 
  
    
      
        
          Y
          
            i
          
        
        −
        
          X
          
            i
          
          ′
        
        
          
            
              β
              ^
            
          
        
      
    
    {\displaystyle Y_{i}-X'_{i}{\hat {\beta }}}
  
 on 
  
    
      
        z
      
    
    {\displaystyle z}
  
 using an appropriate nonparametric regression method.<a class="footnote-ref" id="fnref:1" href="#fn:1"><sup>1</sup></a>
</p>
<h3>Index models</h3>
<p>A single index model takes the form
</p>

Y
        =
        g
        
          (
          
            
              X
              ′
            
            
              β
              
                0
              
            
          
          )
        
        +
        u
        ,
        
      
    
    {\displaystyle Y=g\left(X'\beta _{0}\right)+u,\,}

<p>where 
  
    
      
        Y
      
    
    {\displaystyle Y}
  
, 
  
    
      
        X
      
    
    {\displaystyle X}
  
 and 
  
    
      
        
          β
          
            0
          
        
      
    
    {\displaystyle \beta _{0}}
  
 are defined as earlier and the error term 
  
    
      
        u
      
    
    {\displaystyle u}
  
 satisfies 
  
    
      
        E
        
          (
          
            u
            
              |
            
            X
          
          )
        
        =
        0
      
    
    {\displaystyle E\left(u|X\right)=0}
  
. The single index model takes its name from the parametric part of the model 
  
    
      
        
          x
          ′
        
        β
      
    
    {\displaystyle x'\beta }
  
 which is a <i>scalar</i> single index. The nonparametric part is the unknown function 
  
    
      
        g
        
          (
          ⋅
          )
        
      
    
    {\displaystyle g\left(\cdot \right)}
  
.
</p>
<h4>Ichimura's method</h4>
<p>The single index model method developed by Ichimura (1993) is as follows. Consider the situation in which 
  
    
      
        y
      
    
    {\displaystyle y}
  
 is continuous. Given a known form for the function 
  
    
      
        g
        
          (
          ⋅
          )
        
      
    
    {\displaystyle g\left(\cdot \right)}
  
, 
  
    
      
        
          β
          
            0
          
        
      
    
    {\displaystyle \beta _{0}}
  
 could be estimated using the <a href="/facts/Non-linear_least_squares/3K5Nmz0S">nonlinear least squares</a> method to minimize the function
</p>

∑
          
            i
            =
            1
          
        
        
          
            (
            
              
                Y
                
                  i
                
              
              −
              g
              
                (
                
                  
                    X
                    
                      i
                    
                    ′
                  
                  β
                
                )
              
            
            )
          
          
            2
          
        
        .
      
    
    {\displaystyle \sum _{i=1}\left(Y_{i}-g\left(X'_{i}\beta \right)\right)^{2}.}

<p>Since the functional form of 
  
    
      
        g
        
          (
          ⋅
          )
        
      
    
    {\displaystyle g\left(\cdot \right)}
  
 is not known, we need to estimate it. For a given value for 
  
    
      
        β
      
    
    {\displaystyle \beta }
  
 an estimate of the function
</p>

G
        
          (
          
            
              X
              
                i
              
              ′
            
            β
          
          )
        
        =
        E
        
          (
          
            
              Y
              
                i
              
            
            
              |
            
            
              X
              
                i
              
              ′
            
            β
          
          )
        
        =
        E
        
          [
          
            g
            
              (
              
                
                  X
                  
                    i
                  
                  ′
                
                
                  β
                  
                    o
                  
                
              
              )
            
            
              |
            
            
              X
              
                i
              
              ′
            
            β
          
          ]
        
      
    
    {\displaystyle G\left(X'_{i}\beta \right)=E\left(Y_{i}|X'_{i}\beta \right)=E\left[g\left(X'_{i}\beta _{o}\right)|X'_{i}\beta \right]}

<p>using <a href="/facts/Kernel_density_estimation/NmznyEW8">kernel</a> method. Ichimura (1993) proposes estimating 
  
    
      
        g
        
          (
          
            
              X
              
                i
              
              ′
            
            β
          
          )
        
      
    
    {\displaystyle g\left(X'_{i}\beta \right)}
  
 with
</p>

G
                ^
              
            
          
          
            −
            i
          
        
        
          (
          
            
              X
              
                i
              
              ′
            
            β
          
          )
        
        ,
        
      
    
    {\displaystyle {\hat {G}}_{-i}\left(X'_{i}\beta \right),\,}

<p>the <a href="/facts/Resampling_(statistics)/yq6LXqnX">leave-one-out</a> <a href="/facts/Kernel_density_estimation/NmznyEW8">nonparametric kernel</a> estimator of 
  
    
      
        G
        
          (
          
            
              X
              
                i
              
              ′
            
            β
          
          )
        
      
    
    {\displaystyle G\left(X'_{i}\beta \right)}
  
.
</p>
<h4>Klein and Spady's estimator</h4>
<p>If the dependent variable 
  
    
      
        y
      
    
    {\displaystyle y}
  
 is binary and 
  
    
      
        
          X
          
            i
          
        
      
    
    {\displaystyle X_{i}}
  
 and 
  
    
      
        
          u
          
            i
          
        
      
    
    {\displaystyle u_{i}}
  
 are assumed to be <a href="/facts/Independence_(probability_theory)/NUzQtnUL">independent</a>, Klein and Spady (1993) propose a technique for estimating 
  
    
      
        β
      
    
    {\displaystyle \beta }
  
 using <a href="/facts/Maximum_likelihood/0Yq2dpQD">maximum likelihood</a> methods. The log-likelihood function is given by
</p>

L
        
          (
          β
          )
        
        =
        
          ∑
          
            i
          
        
        
          (
          
            1
            −
            
              Y
              
                i
              
            
          
          )
        
        ln
        ⁡
        
          (
          
            1
            −
            
              
                
                  
                    g
                    ^
                  
                
              
              
                −
                i
              
            
            
              (
              
                
                  X
                  
                    i
                  
                  ′
                
                β
              
              )
            
          
          )
        
        +
        
          ∑
          
            i
          
        
        
          Y
          
            i
          
        
        ln
        ⁡
        
          (
          
            
              
                
                  
                    g
                    ^
                  
                
              
              
                −
                i
              
            
            
              (
              
                
                  X
                  
                    i
                  
                  ′
                
                β
              
              )
            
          
          )
        
        ,
      
    
    {\displaystyle L\left(\beta \right)=\sum _{i}\left(1-Y_{i}\right)\ln \left(1-{\hat {g}}_{-i}\left(X'_{i}\beta \right)\right)+\sum _{i}Y_{i}\ln \left({\hat {g}}_{-i}\left(X'_{i}\beta \right)\right),}

<p>where 
  
    
      
        
          
            
              
                g
                ^
              
            
          
          
            −
            i
          
        
        
          (
          
            
              X
              
                i
              
              ′
            
            β
          
          )
        
      
    
    {\displaystyle {\hat {g}}_{-i}\left(X'_{i}\beta \right)}
  
 is the <a href="/facts/Resampling_(statistics)/yq6LXqnX">leave-one-out</a> estimator.
</p>
<h3>Smooth coefficient/varying coefficient models</h3>
<p>Hastie and Tibshirani (1993) propose a smooth coefficient model given by
</p>

Y
          
            i
          
        
        =
        α
        
          (
          
            Z
            
              i
            
          
          )
        
        +
        
          X
          
            i
          
          ′
        
        β
        
          (
          
            Z
            
              i
            
          
          )
        
        +
        
          u
          
            i
          
        
        =
        
          (
          
            1
            +
            
              X
              
                i
              
              ′
            
          
          )
        
        
          (
          
            
              
                
                  α
                  
                    (
                    
                      Z
                      
                        i
                      
                    
                    )
                  
                
              
              
                
                  β
                  
                    (
                    
                      Z
                      
                        i
                      
                    
                    )
                  
                
              
            
          
          )
        
        +
        
          u
          
            i
          
        
        =
        
          W
          
            i
          
          ′
        
        γ
        
          (
          
            Z
            
              i
            
          
          )
        
        +
        
          u
          
            i
          
        
        ,
      
    
    {\displaystyle Y_{i}=\alpha \left(Z_{i}\right)+X'_{i}\beta \left(Z_{i}\right)+u_{i}=\left(1+X'_{i}\right)\left({\begin{array}{c}\alpha \left(Z_{i}\right)\\\beta \left(Z_{i}\right)\end{array}}\right)+u_{i}=W'_{i}\gamma \left(Z_{i}\right)+u_{i},}

<p>where 
  
    
      
        
          X
          
            i
          
        
      
    
    {\displaystyle X_{i}}
  
 is a 
  
    
      
        k
        ×
        1
      
    
    {\displaystyle k\times 1}
  
 vector and 
  
    
      
        β
        
          (
          z
          )
        
      
    
    {\displaystyle \beta \left(z\right)}
  
 is a vector of unspecified smooth functions of 
  
    
      
        z
      
    
    {\displaystyle z}
  
.
</p><p>
  
    
      
        γ
        
          (
          ⋅
          )
        
      
    
    {\displaystyle \gamma \left(\cdot \right)}
  
 may be expressed as
</p>

γ
        
          (
          
            Z
            
              i
            
          
          )
        
        =
        
          
            (
            
              E
              
                [
                
                  
                    W
                    
                      i
                    
                  
                  
                    W
                    
                      i
                    
                    ′
                  
                  
                    |
                  
                  
                    Z
                    
                      i
                    
                  
                
                ]
              
            
            )
          
          
            −
            1
          
        
        E
        
          [
          
            
              W
              
                i
              
            
            
              Y
              
                i
              
            
            
              |
            
            
              Z
              
                i
              
            
          
          ]
        
        .
      
    
    {\displaystyle \gamma \left(Z_{i}\right)=\left(E\left[W_{i}W'_{i}|Z_{i}\right]\right)^{-1}E\left[W_{i}Y_{i}|Z_{i}\right].}

<h2 id="see-also">See also</h2>
<ul><li><a href="/facts/Nonparametric_regression/3MuUSUFV">Nonparametric regression</a></li>
<li><a href="/facts/Effective_degree_of_freedom/fvwKCU86">Effective degree of freedom</a></li></ul>
<h2 id="notes">Notes</h2>

<ul><li>Robinson, P.M. (1988). "Root-<i>n</i> Consistent Semiparametric Regression". <i>Econometrica</i>. 56 (4). The Econometric Society: 931–954. <a href="/facts/Doi_(identifier)/muM9Etpq">doi</a>:<a href="https://doi.org/10.2307%2F1912705">10.2307/1912705</a>. <a href="/facts/JSTOR_(identifier)/YTeVmaJ7">JSTOR</a> <a href="https://www.jstor.org/stable/1912705">1912705</a>.</li>
<li>Li, Qi; Racine, Jeffrey S. (2007). <i>Nonparametric Econometrics: Theory and Practice</i>. Princeton University Press. <a href="/facts/ISBN_(identifier)/15AdSPa9">ISBN</a> 978-0-691-12161-1.</li>
<li>Racine, J.S.; Qui, L. (2007). "A Partially Linear Kernel Estimator for Categorical Data". <i>Unpublished Manuscript, Mcmaster University</i>.</li>
<li>Ichimura, H. (1993). <a href="http://purl.umn.edu/55563">"Semiparametric Least Squares (SLS) and Weighted SLS Estimation of Single Index Models"</a>. <i>Journal of Econometrics</i>. 58 (1–2): 71–120. <a href="/facts/Doi_(identifier)/muM9Etpq">doi</a>:<a href="https://doi.org/10.1016%2F0304-4076%2893%2990114-K">10.1016/0304-4076(93)90114-K</a>.</li>
<li>Klein, R. W.; R. H. Spady (1993). "An Efficient Semiparametric Estimator for Binary Response Models". <i>Econometrica</i>. 61 (2). The Econometric Society: 387–421. <a href="/facts/CiteSeerX_(identifier)/SceDmd3c">CiteSeerX</a> <a href="https://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.318.4925">10.1.1.318.4925</a>. <a href="/facts/Doi_(identifier)/muM9Etpq">doi</a>:<a href="https://doi.org/10.2307%2F2951556">10.2307/2951556</a>. <a href="/facts/JSTOR_(identifier)/YTeVmaJ7">JSTOR</a> <a href="https://www.jstor.org/stable/2951556">2951556</a>.</li>
<li>Hastie, T.; R. Tibshirani (1993). "Varying-Coefficient Models". <i>Journal of the Royal Statistical Society, Series B</i>. 55: 757–796.</li></ul>

<h2 id="references">References</h2>

<ol>
<li id="fn:1"><p>See Li and Racine (2007) for an in-depth look at nonparametric regression methods. <a href="#fnref:1" class="footnote-back-ref">↩</a></p></li>
</ol>

Semiparametric regression open-in-new

Semiparametric regression