Least-angle regression

<h2 id="pros-and-cons">Pros and cons</h2>
<p>The advantages of the LARS method are:
</p>
<ol><li>It is computationally just as fast as forward selection.</li>
<li>It produces a full piecewise linear solution path, which is useful in <a href="/facts/Cross-validation_(statistics)/pDWg8s0e">cross-validation</a> or similar attempts to tune the model.</li>
<li>If two variables are almost equally correlated with the response, then their coefficients should increase at approximately the same rate. The algorithm thus behaves as intuition would suggest, and also is more stable.</li>
<li>It is easily modified to produce efficient algorithms for other methods producing similar results, like the <a href="/facts/Lasso_(statistics)/R9yE8HDQ">lasso</a> and forward stagewise regression.</li>
<li>It is effective in contexts where <i>p</i> ≫ <i>n</i> (i.e., when the number of predictors <i>p</i> is significantly greater than the number of points <i>n</i>)<a class="footnote-ref" id="fnref:2" href="#fn:2"><sup>2</sup></a></li></ol>
<p>The disadvantages of the LARS method include:
</p>
<ol><li>With any amount of noise in the dependent variable and with high dimensional <a href="/facts/Multicollinearity/bQXzB6sA">multicollinear</a> independent variables, there is no reason to believe that the selected variables will have a high probability of being the actual underlying causal variables.  This problem is not unique to LARS, as it is a general problem with variable selection approaches that seek to find underlying deterministic components. Yet, because LARS is based upon an iterative refitting of the residuals, it appears to be especially sensitive to the effects of noise. This problem is discussed in detail by Weisberg in the discussion section of the Efron et al. (2004) Annals of Statistics article.<a class="footnote-ref" id="fnref:3" href="#fn:3"><sup>3</sup></a> Weisberg provides an empirical example based upon re-analysis of data originally used to validate LARS that the variable selection appears to have problems with highly correlated variables.</li>
<li>Since almost all <a href="/facts/High_dimensional_data/3nCdIa7B">high dimensional data</a> in the real world will just by chance exhibit some degree of collinearity across at least some variables, the problem that LARS has with correlated variables may limit its application to high dimensional data.</li></ol>
<h2 id="algorithm">Algorithm</h2>
<p>The basic steps of the Least-angle regression algorithm are:
</p>
<ul><li>Start with all coefficients 
  
    
      
        β
      
    
    {\displaystyle \beta }
  
 equal to zero.</li>
<li>Find the predictor 
  
    
      
        
          x
          
            j
          
        
      
    
    {\displaystyle x_{j}}
  
 most correlated with 
  
    
      
        y
      
    
    {\displaystyle y}
  
.</li>
<li>Increase the coefficient 
  
    
      
        
          β
          
            j
          
        
      
    
    {\displaystyle \beta _{j}}
  
 in the direction of the sign of its correlation with 
  
    
      
        y
      
    
    {\displaystyle y}
  
. Take residuals 
  
    
      
        r
        =
        y
        −
        
          
            
              y
              ^
            
          
        
      
    
    {\displaystyle r=y-{\hat {y}}}
  
 along the way. Stop when some other predictor 
  
    
      
        
          x
          
            k
          
        
      
    
    {\displaystyle x_{k}}
  
 has as much correlation with 
  
    
      
        r
      
    
    {\displaystyle r}
  
 as 
  
    
      
        
          x
          
            j
          
        
      
    
    {\displaystyle x_{j}}
  
 has.</li>
<li>Increase (
  
    
      
        
          β
          
            j
          
        
      
    
    {\displaystyle \beta _{j}}
  
, 
  
    
      
        
          β
          
            k
          
        
      
    
    {\displaystyle \beta _{k}}
  
) in their joint least squares direction, until some other predictor 
  
    
      
        
          x
          
            m
          
        
      
    
    {\displaystyle x_{m}}
  
 has as much correlation with the residual 
  
    
      
        r
      
    
    {\displaystyle r}
  
.</li>
<li>Increase (
  
    
      
        
          β
          
            j
          
        
      
    
    {\displaystyle \beta _{j}}
  
, 
  
    
      
        
          β
          
            k
          
        
      
    
    {\displaystyle \beta _{k}}
  
, 
  
    
      
        
          β
          
            m
          
        
      
    
    {\displaystyle \beta _{m}}
  
) in their joint least squares direction, until some other predictor 
  
    
      
        
          x
          
            n
          
        
      
    
    {\displaystyle x_{n}}
  
 has as much correlation with the residual 
  
    
      
        r
      
    
    {\displaystyle r}
  
.</li>
<li>Continue until: all predictors are in the model.<a class="footnote-ref" id="fnref:4" href="#fn:4"><sup>4</sup></a></li></ul>
<h2 id="software-implementation">Software implementation</h2>
<p>Least-angle regression is implemented in <a href="/facts/R_(language)/LSrkr8K8"> R</a> via the <a href="https://cran.r-project.org/web/packages/lars/index.html">lars</a> package, in <a href="/facts/Python_(language)/YbuGqofa"> Python</a> with the <a href="http://scikit-learn.org/stable/modules/linear_model.html#least-angle-regression">scikit-learn</a> package, and in <a href="/facts/SAS_(software)/7y2ulIMX"> SAS</a> via the <a href="https://support.sas.com/documentation/cdl/en/statug/63347/HTML/default/viewer.htm#statug_glmselect_a0000000242.htm">GLMSELECT</a> procedure.
</p>
<h2 id="see-also">See also</h2>
<ul><li><a href="/facts/High-dimensional_statistics/3nCdIa7B">High-dimensional statistics</a></li>
<li><a href="/facts/Lasso_(statistics)/R9yE8HDQ">Lasso (statistics)</a></li>
<li><a href="/facts/Regression_analysis/n6z5Tf7K">Regression analysis</a></li>
<li><a href="/facts/Model_selection/R11QK2GC">Model selection</a></li></ul>

<h2 id="references">References</h2>

<ol>
<li id="fn:1"><p>Efron, Bradley; Hastie, Trevor; Johnstone, Iain; Tibshirani, Robert (2004). "Least Angle Regression" (PDF). Annals of Statistics. 32 (2): pp. 407–499. arXiv:math/0406456. doi:10.1214/009053604000000067. MR 2060166. S2CID 204004121. <a href="/wiki/Bradley_Efron" target="_blank">/wiki/Bradley_Efron</a> <a href="#fnref:1" class="footnote-back-ref">↩</a></p></li>
<li id="fn:2"><p>Hastie, Trevor; Robert, Tibshirani; Jerome, Friedman (2009). The Elements of Statistical Learning Data Mining, Inference, and Prediction (2nd ed. 2009.) (PDF). Springer Series in Statistics. Springer New York. p. 76. doi:10.1007/978-0-387-84858-7. ISBN 978-0-387-84857-0. <a href="978-0-387-84857-0" target="_blank">978-0-387-84857-0</a> <a href="#fnref:2" class="footnote-back-ref">↩</a></p></li>
<li id="fn:3"><p>See Discussion by Weisberg following Efron, Bradley; Hastie, Trevor; Johnstone, Iain; Tibshirani, Robert (2004). "Least Angle Regression" (PDF). Annals of Statistics. 32 (2): pp. 407–499. arXiv:math/0406456. doi:10.1214/009053604000000067. MR 2060166. S2CID 204004121. <a href="/wiki/Bradley_Efron" target="_blank">/wiki/Bradley_Efron</a> <a href="#fnref:3" class="footnote-back-ref">↩</a></p></li>
<li id="fn:4"><p>"A simple explanation of the Lasso and Least Angle Regression". Archived from the original on 2015-06-21. <a href="https://web.archive.org/web/20150621032059/http://statweb.stanford.edu/~tibs/lasso/simple.html" target="_blank">https://web.archive.org/web/20150621032059/http://statweb.stanford.edu/~tibs/lasso/simple.html</a> <a href="#fnref:4" class="footnote-back-ref">↩</a></p></li>
</ol>

Least-angle regression open-in-new

Least-angle regression