Proximal gradient methods for learning

Proximal gradient (forward backward splitting) methods for learning is an area of research in <a href="/facts/Optimization/oRn8Iv5I">optimization</a> and <a href="/facts/Statistical_learning_theory/eK1V4mxN">statistical learning theory</a> which studies algorithms for a general class of <a href="/facts/Convex_function/IbzG5SLF">convex</a> <a href="/facts/Regularization_(mathematics)/K601A3y2">regularization</a> problems where the regularization penalty may not be <a href="/facts/Differentiable_function/jQqTgLk1">differentiable</a>. One such example is 
 
 
 
 
 ℓ
 
 1
 
 
 
 
 {\displaystyle \ell _{1}}
 
 regularization (also known as Lasso) of the form

min
          
            w
            ∈
            
              
                R
              
              
                d
              
            
          
        
        
          
            1
            n
          
        
        
          ∑
          
            i
            =
            1
          
          
            n
          
        
        (
        
          y
          
            i
          
        
        −
        ⟨
        w
        ,
        
          x
          
            i
          
        
        ⟩
        
          )
          
            2
          
        
        +
        λ
        ‖
        w
        
          ‖
          
            1
          
        
        ,
        
        
           where 
        
        
          x
          
            i
          
        
        ∈
        
          
            R
          
          
            d
          
        
        
           and 
        
        
          y
          
            i
          
        
        ∈
        
          R
        
        .
      
    
    {\displaystyle \min _{w\in \mathbb {R} ^{d}}{\frac {1}{n}}\sum _{i=1}^{n}(y_{i}-\langle w,x_{i}\rangle )^{2}+\lambda \|w\|_{1},\quad {\text{ where }}x_{i}\in \mathbb {R} ^{d}{\text{ and }}y_{i}\in \mathbb {R} .}

Proximal gradient methods offer a general framework for solving regularization problems from statistical learning theory with penalties that are tailored to a specific problem application. Such customized penalties can help to induce certain structure in problem solutions, such as sparsity (in the case of <a href="/facts/Lasso_(statistics)/R9yE8HDQ">lasso</a>) or group structure (in the case of <a href="/facts/Lasso_(statistics)/R9yE8HDQ">group lasso</a>).

Proximal gradient methods for learning open-in-new

Proximal gradient methods for learning