Menu
Home
People
Places
Arts
History
Plants & Animals
Science
Life & Culture
Technology
Reference.org
Ordinal regression
open-in-new
<h2 id="linear-models-for-ordinal-regression">Linear models for ordinal regression</h2> <p>Ordinal regression can be performed using a <a href="/facts/Generalized_linear_model/Sf1htm5W">generalized linear model</a> (GLM) that fits both a coefficient vector and a set of <i>thresholds</i> to a dataset. Suppose one has a set of observations, represented by length-p vectors x1 through x<i>n</i>, with associated <a href="/facts/Response_variable/dINfzIF2">responses</a> <i>y</i>1 through yn, where each yi is an <a href="/facts/Ordinal_variable/kgLhAaCn">ordinal variable</a> on a scale 1, ..., <i>K</i>. For simplicity, and without loss of generality, we assume y is a non-decreasing vector, that is, yi ≤ {\displaystyle \leq } yi+1. To this data, one fits a length-p coefficient vector w and a set of thresholds <i>θ</i>1, ..., <i>θ</i><i>K</i>−1 with the property that <i>θ</i>1 < <i>θ2</i> < ... < <i>θ</i><i>K</i>−1. This set of thresholds divides the real number line into K disjoint segments, corresponding to the K response levels. </p><p>The model can now be formulated as </p> Pr ( y ≤ i ∣ x ) = σ ( θ i − w ⋅ x ) {\displaystyle \Pr(y\leq i\mid \mathbf {x} )=\sigma (\theta _{i}-\mathbf {w} \cdot \mathbf {x} )} <p>or, the cumulative probability of the response y being at most i is given by a function σ (the inverse <a href="/facts/Link_function/Sf1htm5W">link function</a>) applied to a linear function of x. Several choices exist for σ; the <a href="/facts/Logistic_function/IaQ254t7">logistic function</a> </p> σ ( θ i − w ⋅ x ) = 1 1 + e − ( θ i − w ⋅ x ) {\displaystyle \sigma (\theta _{i}-\mathbf {w} \cdot \mathbf {x} )={\frac {1}{1+e^{-(\theta _{i}-\mathbf {w} \cdot \mathbf {x} )}}}} <p>gives the <a href="/facts/Ordered_logit/tVm2X3Qd">ordered logit</a> model, while using the <a href="/facts/Cumulative_distribution_function/WaKU8tp4">CDF</a> of the standard <a href="/facts/Normal_distribution/UapjjPyQ">normal distribution</a> gives the <a href="/facts/Ordered_probit/mwTHy4U2">ordered probit</a> model. A third option is to use an exponential function </p> σ ( θ i − w ⋅ x ) = 1 − exp ( − exp ( θ i − w ⋅ x ) ) {\displaystyle \sigma (\theta _{i}-\mathbf {w} \cdot \mathbf {x} )=1-\exp(-\exp(\theta _{i}-\mathbf {w} \cdot \mathbf {x} ))} <p>which gives the <a href="/facts/Proportional_hazards_model/6KAwM7x5">proportional hazards model</a>.<a class="footnote-ref" id="fnref:5" href="#fn:5"><sup>5</sup></a> </p> <h3>Latent variable model</h3> <p>The probit version of the above model can be justified by assuming the existence of a real-valued <a href="/facts/Latent_variable/ohg99BnW">latent variable</a> (unobserved quantity) y*, determined by<a class="footnote-ref" id="fnref:6" href="#fn:6"><sup>6</sup></a> </p> y ∗ = w ⋅ x + ε {\displaystyle y^{*}=\mathbf {w} \cdot \mathbf {x} +\varepsilon } <p>where ε is <a href="/facts/Normal_distribution/UapjjPyQ">normally distributed</a> with zero mean and unit variance, <a href="/facts/Conditional_distribution/0eGm3P9W">conditioned</a> on x. The response variable y results from an "incomplete measurement" of y*, where one only determines the interval into which y* falls: </p> y = { 1 if y ∗ ≤ θ 1 , 2 if θ 1 < y ∗ ≤ θ 2 , 3 if θ 2 < y ∗ ≤ θ 3 ⋮ K if θ K − 1 < y ∗ . {\displaystyle y={\begin{cases}1&{\text{if}}~~y^{*}\leq \theta _{1},\\2&{\text{if}}~~\theta _{1}<y^{*}\leq \theta _{2},\\3&{\text{if}}~~\theta _{2}<y^{*}\leq \theta _{3}\\\vdots \\K&{\text{if}}~~\theta _{K-1}<y^{*}.\end{cases}}} <p>Defining <i>θ</i>0 = -∞ and <i>θK</i> = ∞, the above can be summarized as <i>y</i> = <i>k</i> <a href="/facts/If_and_only_if/bYSxGJ66">if and only if</a> <i>θ</i><i>k</i>−1 < <i>y</i>* ≤ <i>θ</i><i>k</i>. </p><p>From these assumptions, one can derive the conditional distribution of y as<a class="footnote-ref" id="fnref:7" href="#fn:7"><sup>7</sup></a> </p> P ( y = k ∣ x ) = P ( θ k − 1 < y ∗ ≤ θ k ∣ x ) = P ( θ k − 1 < w ⋅ x + ε ≤ θ k ) = Φ ( θ k − w ⋅ x ) − Φ ( θ k − 1 − w ⋅ x ) {\displaystyle {\begin{aligned}P(y=k\mid \mathbf {x} )&=P(\theta _{k-1}<y^{*}\leq \theta _{k}\mid \mathbf {x} )\\&=P(\theta _{k-1}<\mathbf {w} \cdot \mathbf {x} +\varepsilon \leq \theta _{k})\\&=\Phi (\theta _{k}-\mathbf {w} \cdot \mathbf {x} )-\Phi (\theta _{k-1}-\mathbf {w} \cdot \mathbf {x} )\end{aligned}}} <p>where Φ is the <a href="/facts/Cumulative_distribution_function/WaKU8tp4">cumulative distribution function</a> of the standard normal distribution, and takes on the role of the inverse link function σ. The <a href="/facts/Log-likelihood/1L8P4NBX">log-likelihood</a> of the model for a single training example x<i>i</i>, yi can now be stated as<a class="footnote-ref" id="fnref:8" href="#fn:8"><sup>8</sup></a> </p> log L ( w , θ ∣ x i , y i ) = ∑ k = 1 K [ y i = k ] log [ Φ ( θ k − w ⋅ x i ) − Φ ( θ k − 1 − w ⋅ x i ) ] {\displaystyle \log {\mathcal {L}}(\mathbf {w} ,\mathbf {\theta } \mid \mathbf {x} _{i},y_{i})=\sum _{k=1}^{K}[y_{i}=k]\log[\Phi (\theta _{k}-\mathbf {w} \cdot \mathbf {x} _{i})-\Phi (\theta _{k-1}-\mathbf {w} \cdot \mathbf {x} _{i})]} <p>(using the <a href="/facts/Iverson_bracket/HUuSwKe8">Iverson bracket</a> [<i>yi</i> = <i>k</i>].) The log-likelihood of the ordered logit model is analogous, using the logistic function instead of Φ.<a class="footnote-ref" id="fnref:9" href="#fn:9"><sup>9</sup></a> </p> <h2 id="alternative-models">Alternative models</h2> <p>In machine learning, alternatives to the latent-variable models of ordinal regression have been proposed. An early result was PRank, a variant of the <a href="/facts/Perceptron/ArxdkAC1">perceptron</a> algorithm that found multiple parallel hyperplanes separating the various ranks; its output is a weight vector w and a sorted vector of <i>K</i>−1 thresholds θ, as in the ordered logit/probit models. The prediction rule for this model is to output the smallest rank k such that wx < <i>θk</i>.<a class="footnote-ref" id="fnref:10" href="#fn:10"><sup>10</sup></a> </p><p>Other methods rely on the principle of large-margin learning that also underlies <a href="/facts/Support_vector_machine/XobxpdBG">support vector machines</a>.<a class="footnote-ref" id="fnref:11" href="#fn:11"><sup>11</sup></a><a class="footnote-ref" id="fnref:12" href="#fn:12"><sup>12</sup></a> </p><p>Another approach is given by Rennie and Srebro, who, realizing that "even just evaluating the likelihood of a predictor is not straight-forward" in the ordered logit and ordered probit models, propose fitting ordinal regression models by adapting common <a href="/facts/Loss_function/xv5ozuhl">loss functions</a> from classification (such as the <a href="/facts/Hinge_loss/k1rbcX7o">hinge loss</a> and <a href="/facts/Log_loss/IG1WD4jb">log loss</a>) to the ordinal case.<a class="footnote-ref" id="fnref:13" href="#fn:13"><sup>13</sup></a> </p> <h2 id="software">Software</h2> <p>ORCA (Ordinal Regression and Classification Algorithms) is an Octave/MATLAB framework including a wide set of ordinal regression methods.<a class="footnote-ref" id="fnref:14" href="#fn:14"><sup>14</sup></a> </p><p>R packages that provide ordinal regression methods include MASS<a class="footnote-ref" id="fnref:15" href="#fn:15"><sup>15</sup></a> and Ordinal.<a class="footnote-ref" id="fnref:16" href="#fn:16"><sup>16</sup></a> </p> <h2 id="see-also">See also</h2> <ul><li><a href="/facts/Logistic_regression/mGj6mc8y">Logistic regression</a></li> <li><a href="/facts/Continuous_ranked_probability_score/h5nm87vd">Continuous ranked probability score</a></li></ul> <h2 id="notes">Notes</h2> <h2 id="further-reading">Further reading</h2> <ul><li><a href="/facts/Alan_Agresti/CYUz3k3v">Agresti, Alan</a> (2010). <i>Analysis of ordinal categorical data</i>. Hoboken, N.J: Wiley. <a href="/facts/ISBN_(identifier)/15AdSPa9">ISBN</a> 978-0470082898.</li> <li><a href="/facts/William_Greene_(economist)/DmoLcmQG">Greene, William H.</a> (2012). <i>Econometric Analysis</i> (Seventh ed.). Boston: Pearson Education. pp. 824–842. <a href="/facts/ISBN_(identifier)/15AdSPa9">ISBN</a> 978-0-273-75356-8.</li> <li>Hardin, James; <a href="/facts/Joseph_Hilbe/n5IVJI9x">Hilbe, Joseph</a> (2007). <i>Generalized Linear Models and Extensions</i> (2nd ed.). College Station: Stata Press. <a href="/facts/ISBN_(identifier)/15AdSPa9">ISBN</a> 978-1-59718-014-6.{{cite book}}: CS1 maint: publisher location (link)</li></ul> <h2 id="references">References</h2> <ol> <li id="fn:1"><p>Winship, Christopher; Mare, Robert D. (1984). "Regression Models with Ordinal Variables" (PDF). American Sociological Review. 49 (4): 512–525. doi:10.2307/2095465. JSTOR 2095465. <a href="http://scholar.harvard.edu/files/cwinship/files/asr_1984.pdf" target="_blank">http://scholar.harvard.edu/files/cwinship/files/asr_1984.pdf</a> <a href="#fnref:1" class="footnote-back-ref">↩</a></p></li> <li id="fn:2"><p>Gutiérrez, P. A.; Pérez-Ortiz, M.; Sánchez-Monedero, J.; Fernández-Navarro, F.; Hervás-Martínez, C. (January 2016). "Ordinal Regression Methods: Survey and Experimental Study". IEEE Transactions on Knowledge and Data Engineering. 28 (1): 127–146. doi:10.1109/TKDE.2015.2457911. hdl:10396/14494. ISSN 1041-4347. <a href="/wiki/Doi_(identifier)" target="_blank">/wiki/Doi_(identifier)</a> <a href="#fnref:2" class="footnote-back-ref">↩</a></p></li> <li id="fn:3"><p>Shashua, Amnon; Levin, Anat (2002). Ranking with large margin principle: Two approaches. NIPS. <a href="/wiki/Neural_Information_Processing_Systems" target="_blank">/wiki/Neural_Information_Processing_Systems</a> <a href="#fnref:3" class="footnote-back-ref">↩</a></p></li> <li id="fn:4"><p>Not to be confused with learning to rank. <a href="/wiki/Learning_to_rank" target="_blank">/wiki/Learning_to_rank</a> <a href="#fnref:4" class="footnote-back-ref">↩</a></p></li> <li id="fn:5"><p>McCullagh, Peter (1980). "Regression models for ordinal data". Journal of the Royal Statistical Society. Series B (Methodological). 42 (2): 109–142. <a href="#fnref:5" class="footnote-back-ref">↩</a></p></li> <li id="fn:6"><p>Wooldridge, Jeffrey M. (2010). Econometric Analysis of Cross Section and Panel Data. MIT Press. pp. 655–657. ISBN 9780262232586. <a href="9780262232586" target="_blank">9780262232586</a> <a href="#fnref:6" class="footnote-back-ref">↩</a></p></li> <li id="fn:7"><p>Wooldridge, Jeffrey M. (2010). Econometric Analysis of Cross Section and Panel Data. MIT Press. pp. 655–657. ISBN 9780262232586. <a href="9780262232586" target="_blank">9780262232586</a> <a href="#fnref:7" class="footnote-back-ref">↩</a></p></li> <li id="fn:8"><p>Wooldridge, Jeffrey M. (2010). Econometric Analysis of Cross Section and Panel Data. MIT Press. pp. 655–657. ISBN 9780262232586. <a href="9780262232586" target="_blank">9780262232586</a> <a href="#fnref:8" class="footnote-back-ref">↩</a></p></li> <li id="fn:9"><p>Agresti, Alan (23 October 2010). "Modeling Ordinal Categorical Data" (PDF). Retrieved 23 July 2015. <a href="http://www.stat.ufl.edu/~aa/ordinal/agresti_ordinal_tutorial.pdf" target="_blank">http://www.stat.ufl.edu/~aa/ordinal/agresti_ordinal_tutorial.pdf</a> <a href="#fnref:9" class="footnote-back-ref">↩</a></p></li> <li id="fn:10"><p>Crammer, Koby; Singer, Yoram (2001). Pranking with ranking. NIPS. <a href="#fnref:10" class="footnote-back-ref">↩</a></p></li> <li id="fn:11"><p>Chu, Wei; Keerthi, S. Sathiya (2007). "Support vector ordinal regression". Neural Computation. 19 (3): 792–815. CiteSeerX 10.1.1.297.3637. doi:10.1162/neco.2007.19.3.792. PMID 17298234. <a href="/wiki/CiteSeerX_(identifier)" target="_blank">/wiki/CiteSeerX_(identifier)</a> <a href="#fnref:11" class="footnote-back-ref">↩</a></p></li> <li id="fn:12"><p>Herbrich, Ralf; Graepel, Thore; Obermayer, Klaus (2000). "Large Margin Rank Boundaries for Ordinal Regression". Advances in Large Margin Classifiers. MIT Press. pp. 115–132. <a href="http://research.microsoft.com/apps/pubs/default.aspx?id=65610" target="_blank">http://research.microsoft.com/apps/pubs/default.aspx?id=65610</a> <a href="#fnref:12" class="footnote-back-ref">↩</a></p></li> <li id="fn:13"><p>Rennie, Jason D. M.; Srebro, Nathan (2005). Loss Functions for Preference Levels: Regression with Discrete Ordered Labels (PDF). Proc. IJCAI Multidisciplinary Workshop on Advances in Preference Handling. <a href="http://ttic.uchicago.edu/~nati/Publications/RennieSrebroIJCAI05.pdf" target="_blank">http://ttic.uchicago.edu/~nati/Publications/RennieSrebroIJCAI05.pdf</a> <a href="#fnref:13" class="footnote-back-ref">↩</a></p></li> <li id="fn:14"><p>orca: Ordinal Regression and Classification Algorithms, AYRNA, 2017-11-21, retrieved 2017-11-21 <a href="https://github.com/ayrna/orca" target="_blank">https://github.com/ayrna/orca</a> <a href="#fnref:14" class="footnote-back-ref">↩</a></p></li> <li id="fn:15"><p>"Modern Applied Statistics with S, 4th ed". www.stats.ox.ac.uk. Retrieved 2020-07-15. <a href="http://www.stats.ox.ac.uk/pub/MASS4/" target="_blank">http://www.stats.ox.ac.uk/pub/MASS4/</a> <a href="#fnref:15" class="footnote-back-ref">↩</a></p></li> <li id="fn:16"><p>Christensen, Rune Haubo B. (2020-06-05), runehaubo/ordinal, retrieved 2020-07-15 <a href="https://github.com/runehaubo/ordinal" target="_blank">https://github.com/runehaubo/ordinal</a> <a href="#fnref:16" class="footnote-back-ref">↩</a></p></li> </ol>