Learning to rank

<h2 id="applications">Applications</h2>
<h3>In information retrieval</h3>

Ranking is a central part of many <a href="/facts/Information_retrieval/MFSmpeNw">information retrieval</a> problems, such as <a href="/facts/Document_retrieval/H0n6LnCQ">document retrieval</a>, <a href="/facts/Collaborative_filtering/NhSFypjN">collaborative filtering</a>, <a href="/facts/Sentiment_analysis/QRdywf6g">sentiment analysis</a>, and <a href="/facts/Online_advertising/Ncsv0GRz">online advertising</a>.
A possible architecture of a machine-learned search engine is shown in the accompanying figure.
Training data consists of queries and documents matching them together with the relevance degree of each match. It may be prepared manually by human assessors (or raters, as <a href="/facts/Google/GT9Sugza">Google</a> calls them), who check results for some queries and determine <a href="/facts/Relevance_(information_retrieval)/0kkQ6B9A">relevance</a> of each result. It is not feasible to check the relevance of all documents, and so typically a technique called pooling is used — only the top few documents, retrieved by some existing ranking models are checked. This technique may introduce selection bias. Alternatively, training data may be derived automatically by analyzing clickthrough logs (i.e. search results which got clicks from users),<a class="footnote-ref" id="fnref:3" href="#fn:3">3</a> query chains,<a class="footnote-ref" id="fnref:4" href="#fn:4">4</a> or such search engines' features as Google's (since-replaced) <a href="/facts/Google_SearchWiki/PgpJSCYw">SearchWiki</a>. Clickthrough logs can be biased by the tendency of users to click on the top search results on the assumption that they are already well-ranked.
Training data is used by a learning algorithm to produce a ranking model which computes the relevance of documents for actual queries.
Typically, users expect a search query to complete in a short time (such as a few hundred milliseconds for web search), which makes it impossible to evaluate a complex ranking model on each document in the corpus, and so a two-phase scheme is used.<a class="footnote-ref" id="fnref:5" href="#fn:5">5</a> First, a small number of potentially relevant documents are identified using simpler retrieval models which permit fast query evaluation, such as the <a href="/facts/Vector_space_model/WC1LqLA5">vector space model</a>, <a href="/facts/Standard_Boolean_model/EjHXxx9y">Boolean model</a>, weighted AND,<a class="footnote-ref" id="fnref:6" href="#fn:6">6</a> or <a href="/facts/Okapi_BM25/YunFGFj3">BM25</a>. This phase is called top-
 
 
 
 k
 
 
 {\displaystyle k}
 
 document retrieval and many heuristics were proposed in the literature to accelerate it, such as using a document's static quality score and tiered indexes.<a class="footnote-ref" id="fnref:7" href="#fn:7">7</a> In the second phase, a more accurate but computationally expensive machine-learned model is used to re-rank these documents.

<h3>In other areas</h3>
Learning to rank algorithms have been applied in areas other than information retrieval:

<ul><li>In <a href="/facts/Machine_translation/DGF3NwuI">machine translation</a> for ranking a set of hypothesized translations;<a class="footnote-ref" id="fnref:8" href="#fn:8">8</a></li>
<li>In <a href="/facts/Computational_biology/GxgNfwcX">computational biology</a> for ranking candidate 3-D structures in protein structure prediction problems;<a class="footnote-ref" id="fnref:9" href="#fn:9">9</a></li>
<li>In <a href="/facts/Recommender_system/HjodW6nS">recommender systems</a> for identifying a ranked list of related news articles to recommend to a user after he or she has read a current news article.<a class="footnote-ref" id="fnref:10" href="#fn:10">10</a></li></ul>
<h2 id="feature-vectors">Feature vectors</h2>
For the convenience of MLR algorithms, query-document pairs are usually represented by numerical vectors, which are called <a href="/facts/Feature_vector/nzAhYxfu">feature vectors</a>. Such an approach is sometimes called bag of features and is analogous to the <a href="/facts/Bag_of_words/TlhIx600">bag of words</a> model and <a href="/facts/Vector_space_model/WC1LqLA5">vector space model</a> used in information retrieval for representation of documents.
Components of such vectors are called <a href="/facts/Feature_(machine_learning)/nzAhYxfu">features</a>, factors or ranking signals. They may be divided into three groups (features from <a href="/facts/Document_retrieval/H0n6LnCQ">document retrieval</a> are shown as examples):

<ul><li>Query-independent or static features — those features, which depend only on the document, but not on the query. For example, <a href="/facts/PageRank/KtJRJFUX">PageRank</a> or document's length. Such features can be precomputed in off-line mode during indexing. They may be used to compute document's static quality score (or static rank), which is often used to speed up search query evaluation.<a class="footnote-ref" id="fnref:11" href="#fn:11">11</a><a class="footnote-ref" id="fnref:12" href="#fn:12">12</a></li>
<li>Query-dependent or dynamic features — those features, which depend both on the contents of the document and the query, such as <a href="/facts/TF-IDF/JDzc5Jnr">TF-IDF</a> score or other non-machine-learned ranking functions.</li>
<li><a href="/facts/Query-level_feature/KQEkIFGG">Query-level features</a> or query features, which depend only on the query. For example, the number of words in a query.</li></ul>
Some examples of features, which were used in the well-known <a href="https://www.microsoft.com/en-us/research/project/letor-learning-rank-information-retrieval/">LETOR</a> dataset:

<ul><li>TF, <a href="/facts/TF-IDF/JDzc5Jnr">TF-IDF</a>, <a href="/facts/Okapi_BM25/YunFGFj3">BM25</a>, and <a href="/facts/Language_modeling/FntSpg0j">language modeling</a> scores of document's <a href="/facts/Information_retrieval/MFSmpeNw">zones</a> (title, body, anchors text, URL) for a given query;</li>
<li>Lengths and <a href="/facts/Inverse_document_frequency/JDzc5Jnr">IDF</a> sums of document's zones;</li>
<li>Document's <a href="/facts/PageRank/KtJRJFUX">PageRank</a>, <a href="/facts/HITS_algorithm/QpTGWgub">HITS</a> ranks and their variants.</li></ul>
Selecting and designing good features is an important area in machine learning, which is called <a href="/facts/Feature_engineering/VkNJFAFb">feature engineering</a>.

<h2 id="evaluation-measures">Evaluation measures</h2>
Main article: <a href="/facts/Evaluation_measures_(information_retrieval)/fl9dhK1Q">Evaluation measures (information_retrieval) § Offline_metrics</a>
There are several measures (metrics) which are commonly used to judge how well an algorithm is doing on training data and to compare the performance of different MLR algorithms. Often a learning-to-rank problem is reformulated as an optimization problem with respect to one of these metrics.
Examples of ranking quality measures:

<ul><li><a href="/facts/Mean_Average_Precision/fl9dhK1Q">Mean average precision</a> (MAP);</li>
<li><a href="/facts/Discounted_cumulative_gain/9uQy4YPn">DCG</a> and <a href="/facts/Normalized_discounted_cumulative_gain/9uQy4YPn">NDCG</a>;</li>
<li><a href="/facts/Precision_(information_retrieval)/Jnmmpl8H">Precision</a>@n, NDCG@n, where "@n" denotes that the metrics are evaluated only on top n documents;</li>
<li><a href="/facts/Mean_reciprocal_rank/YkxnCY8h">Mean reciprocal rank</a>;</li>
<li><a href="/facts/Kendall%2527s_tau/WEheb0LI">Kendall's tau</a>;</li>
<li><a href="/facts/Spearman%2527s_rank_correlation_coefficient/pDDgJPp1">Spearman's rho</a>.</li></ul>
DCG and its normalized variant NDCG are usually preferred in academic research when multiple levels of relevance are used.<a class="footnote-ref" id="fnref:13" href="#fn:13">13</a> Other metrics such as MAP, MRR and precision, are defined only for binary judgments.
Recently, there have been proposed several new evaluation metrics which claim to model user's satisfaction with search results better than the DCG metric:

<ul><li>Expected reciprocal rank (ERR);<a class="footnote-ref" id="fnref:14" href="#fn:14">14</a></li>
<li><a href="/facts/Yandex/RSCatUyn">Yandex</a>'s pfound.<a class="footnote-ref" id="fnref:15" href="#fn:15">15</a></li></ul>
Both of these metrics are based on the assumption that the user is more likely to stop looking at search results after examining a more relevant document, than after a less relevant document.

<h2 id="approaches">Approaches</h2>
Learning to Rank approaches are often categorized using one of three approaches: pointwise (where individual documents are ranked), pairwise (where pairs of documents are ranked into a relative order), and listwise (where an entire list of documents are ordered). 
<a href="/facts/Tie-Yan_Liu/2dFeKtuw">Tie-Yan Liu</a> of Microsoft Research Asia has analyzed existing algorithms for learning to rank problems in his book Learning to Rank for Information Retrieval.<a class="footnote-ref" id="fnref:16" href="#fn:16">16</a> He categorized them into three groups by their input spaces, output spaces, hypothesis spaces (the core function of the model) and <a href="/facts/Loss_function/xv5ozuhl">loss functions</a>: the pointwise, pairwise, and listwise approach. In practice, listwise approaches often outperform pairwise approaches and pointwise approaches. This statement was further supported by a large scale experiment on the performance of different learning-to-rank methods on a large collection of benchmark data sets.<a class="footnote-ref" id="fnref:17" href="#fn:17">17</a>
In this section, without further notice, 
 
 
 
 x
 
 
 {\displaystyle x}
 
 denotes an object to be evaluated, for example, a document or an image, 
 
 
 
 f
 (
 x
 )
 
 
 {\displaystyle f(x)}
 
 denotes a single-value hypothesis, 
 
 
 
 h
 (
 ⋅
 )
 
 
 {\displaystyle h(\cdot )}
 
 denotes a bi-variate or multi-variate function and 
 
 
 
 L
 (
 ⋅
 )
 
 
 {\displaystyle L(\cdot )}
 
 denotes the loss function.

<h3>Pointwise approach</h3>
In this case, it is assumed that each query-document pair in the training data has a numerical or ordinal score. Then the learning-to-rank problem can be approximated by a regression problem — given a single query-document pair, predict its score. Formally speaking, the pointwise approach aims at learning a function 
 
 
 
 f
 (
 x
 )
 
 
 {\displaystyle f(x)}
 
 predicting the real-value or ordinal score of a document 
 
 
 
 x
 
 
 {\displaystyle x}
 
 using the loss function 
 
 
 
 L
 (
 f
 ;
 
 x
 
 j
 
 
 ,
 
 y
 
 j
 
 
 )
 
 
 {\displaystyle L(f;x_{j},y_{j})}
 
.
A number of existing <a href="/facts/Supervised_learning/JlNFNPFt">supervised</a> machine learning algorithms can be readily used for this purpose. <a href="/facts/Ordinal_regression/2oWy4EYh">Ordinal regression</a> and <a href="/facts/Classification_(machine_learning)/jXXHRkXR">classification</a> algorithms can also be used in pointwise approach when they are used to predict the score of a single query-document pair, and it takes a small, finite number of values.

<h3>Pairwise approach</h3>
In this case, the learning-to-rank problem is approximated by a classification problem — learning a <a href="/facts/Binary_classifier/MeYYAMwp">binary classifier</a> 
 
 
 
 h
 (
 
 x
 
 u
 
 
 ,
 
 x
 
 v
 
 
 )
 
 
 {\displaystyle h(x_{u},x_{v})}
 
 that can tell which document is better in a given pair of documents. The classifier shall take two documents as its input and the goal is to minimize a loss function 
 
 
 
 L
 (
 h
 ;
 
 x
 
 u
 
 
 ,
 
 x
 
 v
 
 
 ,
 
 y
 
 u
 ,
 v
 
 
 )
 
 
 {\displaystyle L(h;x_{u},x_{v},y_{u,v})}
 
. The loss function typically reflects the number and magnitude of <a href="/facts/Permutation/JSw9ukmv">inversions</a> in the induced ranking.
In many cases, the binary classifier 
 
 
 
 h
 (
 
 x
 
 u
 
 
 ,
 
 x
 
 v
 
 
 )
 
 
 {\displaystyle h(x_{u},x_{v})}
 
 is implemented with a <a href="/facts/Scoring_function/h5nm87vd">scoring function</a> 
 
 
 
 f
 (
 x
 )
 
 
 {\displaystyle f(x)}
 
. As an example, RankNet <a class="footnote-ref" id="fnref:18" href="#fn:18">18</a> adapts a probability model and defines 
 
 
 
 h
 (
 
 x
 
 u
 
 
 ,
 
 x
 
 v
 
 
 )
 
 
 {\displaystyle h(x_{u},x_{v})}
 
 as the estimated probability of the document 
 
 
 
 
 x
 
 u
 
 
 
 
 {\displaystyle x_{u}}
 
 has higher quality than 
 
 
 
 
 x
 
 v
 
 
 
 
 {\displaystyle x_{v}}
 
:

P
          
            u
            ,
            v
          
        
        (
        f
        )
        =
        
          CDF
        
        (
        f
        (
        
          x
          
            u
          
        
        )
        −
        f
        (
        
          x
          
            v
          
        
        )
        )
        ,
      
    
    {\displaystyle P_{u,v}(f)={\text{CDF}}(f(x_{u})-f(x_{v})),}

where 
 
 
 
 
 CDF
 
 (
 ⋅
 )
 
 
 {\displaystyle {\text{CDF}}(\cdot )}
 
 is a <a href="/facts/Cumulative_distribution_function/WaKU8tp4">cumulative distribution function</a>, for example, the <a href="/facts/Logistic_distribution/QlQNt0uC">standard logistic CDF</a>, i.e.

CDF
        
        (
        x
        )
        =
        
          
            1
            
              1
              +
              exp
              ⁡
              
                [
                
                  −
                  x
                
                ]
              
            
          
        
        .
      
    
    {\displaystyle {\text{CDF}}(x)={\frac {1}{1+\exp \left[-x\right]}}.}

<h3>Listwise approach</h3>
These algorithms try to directly optimize the value of one of the above evaluation measures, averaged over all queries in the training data. This is often difficult in practice because most evaluation measures are not continuous functions with respect to ranking model's parameters, and so continuous approximations or bounds on evaluation measures have to be used. For example the SoftRank algorithm.<a class="footnote-ref" id="fnref:19" href="#fn:19">19</a> LambdaMART is a pairwise algorithm which has been empirically shown to approximate listwise objective functions.<a class="footnote-ref" id="fnref:20" href="#fn:20">20</a>

<h3>List of methods</h3>
A partial list of published learning-to-rank algorithms is shown below with years of first publication of each method:

<table><tbody><tr><th>Year</th><th>Name</th><th>Type</th><th>Notes</th></tr><tr><td>1989</td><td>OPRF<a class="footnote-ref" id="fnref:21" href="#fn:21">21</a></td><td>2pointwise</td><td>Polynomial regression (instead of machine learning, this work refers to pattern recognition, but the idea is the same).</td></tr><tr><td>1992</td><td>SLR<a class="footnote-ref" id="fnref:22" href="#fn:22">22</a></td><td>2pointwise</td><td>Staged logistic regression.</td></tr><tr><td>1994</td><td>NMOpt<a class="footnote-ref" id="fnref:23" href="#fn:23">23</a></td><td>2listwise</td><td>Non-Metric Optimization.</td></tr><tr><td>1999</td><td><a href="https://www.jstor.org/stable/2699986">MART</a> (Multiple Additive Regression Trees)<a class="footnote-ref" id="fnref:24" href="#fn:24">24</a></td><td>2pairwise</td><td></td></tr><tr><td>2000</td><td><a href="http://web.archive.org/web/20140926033154/http://research.microsoft.com/apps/pubs/default.aspx?id=65610">Ranking SVM</a> (RankSVM)</td><td>2pairwise</td><td>A more recent exposition is in,<a class="footnote-ref" id="fnref:25" href="#fn:25">25</a> which describes an application to ranking using clickthrough logs.</td></tr><tr><td>2001</td><td><a href="https://proceedings.neurips.cc/paper_files/paper/2001/file/5531a5834816222280f20d1ef9e95f69-Paper.pdf">Pranking</a></td><td>1pointwise</td><td>Ordinal regression.</td></tr><tr><td>2003</td><td><a href="http://jmlr.csail.mit.edu/papers/volume4/freund03a/freund03a.pdf">RankBoost</a></td><td>2pairwise</td><td></td></tr><tr><td>2005</td><td><a href="https://www.microsoft.com/en-us/research/wp-content/uploads/2005/08/icml_ranking.pdf">RankNet</a></td><td>2pairwise</td><td></td></tr><tr><td>2006</td><td><a href="https://dl.acm.org/doi/pdf/10.1145/1148170.1148205">IR-SVM</a><a class="footnote-ref" id="fnref:26" href="#fn:26">26</a></td><td>2pairwise</td><td>Ranking SVM with query-level normalization in the loss function.</td></tr><tr><td>2006</td><td><a href="https://proceedings.neurips.cc/paper_files/paper/2006/file/af44c4c56f385c43f2529f9b1b018f6a-Paper.pdf">LambdaRank</a></td><td>pairwise/listwise</td><td>RankNet in which pairwise loss function is multiplied by the change in the IR metric caused by a swap.</td></tr><tr><td>2007</td><td><a href="https://dl.acm.org/doi/pdf/10.1145/1277741.1277809">AdaRank</a><a class="footnote-ref" id="fnref:27" href="#fn:27">27</a></td><td>3listwise</td><td></td></tr><tr><td>2007</td><td><a href="http://research.microsoft.com/apps/pubs/default.aspx?id=70364">FRank</a></td><td>2pairwise</td><td>Based on RankNet, uses a different loss function - fidelity loss.</td></tr><tr><td>2007</td><td><a href="http://www.cc.gatech.edu/~zha/papers/fp086-zheng.pdf">GBRank</a></td><td>2pairwise</td><td></td></tr><tr><td>2007</td><td><a href="http://research.microsoft.com/apps/pubs/default.aspx?id=70428">ListNet</a></td><td>3listwise</td><td></td></tr><tr><td>2007</td><td><a href="http://research.microsoft.com/apps/pubs/default.aspx?id=68128">McRank</a></td><td>1pointwise</td><td></td></tr><tr><td>2007</td><td><a href="https://web.archive.org/web/20100807162456/http://www.stat.rutgers.edu/~tzhang/papers/nips07-ranking.pdf">QBRank</a></td><td>2pairwise</td><td></td></tr><tr><td>2007</td><td><a href="http://web.archive.org/web/20110605102007/http://research.microsoft.com/en-us/people/hangli/qin_ipm_2008.pdf">RankCosine</a><a class="footnote-ref" id="fnref:28" href="#fn:28">28</a></td><td>3listwise</td><td></td></tr><tr><td>2007</td><td><a href="https://citeseerx.ist.psu.edu/document?repid=rep1&type=pdf&doi=f83659a19e6373231a7609d13accf27555fd6ed8">RankGP</a><a class="footnote-ref" id="fnref:29" href="#fn:29">29</a></td><td>3listwise</td><td></td></tr><tr><td>2007</td><td><a href="http://staff.cs.utu.fi/~aatapa/publications/inpPaTsAiBoSa07a.pdf">RankRLS</a></td><td>2pairwise</td><td>Regularized least-squares based ranking. The work is extended in<a class="footnote-ref" id="fnref:30" href="#fn:30">30</a> to learning to rank from general preference graphs.</td></tr><tr><td>2007</td><td><a href="http://www.cs.cornell.edu/People/tj/publications/yue_etal_07a.pdf">SVMmap</a></td><td>3listwise</td><td></td></tr><tr><td>2008</td><td><a href="https://web.archive.org/web/20150610212615/http://research.microsoft.com/pubs/69536/tr-2008-109.pdf">LambdaSMART/LambdaMART</a></td><td>pairwise/listwise</td><td>Winning entry in the Yahoo Learning to Rank competition in 2010, using an ensemble of LambdaMART models. Based on MART (1999)<a class="footnote-ref" id="fnref:31" href="#fn:31">31</a> “LambdaSMART”, for Lambda-submodel-MART, or LambdaMART for the case with no submodel.</td></tr><tr><td>2008</td><td><a href="https://dl.acm.org/doi/pdf/10.1145/1390156.1390306">ListMLE</a><a class="footnote-ref" id="fnref:32" href="#fn:32">32</a></td><td>3listwise</td><td>Based on ListNet.</td></tr><tr><td>2008</td><td><a href="https://dl.acm.org/doi/pdf/10.1145/1390334.1390355">PermuRank</a><a class="footnote-ref" id="fnref:33" href="#fn:33">33</a></td><td>3listwise</td><td></td></tr><tr><td>2008</td><td><a href="https://dl.acm.org/doi/pdf/10.1145/1341531.1341544">SoftRank</a><a class="footnote-ref" id="fnref:34" href="#fn:34">34</a></td><td>3listwise</td><td></td></tr><tr><td>2008</td><td><a href="https://dl.acm.org/doi/pdf/10.1145/1367497.1367552">Ranking Refinement</a><a class="footnote-ref" id="fnref:35" href="#fn:35">35</a></td><td>2pairwise</td><td>A semi-supervised approach to learning to rank that uses Boosting.</td></tr><tr><td>2008</td><td><a href="https://web.archive.org/web/20100723152841/http://www-connex.lip6.fr/~amini/SSRankBoost/">SSRankBoost</a><a class="footnote-ref" id="fnref:36" href="#fn:36">36</a></td><td>2pairwise</td><td>An extension of RankBoost to learn with partially labeled data (semi-supervised learning to rank).</td></tr><tr><td>2008</td><td>SortNet<a class="footnote-ref" id="fnref:37" href="#fn:37">37</a></td><td>2pairwise</td><td>SortNet, an adaptive ranking algorithm which orders objects using a neural network as a comparator.</td></tr><tr><td>2009</td><td><a href="https://dl.acm.org/doi/pdf/10.1145/1645953.1646057">MPBoost</a><a class="footnote-ref" id="fnref:38" href="#fn:38">38</a></td><td>2pairwise</td><td>Magnitude-preserving variant of RankBoost. The idea is that the more unequal are labels of a pair of documents, the harder should the algorithm try to rank them.</td></tr><tr><td>2009</td><td><a href="https://web.archive.org/web/20130620070239/http://machinelearning.org/archive/icml2009/papers/498.pdf">BoltzRank</a></td><td>3listwise</td><td>Unlike earlier methods, BoltzRank produces a ranking model that looks during query time not just at a single document, but also at pairs of documents.</td></tr><tr><td>2009</td><td><a href="http://www.iis.sinica.edu.tw/papers/whm/8820-F.pdf">BayesRank</a></td><td>3listwise</td><td>A method combines <a href="/facts/Plackett%25E2%2580%2593Luce_model/0qdoXn49">Plackett-Luce model</a> and neural network to minimize the expected Bayes risk, related to NDCG, from the decision-making aspect.</td></tr><tr><td>2010</td><td><a href="http://web.archive.org/web/20150324205505/https://people.cs.pitt.edu/~valizadegan/Publications/NDCG_Boost.pdf">NDCG Boost</a><a class="footnote-ref" id="fnref:39" href="#fn:39">39</a></td><td>3listwise</td><td>A boosting approach to optimize NDCG.</td></tr><tr><td>2010</td><td><a href="https://arxiv.org/abs/1001.4597">GBlend</a></td><td>2pairwise</td><td>Extends GBRank to the learning-to-blend problem of jointly solving multiple learning-to-rank problems with some shared features.</td></tr><tr><td>2010</td><td><a href="https://web.archive.org/web/20100601205607/http://wume.cse.lehigh.edu/~ovd209/wsdm/proceedings/docs/p151.pdf">IntervalRank</a></td><td>2pairwise & listwise</td><td></td></tr><tr><td>2010</td><td><a href="http://web.archive.org/web/20100922104716/http://www.eecs.tufts.edu/~dsculley/papers/combined-ranking-and-regression.pdf">CRR</a><a class="footnote-ref" id="fnref:40" href="#fn:40">40</a></td><td>2pointwise & pairwise</td><td>Combined Regression and Ranking. Uses <a href="/facts/Stochastic_gradient_descent/HbcaYqQP">stochastic gradient descent</a> to optimize a linear combination of a pointwise quadratic loss and a pairwise hinge loss from Ranking SVM.</td></tr><tr><td>2014</td><td><a href="http://web.archive.org/web/20190312003749/https://storage.googleapis.com/pub-tools-public-publication-data/pdf/42242.pdf">LCR</a><a class="footnote-ref" id="fnref:41" href="#fn:41">41</a></td><td>2pairwise</td><td>Applied local low-rank assumption on collaborative ranking. Received best student paper award at WWW'14.</td></tr><tr><td>2015</td><td><a href="https://www.cv-foundation.org/openaccess/content_cvpr_2015/papers/Schroff_FaceNet_A_Unified_2015_CVPR_paper.pdf">FaceNet</a></td><td>pairwise</td><td>Ranks face images with the triplet metric via deep convolutional network.</td></tr><tr><td>2016</td><td><a href="https://arxiv.org/abs/1603.02754">XGBoost</a></td><td>pairwise</td><td>Supports various ranking objectives and evaluation metrics.</td></tr><tr><td>2017</td><td><a href="http://web.archive.org/web/20180131023750/http://eprints.nottingham.ac.uk/41540/1/dls_sac2017.pdf">ES-Rank</a><a class="footnote-ref" id="fnref:42" href="#fn:42">42</a></td><td>listwise</td><td>Evolutionary Strategy Learning to Rank technique with 7 fitness evaluation metrics.</td></tr><tr><td>2018</td><td><a href="https://dl.acm.org/doi/abs/10.1145/3209978.3209985">DLCM</a><a class="footnote-ref" id="fnref:43" href="#fn:43">43</a></td><td>2listwise</td><td>A multi-variate ranking function that encodes multiple items from an initial ranked list (local context) with a recurrent neural network and create result ranking accordingly.</td></tr><tr><td>2018</td><td><a href="http://www.jmlr.org/papers/volume19/17-179/17-179.pdf">PolyRank</a><a class="footnote-ref" id="fnref:44" href="#fn:44">44</a></td><td>pairwise</td><td>Learns simultaneously the ranking and the underlying <a href="/facts/Generative_model/JUgExNIP">generative model</a> from pairwise comparisons.</td></tr><tr><td>2018</td><td><a href="https://arxiv.org/abs/1803.05796">FATE-Net/FETA-Net</a><a class="footnote-ref" id="fnref:45" href="#fn:45">45</a></td><td>listwise</td><td>End-to-end trainable architectures, which explicitly take all items into account to model context effects.</td></tr><tr><td>2019</td><td><a href="https://web.archive.org/web/20190514175024/http://cs-people.bu.edu/fcakir/papers/fastap_cvpr2019.pdf">FastAP</a><a class="footnote-ref" id="fnref:46" href="#fn:46">46</a></td><td>listwise</td><td>Optimizes Average Precision to learn deep embeddings.</td></tr><tr><td>2019</td><td><a href="https://arxiv.org/abs/1905.06452">Mulberry</a></td><td>listwise & hybrid</td><td>Learns ranking policies maximizing multiple metrics across the entire dataset.</td></tr><tr><td>2019</td><td><a href="https://ecmlpkdd2019.org/downloads/paper/400.pdf">DirectRanker</a></td><td>pairwise</td><td>Generalisation of the RankNet architecture.</td></tr><tr><td>2019</td><td><a href="https://dl.acm.org/doi/abs/10.1145/3341981.3344218">GSF</a><a class="footnote-ref" id="fnref:47" href="#fn:47">47</a></td><td>2listwise</td><td>A permutation-invariant multi-variate ranking function that encodes and ranks items with groupwise scoring functions built with deep neural networks.</td></tr><tr><td>2020</td><td><a href="https://openaccess.thecvf.com/content_CVPR_2020/papers/Rolinek_Optimizing_Rank-Based_Metrics_With_Blackbox_Differentiation_CVPR_2020_paper.pdf">RaMBO</a><a class="footnote-ref" id="fnref:48" href="#fn:48">48</a></td><td>listwise</td><td>Optimizes rank-based metrics using blackbox backpropagation.<a class="footnote-ref" id="fnref:49" href="#fn:49">49</a></td></tr><tr><td>2020</td><td><a href="https://arxiv.org/pdf/1904.06813.pdf">PRM</a><a class="footnote-ref" id="fnref:50" href="#fn:50">50</a></td><td>pairwise</td><td>Transformer network encoding both the dependencies among items and the interactions between the user and items.</td></tr><tr><td>2020</td><td><a href="https://dl.acm.org/doi/abs/10.1145/3397271.3401104">SetRank</a><a class="footnote-ref" id="fnref:51" href="#fn:51">51</a></td><td>2listwise</td><td>A permutation-invariant multi-variate ranking function that encodes and ranks items with self-attention networks.</td></tr><tr><td>2021</td><td><a href="https://arxiv.org/pdf/2012.06731v2.pdf">PiRank</a><a class="footnote-ref" id="fnref:52" href="#fn:52">52</a></td><td>listwise</td><td>Differentiable surrogates for ranking able to exactly recover the desired metrics and scales favourably to large list sizes, significantly improving internet-scale benchmarks.</td></tr><tr><td>2022</td><td><a href="https://link.springer.com/article/10.1007/s10115-022-01726-0">SAS-Rank</a></td><td>listwise</td><td>Combining Simulated Annealing with Evolutionary Strategy for implicit and explicit learning to rank from relevance labels.</td></tr><tr><td>2022</td><td><a href="https://www.researchsquare.com/article/rs-2121120/v1">VNS-Rank</a></td><td>listwise</td><td>Variable Neighborhood Search in 2 Novel Methodologies in AI for Learning to Rank.</td></tr><tr><td>2022</td><td><a href="https://www.researchsquare.com/article/rs-2293850/v1">VNA-Rank</a></td><td>listwise</td><td>Combining Simulated Annealing with Variable Neighbourhood Search for Learning to Rank.</td></tr><tr><td>2023</td><td><a href="https://link.springer.com/article/10.1007/s00521-023-08412-4">GVN-Rank</a></td><td>listwise</td><td>Combining Gradient Ascent with Variable Neighbourhood Search for Learning to Rank.</td></tr></tbody></table>
Note: as most supervised learning-to-rank algorithms can be applied to pointwise, pairwise and listwise case, only those methods which are specifically designed with ranking in mind are shown above.

<h2 id="history">History</h2>
<a href="/facts/Norbert_Fuhr/EhmaaLmN">Norbert Fuhr</a> introduced the general idea of MLR in 1992, describing learning approaches in information retrieval as a generalization of parameter estimation;<a class="footnote-ref" id="fnref:53" href="#fn:53">53</a> a specific variant of this approach (using <a href="/facts/Polynomial_regression/5nL7tBoc">polynomial regression</a>) had been published by him three years earlier.<a class="footnote-ref" id="fnref:54" href="#fn:54">54</a> Bill Cooper proposed <a href="/facts/Logistic_regression/mGj6mc8y">logistic regression</a> for the same purpose in 1992 <a class="footnote-ref" id="fnref:55" href="#fn:55">55</a> and used it with his <a href="/facts/University_of_California_at_Berkeley/MXVEjklr">Berkeley</a> research group to train a successful ranking function for <a href="/facts/Text_Retrieval_Conference/XrFgbLQu">TREC</a>. Manning et al.<a class="footnote-ref" id="fnref:56" href="#fn:56">56</a> suggest that these early works achieved limited results in their time due to little available training data and poor machine learning techniques.
Several conferences, such as <a href="/facts/Neural_Information_Processing_Systems/YlhI2Atl">NeurIPS</a>, <a href="/facts/Special_Interest_Group_on_Information_Retrieval/97SRV9Fa">SIGIR</a> and <a href="/facts/International_Conference_on_Machine_Learning/J3LW1VCJ">ICML</a> have had workshops devoted to the learning-to-rank problem since the mid-2000s (decade).

<h3>Practical usage by search engines</h3>
Commercial <a href="/facts/Web_search_engine/SRntsbz2">web search engines</a> began using machine-learned ranking systems since the 2000s (decade). One of the first search engines to start using it was <a href="/facts/AltaVista/X0C5ts5p">AltaVista</a> (later its technology was acquired by <a href="/facts/Overture_Services%2c_Inc./hChTdaB9">Overture</a>, and then <a href="/facts/Yahoo/nKhTaV3M">Yahoo</a>), which launched a <a href="/facts/Gradient_boosting/rm99gPyH">gradient boosting</a>-trained ranking function in April 2003.<a class="footnote-ref" id="fnref:57" href="#fn:57">57</a><a class="footnote-ref" id="fnref:58" href="#fn:58">58</a>
<a href="/facts/Bing_(search_engine)/r2pIK5HT">Bing</a>'s search is said to be powered by <a href="https://www.microsoft.com/en-us/research/wp-content/uploads/2005/08/icml_ranking.pdf">RankNet</a> algorithm,<a class="footnote-ref" id="fnref:59" href="#fn:59">59</a>[when?] which was invented at <a href="/facts/Microsoft_Research/04A5LpMI">Microsoft Research</a> in 2005.
In November 2009 a Russian search engine <a href="/facts/Yandex/RSCatUyn">Yandex</a> announced<a class="footnote-ref" id="fnref:60" href="#fn:60">60</a> that it had significantly increased its search quality due to deployment of a new proprietary <a href="/facts/MatrixNet/k4UEEyuN">MatrixNet</a> algorithm, a variant of <a href="/facts/Gradient_boosting/rm99gPyH">gradient boosting</a> method which uses oblivious decision trees.<a class="footnote-ref" id="fnref:61" href="#fn:61">61</a> Recently they have also sponsored a machine-learned ranking competition "Internet Mathematics 2009"<a class="footnote-ref" id="fnref:62" href="#fn:62">62</a> based on their own search engine's production data. Yahoo has announced a similar competition in 2010.<a class="footnote-ref" id="fnref:63" href="#fn:63">63</a>
As of 2008, <a href="/facts/Google/GT9Sugza">Google</a>'s <a href="/facts/Peter_Norvig/GkCUHH4H">Peter Norvig</a> denied that their search engine exclusively relies on machine-learned ranking.<a class="footnote-ref" id="fnref:64" href="#fn:64">64</a> <a href="/facts/Cuil/ueotufwz">Cuil</a>'s CEO, Tom Costello, suggests that they prefer hand-built models because they can outperform machine-learned models when measured against metrics like click-through rate or time on landing page, which is because machine-learned models "learn what people say they like, not what people actually like".<a class="footnote-ref" id="fnref:65" href="#fn:65">65</a>
In January 2017, the technology was included in the <a href="/facts/Open-source_software/HzTaB8E9">open source</a> search engine <a href="/facts/Apache_Solr/2WrGtlx2">Apache Solr</a>.<a class="footnote-ref" id="fnref:66" href="#fn:66">66</a> It is also available in the open source <a href="/facts/OpenSearch_(software)/ct6N0WGP">OpenSearch</a> and <a href="/facts/Elasticsearch/aF9WuCFy">Elasticsearch</a>.<a class="footnote-ref" id="fnref:67" href="#fn:67">67</a><a class="footnote-ref" id="fnref:68" href="#fn:68">68</a> These implementations make learning to rank widely accessible for enterprise search.

<h2 id="vulnerabilities">Vulnerabilities</h2>
Similar to recognition applications in <a href="/facts/Computer_vision/Tl2Yyk66">computer vision</a>, recent neural network based ranking algorithms are also found to be susceptible to covert <a href="/facts/Generative_adversarial_network/SGNuGrlo">adversarial attacks</a>, both on the candidates and the queries.<a class="footnote-ref" id="fnref:69" href="#fn:69">69</a> With small perturbations imperceptible to human beings, ranking order could be arbitrarily altered. In addition, model-agnostic transferable adversarial examples are found to be possible, which enables black-box adversarial attacks on deep ranking systems without requiring access to their underlying implementations.<a class="footnote-ref" id="fnref:70" href="#fn:70">70</a><a class="footnote-ref" id="fnref:71" href="#fn:71">71</a>
Conversely, the robustness of such ranking systems can be improved via adversarial defenses such as the Madry defense.<a class="footnote-ref" id="fnref:72" href="#fn:72">72</a>

<h2 id="see-also">See also</h2>
<ul><li><a href="/facts/Content-based_image_retrieval/AQpmsoGE">Content-based image retrieval</a></li>
<li><a href="/facts/Multimedia_information_retrieval/aFhj9phr">Multimedia information retrieval</a></li>
<li><a href="/facts/Image_retrieval/pkcq1rTk">Image retrieval</a></li>
<li><a href="/facts/Triplet_loss/FCtHWMYt">Triplet loss</a></li></ul>

<h2 id="external-links">External links</h2>
Competitions and public datasets
<ul><li><a href="https://www.microsoft.com/en-us/research/project/letor-learning-rank-information-retrieval/">LETOR: A Benchmark Collection for Research on Learning to Rank for Information Retrieval</a></li>
<li><a href="https://web.archive.org/web/20150912134134/http://imat2009.yandex.ru/en">Yandex's Internet Mathematics 2009</a></li>
<li><a href="https://web.archive.org/web/20100301011649/http://learningtorankchallenge.yahoo.com/">Yahoo! Learning to Rank Challenge</a></li>
<li><a href="http://research.microsoft.com/en-us/projects/mslr/default.aspx">Microsoft Learning to Rank Datasets</a></li></ul>

<h2 id="references">References</h2>

<ol>
<li id="fn:1">Tie-Yan Liu (2009), "Learning to Rank for Information Retrieval", Foundations and Trends in Information Retrieval, 3 (3): 225–331, doi:10.1561/1500000016, ISBN 978-1-60198-244-5. Slides from Tie-Yan Liu's talk at WWW 2009 conference are available online Archived 2017-08-08 at the Wayback Machine <a href="978-1-60198-244-5" target="_blank">978-1-60198-244-5</a> <a href="#fnref:1" class="footnote-back-ref">↩</a></li>
<li id="fn:2">Mehryar Mohri, Afshin Rostamizadeh, Ameet Talwalkar (2012) Foundations of Machine Learning, The
MIT Press ISBN 9780262018258. <a href="/wiki/Mehryar_Mohri" target="_blank">/wiki/Mehryar_Mohri</a> <a href="#fnref:2" class="footnote-back-ref">↩</a></li>
<li id="fn:3">Joachims, T. (2002), "Optimizing Search Engines using Clickthrough Data" (PDF), Proceedings of the ACM Conference on Knowledge Discovery and Data Mining, archived (PDF) from the original on 2009-12-29, retrieved 2009-11-11 <a href="http://www.cs.cornell.edu/people/tj/publications/joachims_02c.pdf" target="_blank">http://www.cs.cornell.edu/people/tj/publications/joachims_02c.pdf</a> <a href="#fnref:3" class="footnote-back-ref">↩</a></li>
<li id="fn:4">Joachims T.; Radlinski F. (2005), "Query Chains: Learning to Rank from Implicit Feedback" (PDF), Proceedings of the ACM Conference on Knowledge Discovery and Data Mining, arXiv:cs/0605035, Bibcode:2006cs........5035R, archived (PDF) from the original on 2011-07-27, retrieved 2009-12-19 <a href="http://radlinski.org/papers/Radlinski05QueryChains.pdf" target="_blank">http://radlinski.org/papers/Radlinski05QueryChains.pdf</a> <a href="#fnref:4" class="footnote-back-ref">↩</a></li>
<li id="fn:5">B. Cambazoglu; H. Zaragoza; O. Chapelle; J. Chen; C. Liao; Z. Zheng; J. Degenhardt., "Early exit optimizations for additive machine learned ranking systems" (PDF), WSDM '10: Proceedings of the Third ACM International Conference on Web Search and Data Mining, 2010., archived from the original (PDF) on 2019-08-28, retrieved 2009-12-23 <a href="https://web.archive.org/web/20190828063315/http://olivier.chapelle.cc/pub/wsdm2010.pdf" target="_blank">https://web.archive.org/web/20190828063315/http://olivier.chapelle.cc/pub/wsdm2010.pdf</a> <a href="#fnref:5" class="footnote-back-ref">↩</a></li>
<li id="fn:6">Broder A.; Carmel D.; Herscovici M.; Soffer A.; Zien J. (2003), "Efficient query evaluation using a two-level retrieval process", Proceedings of the twelfth international conference on Information and knowledge management (PDF), pp. 426–434, doi:10.1145/956863.956944, ISBN 978-1-58113-723-1, S2CID 2432701, archived from the original (PDF) on 2009-05-21, retrieved 2009-12-15 <a href="978-1-58113-723-1" target="_blank">978-1-58113-723-1</a> <a href="#fnref:6" class="footnote-back-ref">↩</a></li>
<li id="fn:7">Manning C.; Raghavan P.; Schütze H. (2008), Introduction to Information Retrieval, Cambridge University Press. Section 7.1 Archived 2009-07-19 at the Wayback Machine <a href="http://nlp.stanford.edu/IR-book/html/htmledition/efficient-scoring-and-ranking-1.html" target="_blank">http://nlp.stanford.edu/IR-book/html/htmledition/efficient-scoring-and-ranking-1.html</a> <a href="#fnref:7" class="footnote-back-ref">↩</a></li>
<li id="fn:8">Kevin K. Duh (2009), Learning to Rank with Partially-Labeled Data (PDF), archived (PDF) from the original on 2011-07-20, retrieved 2009-12-27 <a href="http://ssli.ee.washington.edu/people/duh/thesis/uwthesis.pdf" target="_blank">http://ssli.ee.washington.edu/people/duh/thesis/uwthesis.pdf</a> <a href="#fnref:8" class="footnote-back-ref">↩</a></li>
<li id="fn:9">Kevin K. Duh (2009), Learning to Rank with Partially-Labeled Data (PDF), archived (PDF) from the original on 2011-07-20, retrieved 2009-12-27 <a href="http://ssli.ee.washington.edu/people/duh/thesis/uwthesis.pdf" target="_blank">http://ssli.ee.washington.edu/people/duh/thesis/uwthesis.pdf</a> <a href="#fnref:9" class="footnote-back-ref">↩</a></li>
<li id="fn:10">Yuanhua Lv, Taesup Moon, Pranam Kolari, Zhaohui Zheng, Xuanhui Wang, and Yi Chang, Learning to Model Relatedness for News Recommendation Archived 2011-08-27 at the Wayback Machine, in International Conference on World Wide Web (WWW), 2011. <a href="http://sifaka.cs.uiuc.edu/~ylv2/pub/www11-relatedness.pdf" target="_blank">http://sifaka.cs.uiuc.edu/~ylv2/pub/www11-relatedness.pdf</a> <a href="#fnref:10" class="footnote-back-ref">↩</a></li>
<li id="fn:11">Manning C.; Raghavan P.; Schütze H. (2008), Introduction to Information Retrieval, Cambridge University Press. Section 7.1 Archived 2009-07-19 at the Wayback Machine <a href="http://nlp.stanford.edu/IR-book/html/htmledition/efficient-scoring-and-ranking-1.html" target="_blank">http://nlp.stanford.edu/IR-book/html/htmledition/efficient-scoring-and-ranking-1.html</a> <a href="#fnref:11" class="footnote-back-ref">↩</a></li>
<li id="fn:12">Richardson, M.; Prakash, A.; Brill, E. (2006). "Beyond PageRank: Machine Learning for Static Ranking" (PDF). Proceedings of the 15th International World Wide Web Conference. pp. 707–715. Archived (PDF) from the original on 2009-08-15. Retrieved 2009-11-18. <a href="http://research.microsoft.com/en-us/um/people/mattri/papers/www2006/staticrank.pdf" target="_blank">http://research.microsoft.com/en-us/um/people/mattri/papers/www2006/staticrank.pdf</a> <a href="#fnref:12" class="footnote-back-ref">↩</a></li>
<li id="fn:13">"Archived copy". Archived from the original on 2011-01-04. Retrieved 2009-12-14.{{cite web}}: CS1 maint: archived copy as title (link) <a href="http://www.stanford.edu/class/cs276/handouts/lecture15-learning-ranking.ppt" target="_blank">http://www.stanford.edu/class/cs276/handouts/lecture15-learning-ranking.ppt</a> <a href="#fnref:13" class="footnote-back-ref">↩</a></li>
<li id="fn:14">Olivier Chapelle; Donald Metzler; Ya Zhang; Pierre Grinspan (2009), "Expected Reciprocal Rank for Graded Relevance" (PDF), CIKM, archived from the original (PDF) on 2012-02-24 <a href="https://web.archive.org/web/20120224053008/http://research.yahoo.com/files/err.pdf" target="_blank">https://web.archive.org/web/20120224053008/http://research.yahoo.com/files/err.pdf</a> <a href="#fnref:14" class="footnote-back-ref">↩</a></li>
<li id="fn:15">Gulin A.; Karpovich P.; Raskovalov D.; Segalovich I. (2009), "Yandex at ROMIP'2009: optimization of ranking algorithms by machine learning methods" (PDF), Proceedings of ROMIP'2009: 163–168, archived (PDF) from the original on 2009-11-22, retrieved 2009-11-13 (in Russian) <a href="http://romip.ru/romip2009/15_yandex.pdf" target="_blank">http://romip.ru/romip2009/15_yandex.pdf</a> <a href="#fnref:15" class="footnote-back-ref">↩</a></li>
<li id="fn:16">Tie-Yan Liu (2009), "Learning to Rank for Information Retrieval", Foundations and Trends in Information Retrieval, 3 (3): 225–331, doi:10.1561/1500000016, ISBN 978-1-60198-244-5. Slides from Tie-Yan Liu's talk at WWW 2009 conference are available online Archived 2017-08-08 at the Wayback Machine <a href="978-1-60198-244-5" target="_blank">978-1-60198-244-5</a> <a href="#fnref:16" class="footnote-back-ref">↩</a></li>
<li id="fn:17">Tax, Niek; Bockting, Sander; Hiemstra, Djoerd (2015), "A cross-benchmark comparison of 87 learning to rank methods" (PDF), Information Processing & Management, 51 (6): 757–772, doi:10.1016/j.ipm.2015.07.002, S2CID 22782599, archived from the original (PDF) on 2017-08-09, retrieved 2017-10-15 <a href="https://web.archive.org/web/20170809115827/http://wwwhome.cs.utwente.nl/~hiemstra/papers/ipm2015.pdf" target="_blank">https://web.archive.org/web/20170809115827/http://wwwhome.cs.utwente.nl/~hiemstra/papers/ipm2015.pdf</a> <a href="#fnref:17" class="footnote-back-ref">↩</a></li>
<li id="fn:18">Burges, Chris J. C.; Shaked, Tal; Renshaw, Erin; Lazier, Ari; Deeds, Matt; Hamilton, Nicole; Hullender, Greg (1 August 2005). "Learning to Rank using Gradient Descent". Archived from the original on 26 February 2021. Retrieved 31 March 2021. {{cite journal}}: Cite journal requires |journal= (help) <a href="https://www.microsoft.com/en-us/research/publication/learning-to-rank-using-gradient-descent" target="_blank">https://www.microsoft.com/en-us/research/publication/learning-to-rank-using-gradient-descent</a> <a href="#fnref:18" class="footnote-back-ref">↩</a></li>
<li id="fn:19">Taylor, M.J., Guiver, J., Robertson, S.E., & Minka, T.P. (2008). SoftRank: optimizing non-smooth rank metrics. Web Search and Data Mining. <a href="#fnref:19" class="footnote-back-ref">↩</a></li>
<li id="fn:20">Burges, Chris J. C. (2010-06-23). "From RankNet to LambdaRank to LambdaMART: An Overview". {{cite journal}}: Cite journal requires |journal= (help) <a href="https://www.microsoft.com/en-us/research/publication/from-ranknet-to-lambdarank-to-lambdamart-an-overview/" target="_blank">https://www.microsoft.com/en-us/research/publication/from-ranknet-to-lambdarank-to-lambdamart-an-overview/</a> <a href="#fnref:20" class="footnote-back-ref">↩</a></li>
<li id="fn:21">Fuhr, Norbert (1989), "Optimum polynomial retrieval functions based on the probability ranking principle", ACM Transactions on Information Systems, 7 (3): 183–204, doi:10.1145/65943.65944, S2CID 16632383 <a href="/wiki/Doi_(identifier)" target="_blank">/wiki/Doi_(identifier)</a> <a href="#fnref:21" class="footnote-back-ref">↩</a></li>
<li id="fn:22">Cooper, William S.; Gey, Frederic C.; Dabney, Daniel P. (1992), "Probabilistic retrieval based on staged logistic regression", Proceedings of the 15th annual international ACM SIGIR conference on Research and development in information retrieval - SIGIR '92, pp. 198–210, doi:10.1145/133160.133199, ISBN 978-0897915236, S2CID 125993 <a href="978-0897915236" target="_blank">978-0897915236</a> <a href="#fnref:22" class="footnote-back-ref">↩</a></li>
<li id="fn:23">Bartell, Brian T.; Cottrell Garrison W.; Belew, Richard K. (1994), "Automatic Combination of Multiple Ranked Retrieval Systems", Sigir '94, pp. 173–181, doi:10.1007/978-1-4471-2099-5_18, ISBN 978-0387198897, S2CID 18606472, archived from the original on 2018-06-13, retrieved 2020-10-12 <a href="978-0387198897" target="_blank">978-0387198897</a> <a href="#fnref:23" class="footnote-back-ref">↩</a></li>
<li id="fn:24">Friedman, Jerome H. (2001). "Greedy Function Approximation: A Gradient Boosting Machine". The Annals of Statistics. 29 (5): 1189–1232. doi:10.1214/aos/1013203451. ISSN 0090-5364. JSTOR 2699986. <a href="https://www.jstor.org/stable/2699986" target="_blank">https://www.jstor.org/stable/2699986</a> <a href="#fnref:24" class="footnote-back-ref">↩</a></li>
<li id="fn:25">Joachims, T. (2002), "Optimizing Search Engines using Clickthrough Data" (PDF), Proceedings of the ACM Conference on Knowledge Discovery and Data Mining, archived (PDF) from the original on 2009-12-29, retrieved 2009-11-11 <a href="http://www.cs.cornell.edu/people/tj/publications/joachims_02c.pdf" target="_blank">http://www.cs.cornell.edu/people/tj/publications/joachims_02c.pdf</a> <a href="#fnref:25" class="footnote-back-ref">↩</a></li>
<li id="fn:26">Cao, Yunbo; Xu, Jun; Liu, Tie-Yan; Li, Hang; Huang, Yalou; Hon, Hsiao-Wuen (2006-08-06). "Adapting ranking SVM to document retrieval". Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval. SIGIR '06. New York, NY, USA: Association for Computing Machinery. pp. 186–193. doi:10.1145/1148170.1148205. ISBN 978-1-59593-369-0. <a href="978-1-59593-369-0" target="_blank">978-1-59593-369-0</a> <a href="#fnref:26" class="footnote-back-ref">↩</a></li>
<li id="fn:27">Xu, Jun; Li, Hang (2007-07-23). "AdaRank: A boosting algorithm for information retrieval". Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval. SIGIR '07. New York, NY, USA: Association for Computing Machinery. pp. 391–398. doi:10.1145/1277741.1277809. ISBN 978-1-59593-597-7. <a href="978-1-59593-597-7" target="_blank">978-1-59593-597-7</a> <a href="#fnref:27" class="footnote-back-ref">↩</a></li>
<li id="fn:28">Qin, Tao; Zhang, Xu-Dong; Tsai, Ming-Feng; Wang, De-Sheng; Liu, Tie-Yan; Li, Hang (2008-03-01). "Query-level loss functions for information retrieval". Information Processing & Management. Evaluating Exploratory Search Systems. 44 (2): 838–855. doi:10.1016/j.ipm.2007.07.016. ISSN 0306-4573. <a href="https://linkinghub.elsevier.com/retrieve/pii/S0306457307001276" target="_blank">https://linkinghub.elsevier.com/retrieve/pii/S0306457307001276</a> <a href="#fnref:28" class="footnote-back-ref">↩</a></li>
<li id="fn:29">Lin, Jung Yi; Yeh, Jen-Yuan; Chao Chung Liu (July 2012). "Learning to rank for information retrieval using layered multi-population genetic programming". 2012 IEEE International Conference on Computational Intelligence and Cybernetics (CyberneticsCom). IEEE. pp. 45–49. doi:10.1109/cyberneticscom.2012.6381614. ISBN 978-1-4673-0892-2. <a href="978-1-4673-0892-2" target="_blank">978-1-4673-0892-2</a> <a href="#fnref:29" class="footnote-back-ref">↩</a></li>
<li id="fn:30">Pahikkala, Tapio; Tsivtsivadze, Evgeni; Airola, Antti; Järvinen, Jouni; Boberg, Jorma (2009), "An efficient algorithm for learning to rank from preference graphs", Machine Learning, 75 (1): 129–165, doi:10.1007/s10994-008-5097-z. <a href="/wiki/Doi_(identifier)" target="_blank">/wiki/Doi_(identifier)</a> <a href="#fnref:30" class="footnote-back-ref">↩</a></li>
<li id="fn:31">C. Burges. (2010). From RankNet to LambdaRank to LambdaMART: An Overview Archived 2017-11-10 at the Wayback Machine. <a href="https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/MSR-TR-2010-82.pdf" target="_blank">https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/MSR-TR-2010-82.pdf</a> <a href="#fnref:31" class="footnote-back-ref">↩</a></li>
<li id="fn:32">Xia, Fen; Liu, Tie-Yan; Wang, Jue; Zhang, Wensheng; Li, Hang (2008-07-05). "Listwise approach to learning to rank: Theory and algorithm". Proceedings of the 25th international conference on Machine learning - ICML '08. New York, NY, USA: Association for Computing Machinery. pp. 1192–1199. doi:10.1145/1390156.1390306. ISBN 978-1-60558-205-4. <a href="978-1-60558-205-4" target="_blank">978-1-60558-205-4</a> <a href="#fnref:32" class="footnote-back-ref">↩</a></li>
<li id="fn:33">Xu, Jun; Liu, Tie-Yan; Lu, Min; Li, Hang; Ma, Wei-Ying (2008-07-20). "Directly optimizing evaluation measures in learning to rank". Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval. SIGIR '08. New York, NY, USA: Association for Computing Machinery. pp. 107–114. doi:10.1145/1390334.1390355. ISBN 978-1-60558-164-4. <a href="978-1-60558-164-4" target="_blank">978-1-60558-164-4</a> <a href="#fnref:33" class="footnote-back-ref">↩</a></li>
<li id="fn:34">Taylor, Michael; Guiver, John; Robertson, Stephen; Minka, Tom (2008-02-11). "SoftRank: Optimizing non-smooth rank metrics". Proceedings of the international conference on Web search and web data mining - WSDM '08. New York, NY, USA: Association for Computing Machinery. pp. 77–86. doi:10.1145/1341531.1341544. ISBN 978-1-59593-927-2. <a href="978-1-59593-927-2" target="_blank">978-1-59593-927-2</a> <a href="#fnref:34" class="footnote-back-ref">↩</a></li>
<li id="fn:35">Rong Jin, Hamed Valizadegan, Hang Li, Ranking Refinement and Its Application for Information Retrieval Archived 2012-04-06 at the Wayback Machine, in International Conference on World Wide Web (WWW), 2008. <a href="http://www.cs.pitt.edu/~valizadegan/Publications/ranking_refinement.pdf" target="_blank">http://www.cs.pitt.edu/~valizadegan/Publications/ranking_refinement.pdf</a> <a href="#fnref:35" class="footnote-back-ref">↩</a></li>
<li id="fn:36">Massih-Reza Amini, Vinh Truong, Cyril Goutte, A Boosting Algorithm for Learning Bipartite Ranking Functions with Partially Labeled Data Archived 2010-08-02 at the Wayback Machine, International ACM SIGIR conference, 2008. The code Archived 2010-07-23 at the Wayback Machine is available for research purposes. <a href="http://www-connex.lip6.fr/~amini/Publis/SemiSupRanking_sigir08.pdf" target="_blank">http://www-connex.lip6.fr/~amini/Publis/SemiSupRanking_sigir08.pdf</a> <a href="#fnref:36" class="footnote-back-ref">↩</a></li>
<li id="fn:37">Leonardo Rigutini, Tiziano Papini, Marco Maggini, Franco Scarselli, "SortNet: learning to rank by a neural-based sorting algorithm" Archived 2011-11-25 at the Wayback Machine, SIGIR 2008 workshop: Learning to Rank for Information Retrieval, 2008 <a href="http://research.microsoft.com/en-us/um/beijing/events/lr4ir-2008/PROCEEDINGS-LR4IR%202008.PDF" target="_blank">http://research.microsoft.com/en-us/um/beijing/events/lr4ir-2008/PROCEEDINGS-LR4IR%202008.PDF</a> <a href="#fnref:37" class="footnote-back-ref">↩</a></li>
<li id="fn:38">Zhu, Chenguang; Chen, Weizhu; Zhu, Zeyuan Allen; Wang, Gang; Wang, Dong; Chen, Zheng (2009-11-02). "A general magnitude-preserving boosting algorithm for search ranking". Proceedings of the 18th ACM conference on Information and knowledge management. CIKM '09. New York, NY, USA: Association for Computing Machinery. pp. 817–826. doi:10.1145/1645953.1646057. ISBN 978-1-60558-512-3. <a href="978-1-60558-512-3" target="_blank">978-1-60558-512-3</a> <a href="#fnref:38" class="footnote-back-ref">↩</a></li>
<li id="fn:39">Hamed Valizadegan, Rong Jin, Ruofei Zhang, Jianchang Mao, Learning to Rank by Optimizing NDCG Measure Archived 2012-04-06 at the Wayback Machine, in Proceeding of Neural Information Processing Systems (NIPS), 2010. <a href="/wiki/Jianchang_Mao" target="_blank">/wiki/Jianchang_Mao</a> <a href="#fnref:39" class="footnote-back-ref">↩</a></li>
<li id="fn:40">Sculley, D. (2010-07-25). "Combined regression and ranking". Proceedings of the 16th ACM SIGKDD international conference on Knowledge discovery and data mining. KDD '10. New York, NY, USA: Association for Computing Machinery. pp. 979–988. doi:10.1145/1835804.1835928. ISBN 978-1-4503-0055-1. <a href="978-1-4503-0055-1" target="_blank">978-1-4503-0055-1</a> <a href="#fnref:40" class="footnote-back-ref">↩</a></li>
<li id="fn:41">Lee, Joonseok; Bengio, Samy; Kim, Seungyeon; Lebanon, Guy; Singer, Yoram (2014-04-07). "Local collaborative ranking". Proceedings of the 23rd international conference on World wide web. WWW '14. New York, NY, USA: Association for Computing Machinery. pp. 85–96. doi:10.1145/2566486.2567970. ISBN 978-1-4503-2744-2. <a href="978-1-4503-2744-2" target="_blank">978-1-4503-2744-2</a> <a href="#fnref:41" class="footnote-back-ref">↩</a></li>
<li id="fn:42">Ibrahim, Osman Ali Sadek; Landa-Silva, Dario (2017-04-03). "ES-Rank: Evolution strategy learning to rank approach". Proceedings of the Symposium on Applied Computing (PDF). SAC '17. New York, NY, USA: Association for Computing Machinery. pp. 944–950. doi:10.1145/3019612.3019696. ISBN 978-1-4503-4486-9. <a href="978-1-4503-4486-9" target="_blank">978-1-4503-4486-9</a> <a href="#fnref:42" class="footnote-back-ref">↩</a></li>
<li id="fn:43">Ai, Qingyao; Bi, Keping; Jiafeng, Guo; Croft, W. Bruce (2018), "Learning a Deep Listwise Context Model for Ranking Refinement", The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, pp. 135–144, arXiv:1804.05936, doi:10.1145/3209978.3209985, ISBN 9781450356572, S2CID 4956076 <a href="9781450356572" target="_blank">9781450356572</a> <a href="#fnref:43" class="footnote-back-ref">↩</a></li>
<li id="fn:44">Davidov, Ori; Ailon, Nir; Oliveira, Ivo F. D. (2018). "A New and Flexible Approach to the Analysis of Paired Comparison Data". Journal of Machine Learning Research. 19 (60): 1–29. ISSN 1533-7928. Archived from the original on 2019-10-03. Retrieved 2019-09-17. <a href="http://jmlr.org/papers/v19/17-179.html" target="_blank">http://jmlr.org/papers/v19/17-179.html</a> <a href="#fnref:44" class="footnote-back-ref">↩</a></li>
<li id="fn:45">Pfannschmidt, Karlson; Gupta, Pritha; Hüllermeier, Eyke (2018). "Deep Architectures for Learning Context-dependent Ranking Functions". arXiv:1803.05796 [stat.ML]. <a href="/wiki/ArXiv_(identifier)" target="_blank">/wiki/ArXiv_(identifier)</a> <a href="#fnref:45" class="footnote-back-ref">↩</a></li>
<li id="fn:46">Fatih Cakir, Kun He, Xide Xia, Brian Kulis, Stan Sclaroff, Deep Metric Learning to Rank Archived 2019-05-14 at the Wayback Machine, In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019. <a href="http://cs-people.bu.edu/fcakir/papers/fastap_cvpr2019.pdf" target="_blank">http://cs-people.bu.edu/fcakir/papers/fastap_cvpr2019.pdf</a> <a href="#fnref:46" class="footnote-back-ref">↩</a></li>
<li id="fn:47">Ai, Qingyao; Wang, Xuanhui; Bruch, Sebastian; Golbandi, Nadav; Bendersky, Michael; Najork, Marc (2019), "Learning Groupwise Multivariate Scoring Functions Using Deep Neural Networks", Proceedings of the 2019 ACM SIGIR International Conference on Theory of Information Retrieval, pp. 85–92, arXiv:1811.04415, doi:10.1145/3341981.3344218, ISBN 9781450368810, S2CID 199441954 <a href="9781450368810" target="_blank">9781450368810</a> <a href="#fnref:47" class="footnote-back-ref">↩</a></li>
<li id="fn:48">Rolínek, Michal; Musil, Vít; Paulus, Anselm; Vlastelica, Marin; Michaelis, Claudio; Martius, Georg (2020-03-18). "Optimizing Rank-Based Metrics with Blackbox Differentiation". 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 7617–7627. arXiv:1912.03500. doi:10.1109/CVPR42600.2020.00764. ISBN 978-1-7281-7168-5. <a href="978-1-7281-7168-5" target="_blank">978-1-7281-7168-5</a> <a href="#fnref:48" class="footnote-back-ref">↩</a></li>
<li id="fn:49">Vlastelica, Marin; Paulus, Anselm; Musil, Vít; Martius, Georg; Rolínek, Michal (2019-12-04). "Differentiation of Blackbox Combinatorial Solvers". arXiv:1912.02175. {{cite journal}}: Cite journal requires |journal= (help) <a href="/wiki/ArXiv_(identifier)" target="_blank">/wiki/ArXiv_(identifier)</a> <a href="#fnref:49" class="footnote-back-ref">↩</a></li>
<li id="fn:50">Liu, Weiwen; Liu, Qing; Tang, Ruiming; Chen, Junyang; He, Xiuqiang; Heng, Pheng Ann (2020-10-19). "Personalized Re-ranking with Item Relationships for E-commerce". Proceedings of the 29th ACM International Conference on Information & Knowledge Management. CIKM '20. Virtual Event, Ireland: Association for Computing Machinery. pp. 925–934. doi:10.1145/3340531.3412332. ISBN 978-1-4503-6859-9. S2CID 224281012. Archived from the original on 2021-10-17. Retrieved 2021-04-26. <a href="978-1-4503-6859-9" target="_blank">978-1-4503-6859-9</a> <a href="#fnref:50" class="footnote-back-ref">↩</a></li>
<li id="fn:51">Pang, Liang; Xu, Jun; Ai, Qingyao; Lan, Yanyan; Cheng, Xueqi; Wen, Jirong (2020), "SetRank", Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 499–508, doi:10.1145/3397271.3401104, ISBN 9781450380164, S2CID 241534531 <a href="9781450380164" target="_blank">9781450380164</a> <a href="#fnref:51" class="footnote-back-ref">↩</a></li>
<li id="fn:52">Swezey, Robin; Grover, Aditya; Charron, Bruno; Ermon, Stefano (2021-11-27). "PiRank: Scalable Learning To Rank via Differentiable Sorting". Advances in Neural Information Processing Systems. NeurIPS '21. 34. Virtual Event, Ireland. arXiv:2012.06731. <a href="/wiki/ArXiv_(identifier)" target="_blank">/wiki/ArXiv_(identifier)</a> <a href="#fnref:52" class="footnote-back-ref">↩</a></li>
<li id="fn:53">Fuhr, Norbert (1992), "Probabilistic Models in Information Retrieval", Computer Journal, 35 (3): 243–255, doi:10.1093/comjnl/35.3.243 <a href="/wiki/Doi_(identifier)" target="_blank">/wiki/Doi_(identifier)</a> <a href="#fnref:53" class="footnote-back-ref">↩</a></li>
<li id="fn:54">Fuhr, Norbert (1989), "Optimum polynomial retrieval functions based on the probability ranking principle", ACM Transactions on Information Systems, 7 (3): 183–204, doi:10.1145/65943.65944, S2CID 16632383 <a href="/wiki/Doi_(identifier)" target="_blank">/wiki/Doi_(identifier)</a> <a href="#fnref:54" class="footnote-back-ref">↩</a></li>
<li id="fn:55">Cooper, William S.; Gey, Frederic C.; Dabney, Daniel P. (1992), "Probabilistic retrieval based on staged logistic regression", Proceedings of the 15th annual international ACM SIGIR conference on Research and development in information retrieval - SIGIR '92, pp. 198–210, doi:10.1145/133160.133199, ISBN 978-0897915236, S2CID 125993 <a href="978-0897915236" target="_blank">978-0897915236</a> <a href="#fnref:55" class="footnote-back-ref">↩</a></li>
<li id="fn:56">Manning C.; Raghavan P.; Schütze H. (2008), Introduction to Information Retrieval, Cambridge University Press. Sections 7.4 Archived 2009-07-21 at the Wayback Machine and 15.5 Archived 2010-05-09 at the Wayback Machine <a href="http://nlp.stanford.edu/IR-book/html/htmledition/references-and-further-reading-7.html" target="_blank">http://nlp.stanford.edu/IR-book/html/htmledition/references-and-further-reading-7.html</a> <a href="#fnref:56" class="footnote-back-ref">↩</a></li>
<li id="fn:57">Jan O. Pedersen. The MLR Story Archived 2011-07-13 at the Wayback Machine <a href="http://jopedersen.com/Presentations/The_MLR_Story.pdf" target="_blank">http://jopedersen.com/Presentations/The_MLR_Story.pdf</a> <a href="#fnref:57" class="footnote-back-ref">↩</a></li>
<li id="fn:58">U.S. patent 7,197,497 <a href="https://patents.google.com/patent/US7197497" target="_blank">https://patents.google.com/patent/US7197497</a> <a href="#fnref:58" class="footnote-back-ref">↩</a></li>
<li id="fn:59">"Bing Search Blog: User Needs, Features and the Science behind Bing". Archived from the original on 2009-11-25. Retrieved 2009-11-19. <a href="https://www.bing.com/community/blogs/search/archive/2009/06/01/user-needs-features-and-the-science-behind-bing.aspx?PageIndex=4" target="_blank">https://www.bing.com/community/blogs/search/archive/2009/06/01/user-needs-features-and-the-science-behind-bing.aspx?PageIndex=4</a> <a href="#fnref:59" class="footnote-back-ref">↩</a></li>
<li id="fn:60">Yandex corporate blog entry about new ranking model "Snezhinsk" Archived 2012-03-01 at the Wayback Machine (in Russian) <a href="http://webmaster.ya.ru/replies.xml?item_no=5707&ncrnd=5118" target="_blank">http://webmaster.ya.ru/replies.xml?item_no=5707&ncrnd=5118</a> <a href="#fnref:60" class="footnote-back-ref">↩</a></li>
<li id="fn:61">The algorithm wasn't disclosed, but a few details were made public in [1] Archived 2010-06-01 at the Wayback Machine and [2] Archived 2010-06-01 at the Wayback Machine. <a href="http://download.yandex.ru/company/experience/GDD/Zadnie_algoritmy_Karpovich.pdf" target="_blank">http://download.yandex.ru/company/experience/GDD/Zadnie_algoritmy_Karpovich.pdf</a> <a href="#fnref:61" class="footnote-back-ref">↩</a></li>
<li id="fn:62">"Yandex's Internet Mathematics 2009 competition page". Archived from the original on 2015-03-17. Retrieved 2009-11-11. <a href="https://web.archive.org/web/20150317144535/http://imat2009.yandex.ru/academic/mathematic/2009/en/" target="_blank">https://web.archive.org/web/20150317144535/http://imat2009.yandex.ru/academic/mathematic/2009/en/</a> <a href="#fnref:62" class="footnote-back-ref">↩</a></li>
<li id="fn:63">"Yahoo Learning to Rank Challenge". Archived from the original on 2010-03-01. Retrieved 2010-02-26. <a href="https://web.archive.org/web/20100301011649/http://learningtorankchallenge.yahoo.com/" target="_blank">https://web.archive.org/web/20100301011649/http://learningtorankchallenge.yahoo.com/</a> <a href="#fnref:63" class="footnote-back-ref">↩</a></li>
<li id="fn:64">Rajaraman, Anand (2008-05-24). "Are Machine-Learned Models Prone to Catastrophic Errors?". Archived from the original on 2012-07-01. Retrieved 2009-11-11. <a href="/wiki/Anand_Rajaraman" target="_blank">/wiki/Anand_Rajaraman</a> <a href="#fnref:64" class="footnote-back-ref">↩</a></li>
<li id="fn:65">Costello, Tom (2009-06-26). "Cuil Blog: So how is Bing doing?". Archived from the original on 2009-06-27. <a href="https://archive.today/20090627213358/http://www.cuil.com/info/blog/2009/06/26/so-how-is-bing-doing" target="_blank">https://archive.today/20090627213358/http://www.cuil.com/info/blog/2009/06/26/so-how-is-bing-doing</a> <a href="#fnref:65" class="footnote-back-ref">↩</a></li>
<li id="fn:66">"How Bloomberg Integrated Learning-to-Rank into Apache Solr | Tech at Bloomberg". Tech at Bloomberg. 2017-01-23. Archived from the original on 2017-03-01. Retrieved 2017-02-28. <a href="https://www.techatbloomberg.com/blog/bloomberg-integrated-learning-rank-apache-solr/" target="_blank">https://www.techatbloomberg.com/blog/bloomberg-integrated-learning-rank-apache-solr/</a> <a href="#fnref:66" class="footnote-back-ref">↩</a></li>
<li id="fn:67">"Learning to Rank for Amazon OpenSearch Service - Amazon OpenSearch Service". docs.aws.amazon.com. Retrieved 2023-09-22. <a href="https://docs.aws.amazon.com/opensearch-service/latest/developerguide/learning-to-rank.html" target="_blank">https://docs.aws.amazon.com/opensearch-service/latest/developerguide/learning-to-rank.html</a> <a href="#fnref:67" class="footnote-back-ref">↩</a></li>
<li id="fn:68">"Elasticsearch Learning to Rank: the documentation — Elasticsearch Learning to Rank documentation". elasticsearch-learning-to-rank.readthedocs.io. Retrieved 2023-09-22. <a href="https://elasticsearch-learning-to-rank.readthedocs.io/en/latest/" target="_blank">https://elasticsearch-learning-to-rank.readthedocs.io/en/latest/</a> <a href="#fnref:68" class="footnote-back-ref">↩</a></li>
<li id="fn:69">Zhou, Mo; Niu, Zhenxing; Wang, Le; Zhang, Qilin; Hua, Gang (2020). "Adversarial Ranking Attack and Defense". arXiv:2002.11293v2 [cs.CV]. <a href="/wiki/ArXiv_(identifier)" target="_blank">/wiki/ArXiv_(identifier)</a> <a href="#fnref:69" class="footnote-back-ref">↩</a></li>
<li id="fn:70">Zhou, Mo; Niu, Zhenxing; Wang, Le; Zhang, Qilin; Hua, Gang (2020). "Adversarial Ranking Attack and Defense". arXiv:2002.11293v2 [cs.CV]. <a href="/wiki/ArXiv_(identifier)" target="_blank">/wiki/ArXiv_(identifier)</a> <a href="#fnref:70" class="footnote-back-ref">↩</a></li>
<li id="fn:71">Li, Jie; Ji, Rongrong; Liu, Hong; Hong, Xiaopeng; Gao, Yue; Tian, Qi (2019). "Universal Perturbation Attack Against Image Retrieval". International Conference on Computer Vision (ICCV 2019): 4899–4908. arXiv:1812.00552. Archived from the original on 2020-07-06. Retrieved 2020-07-04. <a href="https://openaccess.thecvf.com/content_ICCV_2019/html/Li_Universal_Perturbation_Attack_Against_Image_Retrieval_ICCV_2019_paper.html" target="_blank">https://openaccess.thecvf.com/content_ICCV_2019/html/Li_Universal_Perturbation_Attack_Against_Image_Retrieval_ICCV_2019_paper.html</a> <a href="#fnref:71" class="footnote-back-ref">↩</a></li>
<li id="fn:72">Madry, Aleksander; Makelov, Aleksandar; Schmidt, Ludwig; Tsipras, Dimitris; Vladu, Adrian (2017-06-19). "Towards Deep Learning Models Resistant to Adversarial Attacks". arXiv:1706.06083v4 [stat.ML]. <a href="/wiki/ArXiv_(identifier)" target="_blank">/wiki/ArXiv_(identifier)</a> <a href="#fnref:72" class="footnote-back-ref">↩</a></li>
</ol>

Learning to rank open-in-new

Learning to rank