Menu
Home
People
Places
Arts
History
Plants & Animals
Science
Life & Culture
Technology
Reference.org
XLNet
open-in-new
<h2 id="architecture">Architecture</h2><p> The main idea of XLNet is to model language autoregressively like the <a href="/facts/Generative_pre-trained_transformer/myah4Tjj">GPT models</a>, but allow for <i>all possible <a href="/facts/Permutation/JSw9ukmv">permutations</a></i> of a sentence.<a class="footnote-ref" id="fnref:2" href="#fn:2"><sup>2</sup></a> Concretely, consider the following sentence:</p><blockquote><p>My dog is cute.</p></blockquote><p>In standard autoregressive language modeling, the model would be tasked with predicting the probability of each word, conditioned on the previous words as its context: </p><p>We factorize the joint probability of a sequence of words x 1 , … , x T {\displaystyle x_{1},\ldots ,x_{T}} using the chain rule: Pr ( x 1 , … , x T ) = Pr ( x 1 ) Pr ( x 2 | x 1 ) Pr ( x 3 | x 1 , x 2 ) … Pr ( x T | x 1 , … , x T − 1 ) . {\displaystyle \Pr(x_{1},\ldots ,x_{T})=\Pr(x_{1})\Pr(x_{2}|x_{1})\Pr(x_{3}|x_{1},x_{2})\ldots \Pr(x_{T}|x_{1},\ldots ,x_{T-1}).} </p><p>For example, the sentence "My dog is cute" is factorized as: </p><p> Pr ( My , dog , is , cute ) = Pr ( My ) Pr ( dog | My ) Pr ( is | My , dog ) Pr ( cute | My , dog , is ) . {\displaystyle \Pr({\text{My}},{\text{dog}},{\text{is}},{\text{cute}})=\Pr({\text{My}})\Pr({\text{dog}}|{\text{My}})\Pr({\text{is}}|{\text{My}},{\text{dog}})\Pr({\text{cute}}|{\text{My}},{\text{dog}},{\text{is}}).} </p><p>Schematically, we can write it as </p><p> <MASK> <MASK> <MASK> <MASK> → My <MASK> <MASK> <MASK> → My dog <MASK> <MASK> → My dog is <MASK> → My dog is cute . {\displaystyle {\texttt {<MASK>}}{\texttt {<MASK>}}{\texttt {<MASK>}}{\texttt {<MASK>}}\to {\text{My }}{\texttt {<MASK>}}{\texttt {<MASK>}}{\texttt {<MASK>}}\to {\text{My dog }}{\texttt {<MASK>}}{\texttt {<MASK>}}\to {\text{My dog is }}{\texttt {<MASK>}}\to {\text{My dog is cute}}.} </p><p>However, for XLNet, the model is required to predict the words in a randomly generated order. Suppose we have sampled a randomly generated order 3241, then schematically, the model is required to perform the following prediction task: </p><p> <MASK> <MASK> <MASK> <MASK> → <MASK> <MASK> is <MASK> → <MASK> dog is <MASK> → <MASK> dog is cute → My dog is cute {\displaystyle {\texttt {<MASK>}}{\texttt {<MASK>}}{\texttt {<MASK>}}{\texttt {<MASK>}}\to {\texttt {<MASK>}}{\texttt {<MASK>}}{\text{is }}{\texttt {<MASK>}}\to {\texttt {<MASK>}}{\text{dog is }}{\texttt {<MASK>}}\to {\texttt {<MASK>}}{\text{dog is cute}}\to {\text{My dog is cute}}} </p><p>By considering all permutations, XLNet is able to capture longer-range dependencies and better model the bidirectional context of words. </p> <h3>Two-Stream Self-Attention</h3> <p>To implement permutation language modeling, XLNet uses a two-stream self-attention mechanism. The two streams are: </p> <ul><li>Content stream: This stream encodes the content of each word, as in standard causally masked self-attention.</li> <li>Query stream: This stream encodes the content of each word in the context of what has gone before. In more detail, it is a masked cross-attention mechanism, where the queries are from the query stream, and the key-value pairs are from the content stream.</li></ul> <p>The content stream uses the causal mask M causal = [ 0 − ∞ − ∞ … − ∞ 0 0 − ∞ … − ∞ 0 0 0 … − ∞ ⋮ ⋮ ⋮ ⋱ ⋮ 0 0 0 … 0 ] {\displaystyle M_{\text{causal}}={\begin{bmatrix}0&-\infty &-\infty &\dots &-\infty \\0&0&-\infty &\dots &-\infty \\0&0&0&\dots &-\infty \\\vdots &\vdots &\vdots &\ddots &\vdots \\0&0&0&\dots &0\end{bmatrix}}} permuted by a random <a href="/facts/Permutation_matrix/xC1ran5s">permutation matrix</a> to P M causal P − 1 {\displaystyle PM_{\text{causal}}P^{-1}} . </p><p>The query stream uses the cross-attention mask P ( M causal − ∞ I ) P − 1 {\displaystyle P(M_{\text{causal}}-\infty I)P^{-1}} , where the diagonal is subtracted away specifically to avoid the model "cheating" by looking at the content stream for what the current masked token is. </p><p>Like the causal masking for GPT models, this two-stream masked architecture allows the model to train on all tokens in one forward pass. </p> <h2 id="training">Training</h2> <p>Two models were released:<a class="footnote-ref" id="fnref:3" href="#fn:3"><sup>3</sup></a><a class="footnote-ref" id="fnref:4" href="#fn:4"><sup>4</sup></a> </p> <ul><li>XLNet-Large, cased: 110M parameters, 24-layer, 1024-hidden, 16-heads</li> <li>XLNet-Base, cased: 340M parameters, 12-layer, 768-hidden, 12-heads.</li></ul> <p>It was trained on a dataset that amounted to 32.89 billion tokens after tokenization with SentencePiece. The dataset was composed of <a href="/facts/BookCorpus/IUf51cTw">BooksCorpus</a>, and English Wikipedia, Giga5, ClueWeb 2012-B, and <a href="/facts/Common_Crawl/z0k2mxv3">Common Crawl</a>. </p><p>It was trained on 512 TPU v3 chips, for 5.5 days. At the end of training, it still under-fitted the data, meaning it could have achieved lower loss with more training. It took 0.5 million steps with an <a href="/facts/Stochastic_gradient_descent/HbcaYqQP">Adam optimizer</a>, linear learning rate decay, and a batch size of 8192.<a class="footnote-ref" id="fnref:5" href="#fn:5"><sup>5</sup></a> </p> <h2 id="see-also">See also</h2> <ul><li><a href="/facts/BERT_(language_model)/rDSueM4E">BERT (language model)</a></li> <li><a href="/facts/Transformer_(machine_learning_model)/cDbjx6a8">Transformer (machine learning model)</a></li> <li><a href="/facts/Generative_pre-trained_transformer/myah4Tjj">Generative pre-trained transformer</a></li></ul> <h2 id="references">References</h2> <ol> <li id="fn:1"><p>"xlnet". GitHub. Retrieved 2 January 2024. <a href="https://github.com/zihangdai/xlnet/" target="_blank">https://github.com/zihangdai/xlnet/</a> <a href="#fnref:1" class="footnote-back-ref">↩</a></p></li> <li id="fn:2"><p>"Pretrained models — transformers 2.0.0 documentation". huggingface.co. Retrieved 2024-08-05. <a href="https://huggingface.co/transformers/v2.0.0/pretrained_models.html" target="_blank">https://huggingface.co/transformers/v2.0.0/pretrained_models.html</a> <a href="#fnref:2" class="footnote-back-ref">↩</a></p></li> <li id="fn:3"><p>"xlnet". GitHub. Retrieved 2 January 2024. <a href="https://github.com/zihangdai/xlnet/" target="_blank">https://github.com/zihangdai/xlnet/</a> <a href="#fnref:3" class="footnote-back-ref">↩</a></p></li> <li id="fn:4"><p>"Pretrained models — transformers 2.0.0 documentation". huggingface.co. Retrieved 2024-08-05. <a href="https://huggingface.co/transformers/v2.0.0/pretrained_models.html" target="_blank">https://huggingface.co/transformers/v2.0.0/pretrained_models.html</a> <a href="#fnref:4" class="footnote-back-ref">↩</a></p></li> <li id="fn:5"><p>Yang, Zhilin; Dai, Zihang; Yang, Yiming; Carbonell, Jaime; Salakhutdinov, Ruslan; Le, Quoc V. (2 January 2020). "XLNet: Generalized Autoregressive Pretraining for Language Understanding". arXiv:1906.08237 [cs.CL]. <a href="/wiki/ArXiv_(identifier)" target="_blank">/wiki/ArXiv_(identifier)</a> <a href="#fnref:5" class="footnote-back-ref">↩</a></p></li> </ol>