Menu
Home
People
Places
Arts
History
Plants & Animals
Science
Life & Culture
Technology
Reference.org
GPT-1
open-in-new
<h2 id="reason-for-choosing-bookcorpus">Reason for choosing BookCorpus</h2> <p><a href="/facts/BookCorpus/IUf51cTw">BookCorpus</a> was chosen as a training dataset partly because the long passages of continuous text helped the model learn to handle long-range information.<a class="footnote-ref" id="fnref:9" href="#fn:9"><sup>9</sup></a> It contained over 7,000 unpublished fiction books from various genres. The rest of the datasets available at the time, while being larger, lacked this long-range structure (being "shuffled" at a sentence level).<a class="footnote-ref" id="fnref:10" href="#fn:10"><sup>10</sup></a> </p><p>The BookCorpus text was cleaned by the ftfy library to standardized punctuation and whitespace and then <a href="/facts/Tokenization_(lexical_analysis)/T1JYWpIf">tokenized</a> by <i>spaCy</i>.<a class="footnote-ref" id="fnref:11" href="#fn:11"><sup>11</sup></a> </p> <h2 id="architecture">Architecture</h2> <p>The GPT-1 architecture was a twelve-layer decoder-only <a href="/facts/Transformer_(machine_learning_model)/cDbjx6a8">transformer</a>, using twelve masked self-attention heads, with 64-dimensional states each (for a total of 768). Rather than simple <a href="/facts/Stochastic_gradient_descent/HbcaYqQP">stochastic gradient descent</a>, the <a href="/facts/Stochastic_gradient_descent/HbcaYqQP">Adam optimization algorithm</a> was used; the learning rate was increased linearly from zero over the first 2,000 updates to a maximum of 2.5×10−4, and annealed to 0 using a cosine schedule.<a class="footnote-ref" id="fnref:12" href="#fn:12"><sup>12</sup></a> GPT-1 has 117 million parameters.<a class="footnote-ref" id="fnref:13" href="#fn:13"><sup>13</sup></a> </p><p>While the fine-tuning was adapted to specific tasks, its pre-training was not; to perform the various tasks, minimal changes were performed to its underlying task-agnostic model architecture.<a class="footnote-ref" id="fnref:14" href="#fn:14"><sup>14</sup></a> Despite this, GPT-1 still improved on previous benchmarks in several language processing tasks, outperforming discriminatively-trained models with task-oriented architectures on several diverse tasks.<a class="footnote-ref" id="fnref:15" href="#fn:15"><sup>15</sup></a> </p> <h2 id="performance-and-evaluation">Performance and evaluation</h2> <p>GPT-1 achieved a 5.8% and 1.5% improvement over previous best results<a class="footnote-ref" id="fnref:16" href="#fn:16"><sup>16</sup></a> on natural language inference (also known as <i><a href="/facts/Textual_entailment/6c93bq9J">textual entailment</a></i>) tasks, evaluating the ability to interpret pairs of sentences from various datasets and classify the relationship between them as "entailment", "contradiction" or "neutral".<a class="footnote-ref" id="fnref:17" href="#fn:17"><sup>17</sup></a> Examples of such datasets include QNLI (<a href="/facts/Wikipedia/KDRNKWnc">Wikipedia</a> articles) and MultiNLI (transcribed speech, popular fiction, and government reports, among other sources);<a class="footnote-ref" id="fnref:18" href="#fn:18"><sup>18</sup></a> It similarly outperformed previous models on two tasks related to question answering and <a href="/facts/Commonsense_reasoning/HfL7zQe4">commonsense reasoning</a>—by 5.7% on RACE,<a class="footnote-ref" id="fnref:19" href="#fn:19"><sup>19</sup></a> a dataset of written question-answer pairs from middle and high school exams, and by 8.9% on the Story <a href="/facts/Cloze_test/AUgsSLjQ">Cloze Test</a>.<a class="footnote-ref" id="fnref:20" href="#fn:20"><sup>20</sup></a> </p><p>GPT-1 improved on previous best-performing models by 4.2% on <i>semantic similarity</i> (or <i>paraphrase detection</i>), evaluating the ability to predict whether two sentences are paraphrases of one another, using the <a href="/facts/Quora/sdrkWzr6">Quora</a> Question Pairs (QQP) dataset.<a class="footnote-ref" id="fnref:21" href="#fn:21"><sup>21</sup></a> </p><p>GPT-1 achieved a score of 45.4, versus a previous best of 35.0<a class="footnote-ref" id="fnref:22" href="#fn:22"><sup>22</sup></a> in a text classification task using the Corpus of Linguistic Acceptability (CoLA). Finally, GPT-1 achieved an overall score of 72.8 (compared to a previous record of 68.9) on GLUE, a multi-task test.<a class="footnote-ref" id="fnref:23" href="#fn:23"><sup>23</sup></a> </p> <h2 id="references">References</h2> <ol> <li id="fn:1"><p>Vaswani, Ashish; Shazeer, Noam; Parmar, Niki; Uszkoreit, Jakob; Jones, Llion; Gomez, Aidan N; Kaiser, Łukasz; Polosukhin, Illia (2017). "Attention is All you Need" (PDF). Advances in Neural Information Processing Systems. 30. Curran Associates, Inc. <a href="/wiki/Ashish_Vaswani" target="_blank">/wiki/Ashish_Vaswani</a> <a href="#fnref:1" class="footnote-back-ref">↩</a></p></li> <li id="fn:2"><p>Radford, Alec; Narasimhan, Karthik; Salimans, Tim; Sutskever, Ilya (11 June 2018). "Improving Language Understanding by Generative Pre-Training" (PDF). OpenAI. p. 12. Archived (PDF) from the original on 26 January 2021. Retrieved 23 January 2021. <a href="https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf" target="_blank">https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf</a> <a href="#fnref:2" class="footnote-back-ref">↩</a></p></li> <li id="fn:3"><p>"GPT-1 to GPT-4: Each of OpenAI's GPT Models Explained and Compared". 11 April 2023. Archived from the original on 2023-04-15. Retrieved 2023-04-29. <a href="https://www.makeuseof.com/gpt-models-explained-and-compared/" target="_blank">https://www.makeuseof.com/gpt-models-explained-and-compared/</a> <a href="#fnref:3" class="footnote-back-ref">↩</a></p></li> <li id="fn:4"><p>Radford, Alec; Narasimhan, Karthik; Salimans, Tim; Sutskever, Ilya (11 June 2018). "Improving Language Understanding by Generative Pre-Training" (PDF). OpenAI. p. 12. Archived (PDF) from the original on 26 January 2021. Retrieved 23 January 2021. <a href="https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf" target="_blank">https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf</a> <a href="#fnref:4" class="footnote-back-ref">↩</a></p></li> <li id="fn:5"><p>Tsvetkov, Yulia (22 June 2017). "Opportunities and Challenges in Working with Low-Resource Languages" (PDF). Carnegie Mellon University. Archived (PDF) from the original on 31 March 2020. Retrieved 23 January 2021. <a href="http://www.cs.cmu.edu/~ytsvetko/jsalt-part1.pdf" target="_blank">http://www.cs.cmu.edu/~ytsvetko/jsalt-part1.pdf</a> <a href="#fnref:5" class="footnote-back-ref">↩</a></p></li> <li id="fn:6"><p>Tsvetkov, Yulia (22 June 2017). "Opportunities and Challenges in Working with Low-Resource Languages" (PDF). Carnegie Mellon University. Archived (PDF) from the original on 31 March 2020. Retrieved 23 January 2021. <a href="http://www.cs.cmu.edu/~ytsvetko/jsalt-part1.pdf" target="_blank">http://www.cs.cmu.edu/~ytsvetko/jsalt-part1.pdf</a> <a href="#fnref:6" class="footnote-back-ref">↩</a></p></li> <li id="fn:7"><p>Radford, Alec; Narasimhan, Karthik; Salimans, Tim; Sutskever, Ilya (11 June 2018). "Improving Language Understanding by Generative Pre-Training" (PDF). OpenAI. p. 12. Archived (PDF) from the original on 26 January 2021. Retrieved 23 January 2021. <a href="https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf" target="_blank">https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf</a> <a href="#fnref:7" class="footnote-back-ref">↩</a></p></li> <li id="fn:8"><p>Radford, Alec; Narasimhan, Karthik; Salimans, Tim; Sutskever, Ilya (11 June 2018). "Improving Language Understanding by Generative Pre-Training" (PDF). OpenAI. p. 12. Archived (PDF) from the original on 26 January 2021. Retrieved 23 January 2021. <a href="https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf" target="_blank">https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf</a> <a href="#fnref:8" class="footnote-back-ref">↩</a></p></li> <li id="fn:9"><p>Zhu, Yukun; Kiros, Ryan; Zemel, Richard; Salakhutdinov, Ruslan; Urtasun, Raquel; Torralba, Antonio; Fidler, Sanja (22 June 2015). "Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books". arXiv:1506.06724 [cs.CV]. # of books: 11,038 / # of sentences: 74,004,228 / # of words: 984,846,357 / mean # of words per sentence: 13 / median # of words per sentence: 11 <a href="/wiki/ArXiv_(identifier)" target="_blank">/wiki/ArXiv_(identifier)</a> <a href="#fnref:9" class="footnote-back-ref">↩</a></p></li> <li id="fn:10"><p>Radford, Alec; Narasimhan, Karthik; Salimans, Tim; Sutskever, Ilya (11 June 2018). "Improving Language Understanding by Generative Pre-Training" (PDF). OpenAI. p. 12. Archived (PDF) from the original on 26 January 2021. Retrieved 23 January 2021. <a href="https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf" target="_blank">https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf</a> <a href="#fnref:10" class="footnote-back-ref">↩</a></p></li> <li id="fn:11"><p>Radford, Alec; Narasimhan, Karthik; Salimans, Tim; Sutskever, Ilya (11 June 2018). "Improving Language Understanding by Generative Pre-Training" (PDF). OpenAI. p. 12. Archived (PDF) from the original on 26 January 2021. Retrieved 23 January 2021. <a href="https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf" target="_blank">https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf</a> <a href="#fnref:11" class="footnote-back-ref">↩</a></p></li> <li id="fn:12"><p>Radford, Alec; Narasimhan, Karthik; Salimans, Tim; Sutskever, Ilya (11 June 2018). "Improving Language Understanding by Generative Pre-Training" (PDF). OpenAI. p. 12. Archived (PDF) from the original on 26 January 2021. Retrieved 23 January 2021. <a href="https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf" target="_blank">https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf</a> <a href="#fnref:12" class="footnote-back-ref">↩</a></p></li> <li id="fn:13"><p>"GPT-1 to GPT-4: Each of OpenAI's GPT Models Explained and Compared". 11 April 2023. Archived from the original on 2023-04-15. Retrieved 2023-04-29. <a href="https://www.makeuseof.com/gpt-models-explained-and-compared/" target="_blank">https://www.makeuseof.com/gpt-models-explained-and-compared/</a> <a href="#fnref:13" class="footnote-back-ref">↩</a></p></li> <li id="fn:14"><p>Radford, Alec; Narasimhan, Karthik; Salimans, Tim; Sutskever, Ilya (11 June 2018). "Improving Language Understanding by Generative Pre-Training" (PDF). OpenAI. p. 12. Archived (PDF) from the original on 26 January 2021. Retrieved 23 January 2021. <a href="https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf" target="_blank">https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf</a> <a href="#fnref:14" class="footnote-back-ref">↩</a></p></li> <li id="fn:15"><p>Radford, Alec; Narasimhan, Karthik; Salimans, Tim; Sutskever, Ilya (11 June 2018). "Improving Language Understanding by Generative Pre-Training" (PDF). OpenAI. p. 12. Archived (PDF) from the original on 26 January 2021. Retrieved 23 January 2021. <a href="https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf" target="_blank">https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf</a> <a href="#fnref:15" class="footnote-back-ref">↩</a></p></li> <li id="fn:16"><p>Radford, Alec; Narasimhan, Karthik; Salimans, Tim; Sutskever, Ilya (11 June 2018). "Improving Language Understanding by Generative Pre-Training" (PDF). OpenAI. p. 12. Archived (PDF) from the original on 26 January 2021. Retrieved 23 January 2021. <a href="https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf" target="_blank">https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf</a> <a href="#fnref:16" class="footnote-back-ref">↩</a></p></li> <li id="fn:17"><p>Radford, Alec; Narasimhan, Karthik; Salimans, Tim; Sutskever, Ilya (11 June 2018). "Improving Language Understanding by Generative Pre-Training" (PDF). OpenAI. p. 12. Archived (PDF) from the original on 26 January 2021. Retrieved 23 January 2021. <a href="https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf" target="_blank">https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf</a> <a href="#fnref:17" class="footnote-back-ref">↩</a></p></li> <li id="fn:18"><p>Williams, Adina; Nangia, Nikita; Bowman, Samuel (1 June 2018). "A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference" (PDF). Association for Computational Linguistics. Archived (PDF) from the original on 11 February 2020. Retrieved 23 January 2021. At 433k examples, this resource is one of the largest corpora available for natural language inference (a.k.a. recognizing textual entailment), [...] offering data from ten distinct genres of written and spoken English [...] while supplying an explicit setting for evaluating cross-genre domain adaptation. <a href="https://www.aclweb.org/anthology/N18-1101.pdf" target="_blank">https://www.aclweb.org/anthology/N18-1101.pdf</a> <a href="#fnref:18" class="footnote-back-ref">↩</a></p></li> <li id="fn:19"><p>Lai, Guokun; Xie, Qizhe; Hanxiao, Liu; Yang, Yiming; Hovy, Eduard (15 April 2017). "RACE: Large-scale ReAding Comprehension Dataset From Examinations". arXiv:1704.04683 [cs.CL]. <a href="/wiki/ArXiv_(identifier)" target="_blank">/wiki/ArXiv_(identifier)</a> <a href="#fnref:19" class="footnote-back-ref">↩</a></p></li> <li id="fn:20"><p>Mostafazadeh, Nasrin; Roth, Michael; Louis, Annie; Chambers, Nathanael; Allen, James F. (3 April 2017). "LSDSem 2017 Shared Task: The Story Cloze Test" (PDF). Association for Computational Linguistics. Archived (PDF) from the original on 22 November 2020. Retrieved 23 January 2021. The LSDSem'17 shared task is the Story Cloze Test, a new evaluation for story understanding and script learning. This test provides a system with a four-sentence story and two possible endings, and the system must choose the correct ending. Successful narrative understanding (getting closer to human performance of 100%) requires systems to link various levels of semantics to commonsense knowledge. <a href="https://www.aclweb.org/anthology/W17-0906.pdf" target="_blank">https://www.aclweb.org/anthology/W17-0906.pdf</a> <a href="#fnref:20" class="footnote-back-ref">↩</a></p></li> <li id="fn:21"><p>Radford, Alec; Narasimhan, Karthik; Salimans, Tim; Sutskever, Ilya (11 June 2018). "Improving Language Understanding by Generative Pre-Training" (PDF). OpenAI. p. 12. Archived (PDF) from the original on 26 January 2021. Retrieved 23 January 2021. <a href="https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf" target="_blank">https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf</a> <a href="#fnref:21" class="footnote-back-ref">↩</a></p></li> <li id="fn:22"><p>Radford, Alec; Narasimhan, Karthik; Salimans, Tim; Sutskever, Ilya (11 June 2018). "Improving Language Understanding by Generative Pre-Training" (PDF). OpenAI. p. 12. Archived (PDF) from the original on 26 January 2021. Retrieved 23 January 2021. <a href="https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf" target="_blank">https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf</a> <a href="#fnref:22" class="footnote-back-ref">↩</a></p></li> <li id="fn:23"><p>Wang, Alex; Singh, Amanpreet; Michael, Julian; Hill, Felix; Levy, Omar; Bowman, Samuel R. (20 April 2018). "GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding". arXiv:1804.07461 [cs.CL]. <a href="/wiki/ArXiv_(identifier)" target="_blank">/wiki/ArXiv_(identifier)</a> <a href="#fnref:23" class="footnote-back-ref">↩</a></p></li> </ol>