Stochastic parrot

<h2 id="subsequent-usage">Subsequent usage</h2>
Stochastic parrot is now a <a href="/facts/Neologism/vlq6w82X">neologism</a> used by AI skeptics to allege that LLM's lack understanding of the meaning of their outputs and is sometimes interpreted as a "slur against AI".<a class="footnote-ref" id="fnref:10" href="#fn:10">10</a> Its use expanded further when <a href="/facts/Sam_Altman/jhstWT0n">Sam Altman</a>, CEO of <a href="/facts/OpenAI/V7WVK1t4">Open AI</a>, used the term ironically when he tweeted, "i am a stochastic parrot and so r u", pointing out that, by the same reasoning, one could also disparage humans as mere next-word predictors whose brains (or pens) simply generate statistically likely sequences.<a class="footnote-ref" id="fnref:11" href="#fn:11">11</a> The term was then designated to be the 2023 AI-related Word of the Year for the <a href="/facts/American_Dialect_Society/0FmXAlN0">American Dialect Society</a>, even over the words "ChatGPT" and "LLM".<a class="footnote-ref" id="fnref:12" href="#fn:12">12</a><a class="footnote-ref" id="fnref:13" href="#fn:13">13</a>
The phrase is often referenced by some researchers to describe LLMs as pattern matchers that can generate plausible human-like text through their vast amount of training data, merely parroting in a stochastic fashion. However, other researchers argue that LLMs are, in fact, at least partially able to understand language.<a class="footnote-ref" id="fnref:14" href="#fn:14">14</a>

<h2 id="debate">Debate</h2>
Some LLMs, such as ChatGPT, have become capable of interacting with users in convincingly human-like conversations.<a class="footnote-ref" id="fnref:15" href="#fn:15">15</a> The development of these new systems has deepened the discussion of the extent to which LLMs understand or are simply "parroting".

<h3>Subjective experience</h3>
In the mind of a human being, words and language correspond to things one has experienced.<a class="footnote-ref" id="fnref:16" href="#fn:16">16</a> For LLMs, words may correspond only to other words and patterns of usage fed into their training data.<a class="footnote-ref" id="fnref:17" href="#fn:17">17</a><a class="footnote-ref" id="fnref:18" href="#fn:18">18</a><a class="footnote-ref" id="fnref:19" href="#fn:19">19</a> Proponents of the idea of stochastic parrots thus conclude that LLMs are incapable of actually understanding language.<a class="footnote-ref" id="fnref:20" href="#fn:20">20</a><a class="footnote-ref" id="fnref:21" href="#fn:21">21</a>

<h3>Hallucinations and mistakes</h3>
The tendency of LLMs to pass off fake information as fact is held as support.<a class="footnote-ref" id="fnref:22" href="#fn:22">22</a> Called <a href="/facts/Hallucination_(artificial_intelligence)/gwBoRg6p">hallucinations</a> or confabulations, LLMs will occasionally synthesize information that matches some pattern, but not reality.<a class="footnote-ref" id="fnref:23" href="#fn:23">23</a><a class="footnote-ref" id="fnref:24" href="#fn:24">24</a><a class="footnote-ref" id="fnref:25" href="#fn:25">25</a> That LLMs can’t distinguish fact and fiction leads to the claim that they can’t connect words to a comprehension of the world, as language should do.<a class="footnote-ref" id="fnref:26" href="#fn:26">26</a><a class="footnote-ref" id="fnref:27" href="#fn:27">27</a> Further, LLMs often fail to decipher complex or ambiguous grammar cases that rely on understanding the meaning of language.<a class="footnote-ref" id="fnref:28" href="#fn:28">28</a><a class="footnote-ref" id="fnref:29" href="#fn:29">29</a> As an example, borrowing from Saba et al., is the prompt:<a class="footnote-ref" id="fnref:30" href="#fn:30">30</a>

<blockquote>The wet newspaper that fell down off the table is my favorite newspaper. But now that my favorite newspaper fired the editor I might not like reading it anymore. Can I replace ‘my favorite newspaper’ by ‘the wet newspaper that fell down off the table’ in the second sentence?</blockquote>
Some LLMs respond to this in the affirmative, not understanding that the meaning of "newspaper" is different in these two contexts; it is first an object and second an institution.<a class="footnote-ref" id="fnref:31" href="#fn:31">31</a> Based on these failures, some AI professionals conclude they are no more than stochastic parrots.<a class="footnote-ref" id="fnref:32" href="#fn:32">32</a><a class="footnote-ref" id="fnref:33" href="#fn:33">33</a><a class="footnote-ref" id="fnref:34" href="#fn:34">34</a>

<h3>Benchmarks and experiments</h3>
One argument against the hypothesis that LLMs are stochastic parrot is their results on <a href="/facts/Benchmark_(computing)/XSS42cRe">benchmarks</a> for reasoning, common sense and language understanding. In 2023, some LLMs have shown good results on many language understanding tests, such as the Super General Language Understanding Evaluation (SuperGLUE).<a class="footnote-ref" id="fnref:35" href="#fn:35">35</a><a class="footnote-ref" id="fnref:36" href="#fn:36">36</a> GPT-4 scored in the >90th-percentile on the Uniform Bar Examination and achieved 93% accuracy on the MATH benchmark of high-school Olympiad problems, results that exceed rote pattern-matching expectations.<a class="footnote-ref" id="fnref:37" href="#fn:37">37</a> Such tests, and the smoothness of many LLM responses, help as many as 51% of AI professionals believe they can truly understand language with enough data, according to a 2022 survey.<a class="footnote-ref" id="fnref:38" href="#fn:38">38</a>

<h3>Expert rebuttals</h3>
Leading AI researchers dispute the notion that LLMs merely “parrot” their training data.

<ul><li><a href="/facts/Geoffrey_Hinton/HJU6lC2H">Geoffrey Hinton</a> argues that “to predict the next word accurately you have to understand the sentence”, a view he presented on 60 Minutes in 2023.<a class="footnote-ref" id="fnref:39" href="#fn:39">39</a> He also uses logical puzzles to demonstrate that LLMs actually understand language.<a class="footnote-ref" id="fnref:40" href="#fn:40">40</a></li>
<li>A 2024 <a href="/facts/Scientific_American/jkl8bhUN">Scientific American</a> investigation described a closed Berkeley workshop where state-of-the-art models solved novel tier-4 mathematics problems and produced coherent proofs, indicating reasoning abilities beyond memorization.<a class="footnote-ref" id="fnref:41" href="#fn:41">41</a></li>
<li>The GPT-4 Technical Report showed human-level results on professional and academic exams (e.g., the Uniform Bar Exam and USMLE), challenging the “parrot” characterization.<a class="footnote-ref" id="fnref:42" href="#fn:42">42</a></li></ul>
<h3>Interpretability</h3>
Another technique for investigating if LLMs can understand is termed "mechanistic interpretability". The idea is to <a href="/facts/Reverse_engineering/pmJCV7J7">reverse-engineer</a> a large language model to analyze how it internally processes the information.
One example is Othello-GPT, where a small <a href="/facts/Transformer_(deep_learning_architecture)/cDbjx6a8">transformer</a> was trained to predict legal <a href="/facts/Reversi/Bxp4jHAw">Othello</a> moves. It has been found that this model has an internal representation of the Othello board, and that modifying this representation changes the predicted legal Othello moves in the correct way. This supports the idea that LLMs have a "world model", and are not just doing superficial statistics.<a class="footnote-ref" id="fnref:43" href="#fn:43">43</a><a class="footnote-ref" id="fnref:44" href="#fn:44">44</a> 
In another example, a small transformer was trained on computer programs written in the programming language <a href="/facts/Karel_(programming_language)/X7zj2zwF">Karel</a>. Similar to the Othello-GPT example, this model developed an internal representation of Karel program semantics. Modifying this representation results in appropriate changes to the output. Additionally, the model generates correct programs that are, on average, shorter than those in the training set.<a class="footnote-ref" id="fnref:45" href="#fn:45">45</a>
Researchers also studied "<a href="/facts/Grokking_(machine_learning)/KwAyAf8x">grokking</a>", a phenomenon where an AI model initially memorizes the training data outputs, and then, after further training, suddenly finds a solution that generalizes to unseen data.<a class="footnote-ref" id="fnref:46" href="#fn:46">46</a>

<h3>Shortcuts to reasoning</h3>
However, when tests created to test people for language comprehension are used to test LLMs, they sometimes result in false positives caused by spurious correlations within text data.<a class="footnote-ref" id="fnref:47" href="#fn:47">47</a> Models have shown examples of shortcut learning, which is when a system makes unrelated correlations within data instead of using human-like understanding.<a class="footnote-ref" id="fnref:48" href="#fn:48">48</a> One such experiment conducted in 2019 tested Google’s <a href="/facts/BERT_(language_model)/rDSueM4E">BERT</a> LLM using the argument reasoning comprehension task. BERT was prompted to choose between 2 statements, and find the one most consistent with an argument. Below is an example of one of these prompts:<a class="footnote-ref" id="fnref:49" href="#fn:49">49</a><a class="footnote-ref" id="fnref:50" href="#fn:50">50</a>

<blockquote>
Argument: Felons should be allowed to vote. A person who stole a car at 17 should not be barred from being a full citizen for life.
Statement A: Grand theft auto is a felony.
Statement B: Grand theft auto is not a felony.

</blockquote>
Researchers found that specific words such as "not" hint the model towards the correct answer, allowing near-perfect scores when included but resulting in random selection when hint words were removed.<a class="footnote-ref" id="fnref:51" href="#fn:51">51</a><a class="footnote-ref" id="fnref:52" href="#fn:52">52</a> This problem, and the known difficulties defining intelligence, causes some to argue all benchmarks that find understanding in LLMs are flawed, that they all allow shortcuts to fake understanding.

<h2 id="see-also">See also</h2>
<ul><li><a href="/facts/1_the_Road/LTDa39gg">1 the Road</a> – AI-generated novel</li>
<li><a href="/facts/Chinese_room/YeKegsxS">Chinese room</a></li>
<li><a href="/facts/Criticism_of_artificial_neural_networks/6V1jMlkx">Criticism of artificial neural networks</a></li>
<li><a href="/facts/Criticism_of_deep_learning/JLuwD3ea">Criticism of deep learning</a></li>
<li><a href="/facts/Criticism_of_Google/XIdunl5k">Criticism of Google</a></li>
<li><a href="/facts/Cut-up_technique/55k4wkkS">Cut-up technique</a></li>
<li><a href="/facts/Infinite_monkey_theorem/rE8Rchab">Infinite monkey theorem</a></li>
<li><a href="/facts/Generative_AI/ykT3GGyT">Generative AI</a></li>
<li><a href="/facts/Mark_V._Shaney/tjGYvEXG">Mark V. Shaney</a>, an early chatbot that used a very simple three-word Markov chain algorithm to generate <a href="/facts/Markov_text/5oP5hntk">Markov text</a></li></ul>

<h3>Works cited</h3>
<ul><li>Lindholm, A.; Wahlström, N.; Lindsten, F.; Schön, T. B. (2022). Machine Learning: A First Course for Engineers and Scientists. Cambridge University Press. <a href="/facts/ISBN_(identifier)/15AdSPa9">ISBN</a> 978-1108843607.</li>
<li>Weller, Adrian (July 13, 2021). <a href="https://www.youtube.com/watch?v=N5c2X8vhfBE">On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜</a> (video). <a href="/facts/Alan_Turing_Institute/CFqFw6RB">Alan Turing Institute</a>. Keynote by Emily Bender. The presentation was followed by a panel discussion.</li></ul>
<h2 id="further-reading">Further reading</h2>
<ul><li>Bogost, Ian (December 7, 2022). <a href="https://www.theatlantic.com/technology/archive/2022/12/chatgpt-openai-artificial-intelligence-writing-ethics/672386/">"ChatGPT Is Dumber Than You Think: Treat it like a toy, not a tool"</a>. <a href="/facts/The_Atlantic/zVtl4Lcr">The Atlantic</a>. Retrieved 2024-01-17.</li>
<li>Chomsky, Noam (March 8, 2023). <a href="https://www.nytimes.com/2023/03/08/opinion/noam-chomsky-chatgpt-ai.html">"The False Promise of ChatGPT"</a>. <a href="/facts/The_New_York_Times/T9marWVX">The New York Times</a>. Retrieved 2024-01-17.</li>
<li>Glenberg, Arthur; Jones, Cameron Robert (April 6, 2023). <a href="https://theconversation.com/it-takes-a-body-to-understand-the-world-why-chatgpt-and-other-language-ais-dont-know-what-theyre-saying-201280">"It takes a body to understand the world – why ChatGPT and other language AIs don't know what they're saying"</a>. The Conversation. Retrieved 2024-01-17.</li>
<li>McQuillan, D. (2022). <a href="/facts/Resisting_AI/6t5xyX4L">Resisting AI: An Anti-fascist Approach to Artificial Intelligence</a>. <a href="/facts/Bristol_University_Press/naU7Gvlc">Bristol University Press</a>. <a href="/facts/ISBN_(identifier)/15AdSPa9">ISBN</a> 978-1-5292-1350-8.</li>
<li>Thompson, E. (2022). Escape from Model Land: How Mathematical Models Can Lead Us Astray and What We Can Do about It. Basic Books. <a href="/facts/ISBN_(identifier)/15AdSPa9">ISBN</a> 978-1-5416-0098-0.</li>
<li>Zhong, Qihuang; Ding, Liang; Liu, Juhua; Du, Bo; Tao, Dacheng (2023). "Can ChatGPT Understand Too? A Comparative Study on ChatGPT and Fine-tuned BERT". <a href="/facts/ArXiv_(identifier)/H6EtgnBe">arXiv</a>:<a href="https://arxiv.org/abs/2302.10198">2302.10198</a> [<a href="https://arxiv.org/archive/cs.CL">cs.CL</a>].</li></ul>
<h2 id="external-links">External links</h2>
<ul><li>"<a href="https://commons.wikimedia.org/wiki/File:On_the_Dangers_of_Stochastic_Parrots_Can_Language_Models_Be_Too_Big.pdf">On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜</a>" at <a href="/facts/Wikimedia_Commons/lzkBFxTg">Wikimedia Commons</a></li></ul>

<h2 id="references">References</h2>

<ol>
<li id="fn:1">Bender, Emily M.; Gebru, Timnit; McMillan-Major, Angelina; Mitchell, Margaret (2021). "On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?". Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency. doi:10.1145/3442188.3445922. <a href="/wiki/Doi_(identifier)" target="_blank">/wiki/Doi_(identifier)</a> <a href="#fnref:1" class="footnote-back-ref">↩</a></li>
<li id="fn:2">Bubeck, Sébastien (2023). "Sparks of Artificial General Intelligence: Early experiments with GPT-4". arXiv:2303.12712. A bot will complete this citation soon. Click here to jump the queue <a href="/wiki/ArXiv_(identifier)" target="_blank">/wiki/ArXiv_(identifier)</a> <a href="#fnref:2" class="footnote-back-ref">↩</a></li>
<li id="fn:3">Pelley, Scott (8 October 2023). ""Godfather of Artificial Intelligence" Geoffrey Hinton on the promise, risks of advanced AI". CBS News. Retrieved 2 July 2025. <a href="https://www.cbsnews.com/news/geoffrey-hinton-ai-dangers-60-minutes-transcript/" target="_blank">https://www.cbsnews.com/news/geoffrey-hinton-ai-dangers-60-minutes-transcript/</a> <a href="#fnref:3" class="footnote-back-ref">↩</a></li>
<li id="fn:4">Bender, Emily M.; Gebru, Timnit; McMillan-Major, Angelina; Mitchell, Margaret (2021). "On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?". Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency. doi:10.1145/3442188.3445922. <a href="/wiki/Doi_(identifier)" target="_blank">/wiki/Doi_(identifier)</a> <a href="#fnref:4" class="footnote-back-ref">↩</a></li>
<li id="fn:5">Lindholm et al. 2022, pp. 322–3. - Lindholm, A.; Wahlström, N.; Lindsten, F.; Schön, T. B. (2022). Machine Learning: A First Course for Engineers and Scientists. Cambridge University Press. ISBN 978-1108843607. <a href="#fnref:5" class="footnote-back-ref">↩</a></li>
<li id="fn:6">Uddin, Muhammad Saad (April 20, 2023). "Stochastic Parrots: A Novel Look at Large Language Models and Their Limitations". Towards AI. Retrieved 2023-05-12. <a href="https://towardsai.net/p/machine-learning/stochastic-parrots-a-novel-look-at-large-language-models-and-their-limitations" target="_blank">https://towardsai.net/p/machine-learning/stochastic-parrots-a-novel-look-at-large-language-models-and-their-limitations</a> <a href="#fnref:6" class="footnote-back-ref">↩</a></li>
<li id="fn:7">Pelley, Scott (8 October 2023). ""Godfather of Artificial Intelligence" Geoffrey Hinton on the promise, risks of advanced AI". CBS News. Retrieved 2 July 2025. <a href="https://www.cbsnews.com/news/geoffrey-hinton-ai-dangers-60-minutes-transcript/" target="_blank">https://www.cbsnews.com/news/geoffrey-hinton-ai-dangers-60-minutes-transcript/</a> <a href="#fnref:7" class="footnote-back-ref">↩</a></li>
<li id="fn:8">Bubeck, Sébastien (2023). "Sparks of Artificial General Intelligence: Early experiments with GPT-4". arXiv:2303.12712. A bot will complete this citation soon. Click here to jump the queue <a href="/wiki/ArXiv_(identifier)" target="_blank">/wiki/ArXiv_(identifier)</a> <a href="#fnref:8" class="footnote-back-ref">↩</a></li>
<li id="fn:9">Lindholm et al. 2022, pp. 322–3. - Lindholm, A.; Wahlström, N.; Lindsten, F.; Schön, T. B. (2022). Machine Learning: A First Course for Engineers and Scientists. Cambridge University Press. ISBN 978-1108843607. <a href="#fnref:9" class="footnote-back-ref">↩</a></li>
<li id="fn:10">Zimmer, Ben (2024-01-18). "'Stochastic Parrot': A Name for AI That Sounds a Bit Less Intelligent". Wall Street Journal. Retrieved 2024-04-01. <a href="https://www.wsj.com/arts-culture/books/stochastic-parrot-a-name-for-ai-that-sounds-a-bit-less-intelligent-789372f5" target="_blank">https://www.wsj.com/arts-culture/books/stochastic-parrot-a-name-for-ai-that-sounds-a-bit-less-intelligent-789372f5</a> <a href="#fnref:10" class="footnote-back-ref">↩</a></li>
<li id="fn:11">Zimmer, Ben (2024-01-18). "'Stochastic Parrot': A Name for AI That Sounds a Bit Less Intelligent". Wall Street Journal. Retrieved 2024-04-01. <a href="https://www.wsj.com/arts-culture/books/stochastic-parrot-a-name-for-ai-that-sounds-a-bit-less-intelligent-789372f5" target="_blank">https://www.wsj.com/arts-culture/books/stochastic-parrot-a-name-for-ai-that-sounds-a-bit-less-intelligent-789372f5</a> <a href="#fnref:11" class="footnote-back-ref">↩</a></li>
<li id="fn:12">Zimmer, Ben (2024-01-18). "'Stochastic Parrot': A Name for AI That Sounds a Bit Less Intelligent". Wall Street Journal. Retrieved 2024-04-01. <a href="https://www.wsj.com/arts-culture/books/stochastic-parrot-a-name-for-ai-that-sounds-a-bit-less-intelligent-789372f5" target="_blank">https://www.wsj.com/arts-culture/books/stochastic-parrot-a-name-for-ai-that-sounds-a-bit-less-intelligent-789372f5</a> <a href="#fnref:12" class="footnote-back-ref">↩</a></li>
<li id="fn:13">Corbin, Sam (2024-01-15). "Among Linguists, the Word of the Year Is More of a Vibe". The New York Times. ISSN 0362-4331. Retrieved 2024-04-01. <a href="https://www.nytimes.com/2024/01/15/crosswords/linguistics-word-of-the-year.html" target="_blank">https://www.nytimes.com/2024/01/15/crosswords/linguistics-word-of-the-year.html</a> <a href="#fnref:13" class="footnote-back-ref">↩</a></li>
<li id="fn:14">Arkoudas, Konstantine (2023-08-21). "ChatGPT is no Stochastic Parrot. But it also Claims that 1 is Greater than 1". Philosophy & Technology. 36 (3): 54. doi:10.1007/s13347-023-00619-6. ISSN 2210-5441. <a href="https://doi.org/10.1007/s13347-023-00619-6" target="_blank">https://doi.org/10.1007/s13347-023-00619-6</a> <a href="#fnref:14" class="footnote-back-ref">↩</a></li>
<li id="fn:15">Arkoudas, Konstantine (2023-08-21). "ChatGPT is no Stochastic Parrot. But it also Claims that 1 is Greater than 1". Philosophy & Technology. 36 (3): 54. doi:10.1007/s13347-023-00619-6. ISSN 2210-5441. <a href="https://doi.org/10.1007/s13347-023-00619-6" target="_blank">https://doi.org/10.1007/s13347-023-00619-6</a> <a href="#fnref:15" class="footnote-back-ref">↩</a></li>
<li id="fn:16">Fayyad, Usama M. (2023-05-26). "From Stochastic Parrots to Intelligent Assistants—The Secrets of Data and Human Interventions". IEEE Intelligent Systems. 38 (3): 63–67. doi:10.1109/MIS.2023.3268723. ISSN 1541-1672. <a href="https://ieeexplore.ieee.org/document/10148666" target="_blank">https://ieeexplore.ieee.org/document/10148666</a> <a href="#fnref:16" class="footnote-back-ref">↩</a></li>
<li id="fn:17">Saba, Walid S. (2023). "Stochastic LLMS do not Understand Language: Towards Symbolic, Explainable and Ontologically Based LLMS". In Almeida, João Paulo A.; Borbinha, José; Guizzardi, Giancarlo; Link, Sebastian; Zdravkovic, Jelena (eds.). Conceptual Modeling. Lecture Notes in Computer Science. Vol. 14320. Cham: Springer Nature Switzerland. pp. 3–19. arXiv:2309.05918. doi:10.1007/978-3-031-47262-6_1. ISBN 978-3-031-47262-6. <a href="978-3-031-47262-6" target="_blank">978-3-031-47262-6</a> <a href="#fnref:17" class="footnote-back-ref">↩</a></li>
<li id="fn:18">Mitchell, Melanie; Krakauer, David C. (2023-03-28). "The debate over understanding in AI's large language models". Proceedings of the National Academy of Sciences. 120 (13): e2215907120. arXiv:2210.13966. Bibcode:2023PNAS..12015907M. doi:10.1073/pnas.2215907120. ISSN 0027-8424. PMC 10068812. PMID 36943882. <a href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10068812" target="_blank">https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10068812</a> <a href="#fnref:18" class="footnote-back-ref">↩</a></li>
<li id="fn:19">Bender, Emily M.; Gebru, Timnit; McMillan-Major, Angelina; Mitchell, Margaret (2021). "On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?". Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency. doi:10.1145/3442188.3445922. <a href="/wiki/Doi_(identifier)" target="_blank">/wiki/Doi_(identifier)</a> <a href="#fnref:19" class="footnote-back-ref">↩</a></li>
<li id="fn:20">Saba, Walid S. (2023). "Stochastic LLMS do not Understand Language: Towards Symbolic, Explainable and Ontologically Based LLMS". In Almeida, João Paulo A.; Borbinha, José; Guizzardi, Giancarlo; Link, Sebastian; Zdravkovic, Jelena (eds.). Conceptual Modeling. Lecture Notes in Computer Science. Vol. 14320. Cham: Springer Nature Switzerland. pp. 3–19. arXiv:2309.05918. doi:10.1007/978-3-031-47262-6_1. ISBN 978-3-031-47262-6. <a href="978-3-031-47262-6" target="_blank">978-3-031-47262-6</a> <a href="#fnref:20" class="footnote-back-ref">↩</a></li>
<li id="fn:21">Bender, Emily M.; Gebru, Timnit; McMillan-Major, Angelina; Mitchell, Margaret (2021). "On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?". Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency. doi:10.1145/3442188.3445922. <a href="/wiki/Doi_(identifier)" target="_blank">/wiki/Doi_(identifier)</a> <a href="#fnref:21" class="footnote-back-ref">↩</a></li>
<li id="fn:22">Fayyad, Usama M. (2023-05-26). "From Stochastic Parrots to Intelligent Assistants—The Secrets of Data and Human Interventions". IEEE Intelligent Systems. 38 (3): 63–67. doi:10.1109/MIS.2023.3268723. ISSN 1541-1672. <a href="https://ieeexplore.ieee.org/document/10148666" target="_blank">https://ieeexplore.ieee.org/document/10148666</a> <a href="#fnref:22" class="footnote-back-ref">↩</a></li>
<li id="fn:23">Saba, Walid S. (2023). "Stochastic LLMS do not Understand Language: Towards Symbolic, Explainable and Ontologically Based LLMS". In Almeida, João Paulo A.; Borbinha, José; Guizzardi, Giancarlo; Link, Sebastian; Zdravkovic, Jelena (eds.). Conceptual Modeling. Lecture Notes in Computer Science. Vol. 14320. Cham: Springer Nature Switzerland. pp. 3–19. arXiv:2309.05918. doi:10.1007/978-3-031-47262-6_1. ISBN 978-3-031-47262-6. <a href="978-3-031-47262-6" target="_blank">978-3-031-47262-6</a> <a href="#fnref:23" class="footnote-back-ref">↩</a></li>
<li id="fn:24">Mitchell, Melanie; Krakauer, David C. (2023-03-28). "The debate over understanding in AI's large language models". Proceedings of the National Academy of Sciences. 120 (13): e2215907120. arXiv:2210.13966. Bibcode:2023PNAS..12015907M. doi:10.1073/pnas.2215907120. ISSN 0027-8424. PMC 10068812. PMID 36943882. <a href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10068812" target="_blank">https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10068812</a> <a href="#fnref:24" class="footnote-back-ref">↩</a></li>
<li id="fn:25">Fayyad, Usama M. (2023-05-26). "From Stochastic Parrots to Intelligent Assistants—The Secrets of Data and Human Interventions". IEEE Intelligent Systems. 38 (3): 63–67. doi:10.1109/MIS.2023.3268723. ISSN 1541-1672. <a href="https://ieeexplore.ieee.org/document/10148666" target="_blank">https://ieeexplore.ieee.org/document/10148666</a> <a href="#fnref:25" class="footnote-back-ref">↩</a></li>
<li id="fn:26">Saba, Walid S. (2023). "Stochastic LLMS do not Understand Language: Towards Symbolic, Explainable and Ontologically Based LLMS". In Almeida, João Paulo A.; Borbinha, José; Guizzardi, Giancarlo; Link, Sebastian; Zdravkovic, Jelena (eds.). Conceptual Modeling. Lecture Notes in Computer Science. Vol. 14320. Cham: Springer Nature Switzerland. pp. 3–19. arXiv:2309.05918. doi:10.1007/978-3-031-47262-6_1. ISBN 978-3-031-47262-6. <a href="978-3-031-47262-6" target="_blank">978-3-031-47262-6</a> <a href="#fnref:26" class="footnote-back-ref">↩</a></li>
<li id="fn:27">Fayyad, Usama M. (2023-05-26). "From Stochastic Parrots to Intelligent Assistants—The Secrets of Data and Human Interventions". IEEE Intelligent Systems. 38 (3): 63–67. doi:10.1109/MIS.2023.3268723. ISSN 1541-1672. <a href="https://ieeexplore.ieee.org/document/10148666" target="_blank">https://ieeexplore.ieee.org/document/10148666</a> <a href="#fnref:27" class="footnote-back-ref">↩</a></li>
<li id="fn:28">Saba, Walid S. (2023). "Stochastic LLMS do not Understand Language: Towards Symbolic, Explainable and Ontologically Based LLMS". In Almeida, João Paulo A.; Borbinha, José; Guizzardi, Giancarlo; Link, Sebastian; Zdravkovic, Jelena (eds.). Conceptual Modeling. Lecture Notes in Computer Science. Vol. 14320. Cham: Springer Nature Switzerland. pp. 3–19. arXiv:2309.05918. doi:10.1007/978-3-031-47262-6_1. ISBN 978-3-031-47262-6. <a href="978-3-031-47262-6" target="_blank">978-3-031-47262-6</a> <a href="#fnref:28" class="footnote-back-ref">↩</a></li>
<li id="fn:29">Mitchell, Melanie; Krakauer, David C. (2023-03-28). "The debate over understanding in AI's large language models". Proceedings of the National Academy of Sciences. 120 (13): e2215907120. arXiv:2210.13966. Bibcode:2023PNAS..12015907M. doi:10.1073/pnas.2215907120. ISSN 0027-8424. PMC 10068812. PMID 36943882. <a href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10068812" target="_blank">https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10068812</a> <a href="#fnref:29" class="footnote-back-ref">↩</a></li>
<li id="fn:30">Saba, Walid S. (2023). "Stochastic LLMS do not Understand Language: Towards Symbolic, Explainable and Ontologically Based LLMS". In Almeida, João Paulo A.; Borbinha, José; Guizzardi, Giancarlo; Link, Sebastian; Zdravkovic, Jelena (eds.). Conceptual Modeling. Lecture Notes in Computer Science. Vol. 14320. Cham: Springer Nature Switzerland. pp. 3–19. arXiv:2309.05918. doi:10.1007/978-3-031-47262-6_1. ISBN 978-3-031-47262-6. <a href="978-3-031-47262-6" target="_blank">978-3-031-47262-6</a> <a href="#fnref:30" class="footnote-back-ref">↩</a></li>
<li id="fn:31">Saba, Walid S. (2023). "Stochastic LLMS do not Understand Language: Towards Symbolic, Explainable and Ontologically Based LLMS". In Almeida, João Paulo A.; Borbinha, José; Guizzardi, Giancarlo; Link, Sebastian; Zdravkovic, Jelena (eds.). Conceptual Modeling. Lecture Notes in Computer Science. Vol. 14320. Cham: Springer Nature Switzerland. pp. 3–19. arXiv:2309.05918. doi:10.1007/978-3-031-47262-6_1. ISBN 978-3-031-47262-6. <a href="978-3-031-47262-6" target="_blank">978-3-031-47262-6</a> <a href="#fnref:31" class="footnote-back-ref">↩</a></li>
<li id="fn:32">Saba, Walid S. (2023). "Stochastic LLMS do not Understand Language: Towards Symbolic, Explainable and Ontologically Based LLMS". In Almeida, João Paulo A.; Borbinha, José; Guizzardi, Giancarlo; Link, Sebastian; Zdravkovic, Jelena (eds.). Conceptual Modeling. Lecture Notes in Computer Science. Vol. 14320. Cham: Springer Nature Switzerland. pp. 3–19. arXiv:2309.05918. doi:10.1007/978-3-031-47262-6_1. ISBN 978-3-031-47262-6. <a href="978-3-031-47262-6" target="_blank">978-3-031-47262-6</a> <a href="#fnref:32" class="footnote-back-ref">↩</a></li>
<li id="fn:33">Fayyad, Usama M. (2023-05-26). "From Stochastic Parrots to Intelligent Assistants—The Secrets of Data and Human Interventions". IEEE Intelligent Systems. 38 (3): 63–67. doi:10.1109/MIS.2023.3268723. ISSN 1541-1672. <a href="https://ieeexplore.ieee.org/document/10148666" target="_blank">https://ieeexplore.ieee.org/document/10148666</a> <a href="#fnref:33" class="footnote-back-ref">↩</a></li>
<li id="fn:34">Bender, Emily M.; Gebru, Timnit; McMillan-Major, Angelina; Mitchell, Margaret (2021). "On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?". Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency. doi:10.1145/3442188.3445922. <a href="/wiki/Doi_(identifier)" target="_blank">/wiki/Doi_(identifier)</a> <a href="#fnref:34" class="footnote-back-ref">↩</a></li>
<li id="fn:35">Mitchell, Melanie; Krakauer, David C. (2023-03-28). "The debate over understanding in AI's large language models". Proceedings of the National Academy of Sciences. 120 (13): e2215907120. arXiv:2210.13966. Bibcode:2023PNAS..12015907M. doi:10.1073/pnas.2215907120. ISSN 0027-8424. PMC 10068812. PMID 36943882. <a href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10068812" target="_blank">https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10068812</a> <a href="#fnref:35" class="footnote-back-ref">↩</a></li>
<li id="fn:36">Wang, Alex; Pruksachatkun, Yada; Nangia, Nikita; Singh, Amanpreet; Michael, Julian; Hill, Felix; Levy, Omer; Bowman, Samuel R. (2019-05-02). "SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems". arXiv:1905.00537 [cs.CL]. <a href="/wiki/ArXiv_(identifier)" target="_blank">/wiki/ArXiv_(identifier)</a> <a href="#fnref:36" class="footnote-back-ref">↩</a></li>
<li id="fn:37">"GPT-4 Technical Report". 2023. arXiv:2303.08774. A bot will complete this citation soon. Click here to jump the queue <a href="/wiki/ArXiv_(identifier)" target="_blank">/wiki/ArXiv_(identifier)</a> <a href="#fnref:37" class="footnote-back-ref">↩</a></li>
<li id="fn:38">Mitchell, Melanie; Krakauer, David C. (2023-03-28). "The debate over understanding in AI's large language models". Proceedings of the National Academy of Sciences. 120 (13): e2215907120. arXiv:2210.13966. Bibcode:2023PNAS..12015907M. doi:10.1073/pnas.2215907120. ISSN 0027-8424. PMC 10068812. PMID 36943882. <a href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10068812" target="_blank">https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10068812</a> <a href="#fnref:38" class="footnote-back-ref">↩</a></li>
<li id="fn:39">Pelley, Scott (8 October 2023). ""Godfather of Artificial Intelligence" Geoffrey Hinton on the promise, risks of advanced AI". CBS News. Retrieved 2 July 2025. <a href="https://www.cbsnews.com/news/geoffrey-hinton-ai-dangers-60-minutes-transcript/" target="_blank">https://www.cbsnews.com/news/geoffrey-hinton-ai-dangers-60-minutes-transcript/</a> <a href="#fnref:39" class="footnote-back-ref">↩</a></li>
<li id="fn:40">60 Minutes (2023-10-09). "Godfather of AI" Geoffrey Hinton: The 60 Minutes Interview. Retrieved 2025-07-02 – via YouTube.{{cite AV media}}: CS1 maint: numeric names: authors list (link) <a href="https://www.youtube.com/watch?v=qrvK_KuIeJk" target="_blank">https://www.youtube.com/watch?v=qrvK_KuIeJk</a> <a href="#fnref:40" class="footnote-back-ref">↩</a></li>
<li id="fn:41">Morris, Ian (24 March 2024). "Inside the secret meeting where mathematicians struggled to outsmart AI". Scientific American. Retrieved 2 July 2025. <a href="https://www.scientificamerican.com/article/inside-the-secret-meeting-where-mathematicians-struggled-to-outsmart-ai/" target="_blank">https://www.scientificamerican.com/article/inside-the-secret-meeting-where-mathematicians-struggled-to-outsmart-ai/</a> <a href="#fnref:41" class="footnote-back-ref">↩</a></li>
<li id="fn:42">"GPT-4 Technical Report". 2023. arXiv:2303.08774. A bot will complete this citation soon. Click here to jump the queue <a href="/wiki/ArXiv_(identifier)" target="_blank">/wiki/ArXiv_(identifier)</a> <a href="#fnref:42" class="footnote-back-ref">↩</a></li>
<li id="fn:43">Li, Kenneth; Hopkins, Aspen K.; Bau, David; Viégas, Fernanda; Pfister, Hanspeter; Wattenberg, Martin (2023-02-27), Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task, arXiv:2210.13382 <a href="/wiki/ArXiv_(identifier)" target="_blank">/wiki/ArXiv_(identifier)</a> <a href="#fnref:43" class="footnote-back-ref">↩</a></li>
<li id="fn:44">Li, Kenneth (2023-01-21). "Large Language Model: world models or surface statistics?". The Gradient. Retrieved 2024-04-04. <a href="https://thegradient.pub/othello/" target="_blank">https://thegradient.pub/othello/</a> <a href="#fnref:44" class="footnote-back-ref">↩</a></li>
<li id="fn:45">Jin, Charles; Rinard, Martin (2023-05-24), Evidence of Meaning in Language Models Trained on Programs, arXiv:2305.11169 <a href="/wiki/ArXiv_(identifier)" target="_blank">/wiki/ArXiv_(identifier)</a> <a href="#fnref:45" class="footnote-back-ref">↩</a></li>
<li id="fn:46">Schreiner, Maximilian (2023-08-11). "Grokking in machine learning: When Stochastic Parrots build models". the decoder. Retrieved 2024-05-25. <a href="https://the-decoder.com/grokking-in-machine-learning-when-stochastic-parrots-build-models/" target="_blank">https://the-decoder.com/grokking-in-machine-learning-when-stochastic-parrots-build-models/</a> <a href="#fnref:46" class="footnote-back-ref">↩</a></li>
<li id="fn:47">Choudhury, Sagnik Ray; Rogers, Anna; Augenstein, Isabelle (2022-09-15), Machine Reading, Fast and Slow: When Do Models "Understand" Language?, arXiv:2209.07430 <a href="/wiki/ArXiv_(identifier)" target="_blank">/wiki/ArXiv_(identifier)</a> <a href="#fnref:47" class="footnote-back-ref">↩</a></li>
<li id="fn:48">Geirhos, Robert; Jacobsen, Jörn-Henrik; Michaelis, Claudio; Zemel, Richard; Brendel, Wieland; Bethge, Matthias; Wichmann, Felix A. (2020-11-10). "Shortcut learning in deep neural networks". Nature Machine Intelligence. 2 (11): 665–673. arXiv:2004.07780. doi:10.1038/s42256-020-00257-z. ISSN 2522-5839. <a href="https://www.nature.com/articles/s42256-020-00257-z" target="_blank">https://www.nature.com/articles/s42256-020-00257-z</a> <a href="#fnref:48" class="footnote-back-ref">↩</a></li>
<li id="fn:49">Mitchell, Melanie; Krakauer, David C. (2023-03-28). "The debate over understanding in AI's large language models". Proceedings of the National Academy of Sciences. 120 (13): e2215907120. arXiv:2210.13966. Bibcode:2023PNAS..12015907M. doi:10.1073/pnas.2215907120. ISSN 0027-8424. PMC 10068812. PMID 36943882. <a href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10068812" target="_blank">https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10068812</a> <a href="#fnref:49" class="footnote-back-ref">↩</a></li>
<li id="fn:50">Niven, Timothy; Kao, Hung-Yu (2019-09-16), Probing Neural Network Comprehension of Natural Language Arguments, arXiv:1907.07355 <a href="/wiki/ArXiv_(identifier)" target="_blank">/wiki/ArXiv_(identifier)</a> <a href="#fnref:50" class="footnote-back-ref">↩</a></li>
<li id="fn:51">Mitchell, Melanie; Krakauer, David C. (2023-03-28). "The debate over understanding in AI's large language models". Proceedings of the National Academy of Sciences. 120 (13): e2215907120. arXiv:2210.13966. Bibcode:2023PNAS..12015907M. doi:10.1073/pnas.2215907120. ISSN 0027-8424. PMC 10068812. PMID 36943882. <a href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10068812" target="_blank">https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10068812</a> <a href="#fnref:51" class="footnote-back-ref">↩</a></li>
<li id="fn:52">Niven, Timothy; Kao, Hung-Yu (2019-09-16), Probing Neural Network Comprehension of Natural Language Arguments, arXiv:1907.07355 <a href="/wiki/ArXiv_(identifier)" target="_blank">/wiki/ArXiv_(identifier)</a> <a href="#fnref:52" class="footnote-back-ref">↩</a></li>
</ol>

Stochastic parrot open-in-new

Stochastic parrot