Up until the 1980s, most natural language processing systems were based on complex sets of hand-written rules. Starting in the late 1980s, however, there was a revolution in natural language processing with the introduction of machine learning algorithms for language processing. This was due to both the steady increase in computational power (see Moore's law) and the gradual lessening of the dominance of Chomskyan theories of linguistics (e.g. transformational grammar), whose theoretical underpinnings discouraged the sort of corpus linguistics that underlies the machine-learning approach to language processing.
Symbolic approach, i.e., the hand-coding of a set of rules for manipulating symbols, coupled with a dictionary lookup, was historically the first approach used both by AI in general and by NLP in particular: such as by writing grammars or devising heuristic rules for stemming.
Intermediate tasks (e.g., part-of-speech tagging and dependency parsing) are not needed anymore.
The following is a list of some of the most commonly researched tasks in natural language processing. Some of these tasks have direct real-world applications, while others more commonly serve as subtasks that are used to aid in solving larger tasks.
Though natural language processing tasks are closely intertwined, they can be subdivided into categories for convenience. A coarse division is given below.
Based on long-standing trends in the field, it is possible to extrapolate future directions of NLP. As of 2020, three trends among the topics of the long-standing series of CoNLL Shared Tasks can be observed:
Most higher-level NLP applications involve aspects that emulate intelligent behaviour and apparent comprehension of natural language. More broadly speaking, the technical operationalization of increasingly advanced aspects of cognitive behaviour represents one of the developmental trajectories of NLP (see trends among CoNLL shared tasks above).
Ties with cognitive linguistics are part of the historical heritage of NLP, but they have been less frequently addressed since the statistical turn during the 1990s. Nevertheless, approaches to develop cognitive models towards technically operationalizable frameworks have been pursued in the context of various frameworks, e.g., of cognitive grammar, functional grammar, construction grammar, computational psycholinguistics and cognitive neuroscience (e.g., ACT-R), however, with limited uptake in mainstream NLP (as measured by presence on major conferences of the ACL). More recently, ideas of cognitive NLP have been revived as an approach to achieve explainability, e.g., under the notion of "cognitive AI". Likewise, ideas of cognitive NLP are inherent to neural models multimodal NLP (although rarely made explicit) and developments in artificial intelligence, specifically tools and technologies using large language model approaches and new directions in artificial general intelligence based on the free energy principle by British neuroscientist and theoretician at University College London Karl J. Friston.
"NLP". https://cs.stanford.edu/people/eroberts/courses/soco/projects/2004-05/nlp/overview_history.html
Hutchins, J. (2005). "The history of machine translation in a nutshell" (PDF).[self-published source] http://www.hutchinsweb.me.uk/Nutshell-2005.pdf
"ALPAC: the (in)famous report", John Hutchins, MT News International, no. 14, June 1996, pp. 9–12.
Crevier 1993, pp. 146–148 harvnb error: no target: CITEREFCrevier1993 (help), see also Buchanan 2005, p. 56 harvnb error: no target: CITEREFBuchanan2005 (help): "Early programs were necessarily limited in scope by the size and speed of memory"
Koskenniemi, Kimmo (1983), Two-level morphology: A general computational model of word-form recognition and production (PDF), Department of General Linguistics, University of Helsinki /wiki/Kimmo_Koskenniemi
Joshi, A. K., & Weinstein, S. (1981, August). Control of Inference: Role of Some Aspects of Discourse Structure-Centering. In IJCAI (pp. 385–387). https://www.ijcai.org/Proceedings/81-1/Papers/071.pdf
Guida, G.; Mauri, G. (July 1986). "Evaluation of natural language processing systems: Issues and approaches". Proceedings of the IEEE. 74 (7): 1026–1035. doi:10.1109/PROC.1986.13580. ISSN 1558-2256. S2CID 30688575. /wiki/Doi_(identifier)
Chomskyan linguistics encourages the investigation of "corner cases" that stress the limits of its theoretical models (comparable to pathological phenomena in mathematics), typically created using thought experiments, rather than the systematic investigation of typical phenomena that occur in real-world data, as is the case in corpus linguistics. The creation and use of such corpora of real-world data is a fundamental part of machine-learning algorithms for natural language processing. In addition, theoretical underpinnings of Chomskyan linguistics such as the so-called "poverty of the stimulus" argument entail that general learning algorithms, as are typically used in machine learning, cannot be successful in language processing. As a result, the Chomskyan paradigm discouraged the application of such models to language processing. /wiki/Corner_case
Bengio, Yoshua; Ducharme, Réjean; Vincent, Pascal; Janvin, Christian (March 1, 2003). "A neural probabilistic language model". The Journal of Machine Learning Research. 3: 1137–1155 – via ACM Digital Library. https://dl.acm.org/doi/10.5555/944919.944966
Mikolov, Tomáš; Karafiát, Martin; Burget, Lukáš; Černocký, Jan; Khudanpur, Sanjeev (26 September 2010). "Recurrent neural network based language model" (PDF). Interspeech 2010. pp. 1045–1048. doi:10.21437/Interspeech.2010-343. S2CID 17048224. {{cite book}}: |journal= ignored (help) https://gwern.net/doc/ai/nn/rnn/2010-mikolov.pdf
Goldberg, Yoav (2016). "A Primer on Neural Network Models for Natural Language Processing". Journal of Artificial Intelligence Research. 57: 345–420. arXiv:1807.10854. doi:10.1613/jair.4992. S2CID 8273530. /wiki/ArXiv_(identifier)
Goodfellow, Ian; Bengio, Yoshua; Courville, Aaron (2016). Deep Learning. MIT Press. http://www.deeplearningbook.org/
Jozefowicz, Rafal; Vinyals, Oriol; Schuster, Mike; Shazeer, Noam; Wu, Yonghui (2016). Exploring the Limits of Language Modeling. arXiv:1602.02410. Bibcode:2016arXiv160202410J. /wiki/ArXiv_(identifier)
Choe, Do Kook; Charniak, Eugene. "Parsing as Language Modeling". Emnlp 2016. Archived from the original on 2018-10-23. Retrieved 2018-10-22. https://web.archive.org/web/20181023034804/https://aclanthology.coli.uni-saarland.de/papers/D16-1257/d16-1257
Vinyals, Oriol; et al. (2014). "Grammar as a Foreign Language" (PDF). Nips2015. arXiv:1412.7449. Bibcode:2014arXiv1412.7449V. https://papers.nips.cc/paper/5635-grammar-as-a-foreign-language.pdf
Turchin, Alexander; Florez Builes, Luisa F. (2021-03-19). "Using Natural Language Processing to Measure and Improve Quality of Diabetes Care: A Systematic Review". Journal of Diabetes Science and Technology. 15 (3): 553–560. doi:10.1177/19322968211000831. ISSN 1932-2968. PMC 8120048. PMID 33736486. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8120048
Lee, Jennifer; Yang, Samuel; Holland-Hall, Cynthia; Sezgin, Emre; Gill, Manjot; Linwood, Simon; Huang, Yungui; Hoffman, Jeffrey (2022-06-10). "Prevalence of Sensitive Terms in Clinical Notes Using Natural Language Processing Techniques: Observational Study". JMIR Medical Informatics. 10 (6): e38482. doi:10.2196/38482. ISSN 2291-9694. PMC 9233261. PMID 35687381. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9233261
Winograd, Terry (1971). Procedures as a Representation for Data in a Computer Program for Understanding Natural Language (Thesis). http://hci.stanford.edu/winograd/shrdlu/
Schank, Roger C.; Abelson, Robert P. (1977). Scripts, Plans, Goals, and Understanding: An Inquiry Into Human Knowledge Structures. Hillsdale: Erlbaum. ISBN 0-470-99033-3. 0-470-99033-3
Mark Johnson. How the statistical revolution changes (computational) linguistics. Proceedings of the EACL 2009 Workshop on the Interaction between Linguistics and Computational Linguistics. http://www.aclweb.org/anthology/W09-0103
Philip Resnik. Four revolutions. Language Log, February 5, 2011. http://languagelog.ldc.upenn.edu/nll/?p=2946
Socher, Richard. "Deep Learning For NLP-ACL 2012 Tutorial". www.socher.org. Retrieved 2020-08-17. This was an early Deep Learning tutorial at the ACL 2012 and met with both interest and (at the time) skepticism by most participants. Until then, neural learning was basically rejected because of its lack of statistical interpretability. Until 2015, deep learning had evolved into the major framework of NLP. [Link is broken, try http://web.stanford.edu/class/cs224n/] https://www.socher.org/index.php/Main/DeepLearningForNLP-ACL2012Tutorial
Segev, Elad (2022). Semantic Network Analysis in Social Sciences. London: Routledge. ISBN 9780367636524. Archived from the original on 5 December 2021. Retrieved 5 December 2021. 9780367636524
Yi, Chucai; Tian, Yingli (2012), "Assistive Text Reading from Complex Background for Blind Persons", Camera-Based Document Analysis and Recognition, Lecture Notes in Computer Science, vol. 7139, Springer Berlin Heidelberg, pp. 15–28, CiteSeerX 10.1.1.668.869, doi:10.1007/978-3-642-29364-1_2, ISBN 9783642293634 9783642293634
"Natural Language Processing (NLP) - A Complete Guide". www.deeplearning.ai. 2023-01-11. Retrieved 2024-05-05. https://www.deeplearning.ai/resources/natural-language-processing/
"What is Natural Language Processing? Intro to NLP in Machine Learning". GyanSetu!. 2020-12-06. Retrieved 2021-01-09. https://www.gyansetu.in/what-is-natural-language-processing/
Kishorjit, N.; Vidya, Raj RK.; Nirmal, Y.; Sivaji, B. (2012). "Manipuri Morpheme Identification" (PDF). Proceedings of the 3rd Workshop on South and Southeast Asian Natural Language Processing (SANLP). COLING 2012, Mumbai, December 2012: 95–108.{{cite journal}}: CS1 maint: location (link) http://aclweb.org/anthology//W/W12/W12-5008.pdf
Klein, Dan; Manning, Christopher D. (2002). "Natural language grammar induction using a constituent-context model" (PDF). Advances in Neural Information Processing Systems. http://papers.nips.cc/paper/1945-natural-language-grammar-induction-using-a-constituent-context-model.pdf
Kariampuzha, William; Alyea, Gioconda; Qu, Sue; Sanjak, Jaleal; Mathé, Ewy; Sid, Eric; Chatelaine, Haley; Yadaw, Arjun; Xu, Yanji; Zhu, Qian (2023). "Precision information extraction for rare disease epidemiology at scale". Journal of Translational Medicine. 21 (1): 157. doi:10.1186/s12967-023-04011-y. PMC 9972634. PMID 36855134. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9972634
"Natural Language Processing (NLP) - A Complete Guide". www.deeplearning.ai. 2023-01-11. Retrieved 2024-05-05. https://www.deeplearning.ai/resources/natural-language-processing/
PASCAL Recognizing Textual Entailment Challenge (RTE-7) https://tac.nist.gov//2011/RTE/ https://tac.nist.gov//2011/RTE/
Lippi, Marco; Torroni, Paolo (2016-04-20). "Argumentation Mining: State of the Art and Emerging Trends". ACM Transactions on Internet Technology. 16 (2): 1–25. doi:10.1145/2850417. hdl:11585/523460. ISSN 1533-5399. S2CID 9561587. https://dl.acm.org/doi/10.1145/2850417
"Argument Mining – IJCAI2016 Tutorial". www.i3s.unice.fr. Retrieved 2021-03-09. https://www.i3s.unice.fr/~villata/tutorialIJCAI2016.html
"NLP Approaches to Computational Argumentation – ACL 2016, Berlin". Retrieved 2021-03-09. http://acl2016tutorial.arg.tech/
Administration. "Centre for Language Technology (CLT)". Macquarie University. Retrieved 2021-01-11. https://www.mq.edu.au/research/research-centres-groups-and-facilities/innovative-technologies/centres/centre-for-language-technology-clt
"Shared Task: Grammatical Error Correction". www.comp.nus.edu.sg. Retrieved 2021-01-11. https://www.comp.nus.edu.sg/~nlp/conll13st.html
"Shared Task: Grammatical Error Correction". www.comp.nus.edu.sg. Retrieved 2021-01-11. https://www.comp.nus.edu.sg/~nlp/conll14st.html
Duan, Yucong; Cruz, Christophe (2011). "Formalizing Semantic of Natural Language through Conceptualization from Existence". International Journal of Innovation, Management and Technology. 2 (1): 37–42. Archived from the original on 2011-10-09. https://web.archive.org/web/20111009135952/http://www.ijimt.org/abstract/100-E00187.htm
"U B U W E B :: Racter". www.ubu.com. Retrieved 2020-08-17. http://www.ubu.com/historical/racter/index.html
Writer, Beta (2019). Lithium-Ion Batteries. doi:10.1007/978-3-030-16800-1. ISBN 978-3-030-16799-8. S2CID 155818532. 978-3-030-16799-8
"Document Understanding AI on Google Cloud (Cloud Next '19) – YouTube". www.youtube.com. 11 April 2019. Archived from the original on 2021-10-30. Retrieved 2021-01-11. https://ghostarchive.org/varchive/youtube/20211030/7dtl650D0y0
Robertson, Adi (2022-04-06). "OpenAI's DALL-E AI image generator can now edit pictures, too". The Verge. Retrieved 2022-06-07. https://www.theverge.com/2022/4/6/23012123/openai-clip-dalle-2-ai-text-to-image-generator-testing
"The Stanford Natural Language Processing Group". nlp.stanford.edu. Retrieved 2022-06-07. https://nlp.stanford.edu/projects/text2scene.shtml
Coyne, Bob; Sproat, Richard (2001-08-01). "WordsEye". Proceedings of the 28th annual conference on Computer graphics and interactive techniques. SIGGRAPH '01. New York, NY, USA: Association for Computing Machinery. pp. 487–496. doi:10.1145/383259.383316. ISBN 978-1-58113-374-5. S2CID 3842372. 978-1-58113-374-5
"Google announces AI advances in text-to-video, language translation, more". VentureBeat. 2022-11-02. Retrieved 2022-11-09. https://venturebeat.com/ai/google-announces-ai-advances-in-text-to-video-language-translation-more/
Vincent, James (2022-09-29). "Meta's new text-to-video AI generator is like DALL-E for video". The Verge. Retrieved 2022-11-09. https://www.theverge.com/2022/9/29/23378210/meta-text-to-video-ai-generation-make-a-video-model-dall-e
"Previous shared tasks | CoNLL". www.conll.org. Retrieved 2021-01-11. https://www.conll.org/previous-tasks
"Cognition". Lexico. Oxford University Press and Dictionary.com. Archived from the original on July 15, 2020. Retrieved 6 May 2020. https://web.archive.org/web/20200715113427/https://www.lexico.com/definition/cognition
"Ask the Cognitive Scientist". American Federation of Teachers. 8 August 2014. Cognitive science is an interdisciplinary field of researchers from Linguistics, psychology, neuroscience, philosophy, computer science, and anthropology that seek to understand the mind. http://www.aft.org/newspubs/periodicals/ae/summer2002/willingham.cfm
Robinson, Peter (2008). Handbook of Cognitive Linguistics and Second Language Acquisition. Routledge. pp. 3–8. ISBN 978-0-805-85352-0. 978-0-805-85352-0
Lakoff, George (1999). Philosophy in the Flesh: The Embodied Mind and Its Challenge to Western Philosophy; Appendix: The Neural Theory of Language Paradigm. New York Basic Books. pp. 569–583. ISBN 978-0-465-05674-3. 978-0-465-05674-3
Strauss, Claudia (1999). A Cognitive Theory of Cultural Meaning. Cambridge University Press. pp. 156–164. ISBN 978-0-521-59541-4. 978-0-521-59541-4
US patent 9269353
https://worldwide.espacenet.com/textdoc?DB=EPODOC&IDX=US9269353
"Universal Conceptual Cognitive Annotation (UCCA)". Universal Conceptual Cognitive Annotation (UCCA). Retrieved 2021-01-11. https://universalconceptualcognitiveannotation.github.io/
Rodríguez, F. C., & Mairal-Usón, R. (2016). Building an RRG computational grammar. Onomazein, (34), 86–117. https://www.redalyc.org/pdf/1345/134549291020.pdf
"Fluid Construction Grammar – A fully operational processing system for construction grammars". Retrieved 2021-01-11. https://www.fcg-net.org/
"ACL Member Portal | The Association for Computational Linguistics Member Portal". www.aclweb.org. Retrieved 2021-01-11. https://www.aclweb.org/portal/
"Chunks and Rules". W3C. Retrieved 2021-01-11. https://www.w3.org/Data/demos/chunks/chunks.html
Socher, Richard; Karpathy, Andrej; Le, Quoc V.; Manning, Christopher D.; Ng, Andrew Y. (2014). "Grounded Compositional Semantics for Finding and Describing Images with Sentences". Transactions of the Association for Computational Linguistics. 2: 207–218. doi:10.1162/tacl_a_00177. S2CID 2317858. https://doi.org/10.1162%2Ftacl_a_00177
Dasgupta, Ishita; Lampinen, Andrew K.; Chan, Stephanie C. Y.; Creswell, Antonia; Kumaran, Dharshan; McClelland, James L.; Hill, Felix (2022). "Language models show human-like content effects on reasoning, Dasgupta, Lampinen et al". arXiv:2207.07051 [cs.CL]. /wiki/ArXiv_(identifier)
Friston, Karl J. (2022). Active Inference: The Free Energy Principle in Mind, Brain, and Behavior; Chapter 4 The Generative Models of Active Inference. The MIT Press. ISBN 978-0-262-36997-8. 978-0-262-36997-8