Menu
Home
People
Places
Arts
History
Plants & Animals
Science
Life & Culture
Technology
Reference.org
SemEval
open-in-new
<h2 id="history">History</h2> <h3>Early evaluation of algorithms for word sense disambiguation</h3> <p>From the earliest days, assessing the quality of word sense disambiguation algorithms had been primarily a matter of <a href="/facts/Natural_language_processing/1hjMKsSN">intrinsic evaluation</a>, and “almost no attempts had been made to evaluate embedded WSD components”.<a class="footnote-ref" id="fnref:3" href="#fn:3"><sup>3</sup></a> Only very recently (2006) had extrinsic evaluations begun to provide some evidence for the value of WSD in end-user applications.<a class="footnote-ref" id="fnref:4" href="#fn:4"><sup>4</sup></a> Until 1990 or so, discussions of the sense disambiguation task focused mainly on illustrative examples rather than comprehensive evaluation. The early 1990s saw the beginnings of more systematic and rigorous intrinsic evaluations, including more formal experimentation on small sets of ambiguous words.<a class="footnote-ref" id="fnref:5" href="#fn:5"><sup>5</sup></a> </p> <h3>Senseval to SemEval</h3> <p>In April 1997, Martha Palmer and Marc Light organized a workshop entitled <i>Tagging with Lexical Semantics: Why, What, and How?</i> in conjunction with the Conference on Applied Natural Language Processing.<a class="footnote-ref" id="fnref:6" href="#fn:6"><sup>6</sup></a> At the time, there was a clear recognition that manually annotated <a href="/facts/Text_corpus/bYeEQQdh">corpora</a> had revolutionized other areas of NLP, such as <a href="/facts/Part-of-speech_tagging/BYAPmH3r">part-of-speech tagging</a> and <a href="/facts/Parsing/nRKB01S3">parsing</a>, and that corpus-driven approaches had the potential to revolutionize automatic semantic analysis as well.<a class="footnote-ref" id="fnref:7" href="#fn:7"><sup>7</sup></a> Kilgarriff recalled that there was "a high degree of consensus that the field needed evaluation", and several practical proposals by Resnik and Yarowsky kicked off a discussion that led to the creation of the Senseval evaluation exercises.<a class="footnote-ref" id="fnref:8" href="#fn:8"><sup>8</sup></a><a class="footnote-ref" id="fnref:9" href="#fn:9"><sup>9</sup></a><a class="footnote-ref" id="fnref:10" href="#fn:10"><sup>10</sup></a> </p> <h3>SemEval's 3, 2 or 1 year(s) cycle</h3> <p>After SemEval-2010, many participants feel that the 3-year cycle is a long wait. Many other shared tasks such as Conference on Natural Language Learning (CoNLL) and Recognizing Textual Entailments (RTE) run annually. For this reason, the SemEval coordinators gave the opportunity for task organizers to choose between a 2-year or a 3-year cycle.<a class="footnote-ref" id="fnref:11" href="#fn:11"><sup>11</sup></a> The SemEval community favored the 3-year cycle. Although the votes within the SemEval community favored a 3-year cycle, organizers and coordinators had settled to split the SemEval task into 2 evaluation workshops. This was triggered by the introduction of the new *SEM conference. The SemEval organizers thought it would be appropriate to associate our event with the *SEM conference and collocate the SemEval workshop with the *SEM conference. The organizers got very positive responses (from the task coordinators/organizers and participants) about the association with the yearly *SEM, and 8 tasks were willing to switch to 2012. Thus was born SemEval-2012 and SemEval-2013. The current plan is to switch to a yearly SemEval schedule to associate it with the *SEM conference but not every task needs to run every year.<a class="footnote-ref" id="fnref:12" href="#fn:12"><sup>12</sup></a> </p> <h4>List of Senseval and SemEval Workshops</h4> <ul><li><i>Senseval-1</i> took place in the summer of 1998 for English, French, and Italian, culminating in a workshop held at Herstmonceux Castle, Sussex, England on September 2–4.</li> <li><i>Senseval-2</i> took place in the summer of 2001, and was followed by a workshop held in July 2001 in Toulouse, in conjunction with ACL 2001. Senseval-2 included tasks for <a href="/facts/Basque_language/rEMpH97p">Basque</a>, <a href="/facts/Chinese_language/771ruUY1">Chinese</a>, <a href="/facts/Czech_language/ySvEftdh">Czech</a>, <a href="/facts/Danish_language/sTcGXTMJ">Danish</a>, <a href="/facts/Dutch_language/yIB5lbsx">Dutch</a>, <a href="/facts/English_language/tpkxGt7b">English</a>, <a href="/facts/Estonian_language/ywCJos4e">Estonian</a><a href="/facts/%2c/mcW9dcGT">,</a> <a href="/facts/Italian_language/YXsmvlQG">Italian</a>, <a href="/facts/Japanese_language/EEv485a5">Japanese</a>, <a href="/facts/Korean_language/dmNnRxAA">Korean</a>, <a href="/facts/Spanish_language/95BgYNfv">Spanish</a> and <a href="/facts/Swedish_language/1DkSIXkH">Swedish</a>.</li> <li><i>Senseval-3</i> took place in March–April 2004, followed by a workshop held in July 2004 in Barcelona, in conjunction with ACL 2004. Senseval-3 included 14 different tasks for core word sense disambiguation, as well as identification of semantic roles, multilingual annotations, logic forms, subcategorization acquisition.</li> <li><i>SemEval-2007 (Senseval-4)</i> took place in 2007, followed by a workshop held in conjunction with ACL in Prague. SemEval-2007 included 18 different tasks targeting the evaluation of systems for the semantic analysis of text. A special issue of <i>Language Resources and Evaluation</i> is devoted to the result.<a class="footnote-ref" id="fnref:13" href="#fn:13"><sup>13</sup></a></li> <li><i>SemEval-2010</i> took place in 2010, followed by a workshop held in conjunction with ACL in Uppsala. SemEval-2010 included 18 different tasks targeting the evaluation of semantic analysis systems.</li> <li><i>SemEval-2012</i> took place in 2012; it was associated with the new *SEM, First Joint Conference on Lexical and Computational Semantics, and co-located with NAACL, Montreal, Canada. SemEval-2012 included 8 different tasks targeting at evaluating computational semantic systems. However, there was no WSD task involved in SemEval-2012, the WSD related tasks were scheduled in the upcoming SemEval-2013.</li> <li><i>SemEval-2013</i> was associated with NAACL 2013, North American Association of Computational Linguistics, Georgia, USA and took place in 2013. It included 13 different tasks targeting at evaluating computational semantic systems.</li> <li><i>SemEval-2014</i> took place in 2014. It was co-located with COLING 2014, 25th International Conference on Computational Linguistics and *SEM 2014, Second Joint Conference on Lexical and Computational Semantics, Dublin, Ireland. There were 10 different tasks in SemEval-2014 evaluating various computational semantic systems.</li> <li><i>SemEval-2015</i> took place in 2015. It was co-located with NAACL-HLT 2015, 2015 Conference of the North American Chapter of the Association for Computational Linguistics – Human Language Technologies and *SEM 2015, Third Joint Conference on Lexical and Computational Semantics, Denver, USA. There were 17 different tasks in SemEval-2015 evaluating various computational semantic systems.</li></ul> <h2 id="semeval-workshop-framework">SemEval Workshop framework</h2> <p>The framework of the SemEval/Senseval evaluation workshops emulates the <a href="/facts/Message_Understanding_Conference/zprq5Lx6">Message Understanding Conferences</a> (MUCs) and other evaluation workshops ran by ARPA (Advanced Research Projects Agency, renamed the <a href="/facts/DARPA/Uf9sarhT">Defense Advanced Research Projects Agency (DARPA)</a>). </p> <p>Stages of SemEval/Senseval evaluation workshops<a class="footnote-ref" id="fnref:14" href="#fn:14"><sup>14</sup></a> </p> <ol><li>Firstly, all likely participants were invited to express their interest and participate in the exercise design.</li> <li>A timetable towards a final workshop was worked out.</li> <li>A plan for selecting evaluation materials was agreed.</li> <li><a href="/facts/Natural_language_processing/1hjMKsSN">'Gold standards'</a> for the individual tasks were acquired, often human annotators were considered as a gold standard to measure <a href="/facts/Precision_and_recall/Jnmmpl8H">precision and recall</a> scores of computer systems. These 'gold standards' are what the computational systems strive towards. In WSD tasks, human annotators were set on the task of generating a set of correct WSD answers (i.e. the correct sense for a given word in a given context)</li> <li>The gold standard materials, without answers, were released to participants, who then had a short time to run their programs over them and return their sets of answers to the organizers.</li> <li>The organizers then scored the answers and the scores were announced and discussed at a workshop.</li></ol> <h2 id="semantic-evaluation-tasks">Semantic evaluation tasks</h2> <p>Senseval-1 & Senseval-2 focused on evaluation WSD systems on major languages that were available corpus and computerized dictionary. Senseval-3 looked beyond the <a href="/facts/Lexeme/CtHGorv4">lexemes</a> and started to evaluate systems that looked into wider areas of semantics, such as Semantic Roles (technically known as <a href="/facts/Theta_roles/VbyEGF35">Theta roles</a> in formal semantics), <a href="/facts/Logic_Form/RWQGhnjJ">Logic Form</a> Transformation (commonly semantics of phrases, clauses or sentences were represented in <a href="/facts/First-order_logic/lcISqWIg">first-order logic forms</a>) and Senseval-3 explored performances of semantics analysis on <a href="/facts/Machine_translation/DGF3NwuI">Machine translation</a>. </p><p>As the types of different computational semantic systems grew beyond the coverage of WSD, Senseval evolved into SemEval, where more aspects of computational semantic systems were evaluated. </p> <h3>Overview of Issues in Semantic Analysis</h3> <p>The SemEval exercises provide a mechanism for examining issues in semantic analysis of texts. The topics of interest fall short of the logical rigor that is found in formal computational semantics, attempting to identify and characterize the kinds of issues relevant to human understanding of language. The primary goal is to replicate human processing by means of computer systems. The tasks (shown below) are developed by individuals and groups to deal with identifiable issues, as they take on some concrete form. </p><p>The first major area in semantic analysis is the identification of the intended meaning at the word level (taken to include idiomatic expressions). This is word-sense disambiguation (a concept that is evolving away from the notion that words have discrete senses, but rather are characterized by the ways in which they are used, i.e., their contexts). The tasks in this area include lexical sample and all-word disambiguation, multi- and cross-lingual disambiguation, and lexical substitution. Given the difficulties of identifying word senses, other tasks relevant to this topic include word-sense induction, subcategorization acquisition, and evaluation of lexical resources. </p><p>The second major area in semantic analysis is the understanding of how different sentence and textual elements fit together. Tasks in this area include semantic role labeling, semantic relation analysis, and coreference resolution. Other tasks in this area look at more specialized issues of semantic analysis, such as temporal information processing, metonymy resolution, and sentiment analysis. The tasks in this area have many potential applications, such as information extraction, question answering, document summarization, machine translation, construction of thesauri and semantic networks, language modeling, paraphrasing, and recognizing textual entailment. In each of these potential applications, the contribution of the types of semantic analysis constitutes the most outstanding research issue. </p><p>For example, in the <a href="/facts/Word_sense_induction/3pcwGFTo">word sense induction</a> and <a href="/facts/Word-sense_disambiguation/Mfcw7gsu">disambiguation</a> task, there are three separate phases: </p> <ol><li>In the training phase, evaluation task participants were asked to use a training dataset to induce the sense inventories for a set of polysemous words. The training dataset consisting of a set of polysemous nouns/verbs and the sentence instances that they occurred in. No other resources were allowed other than morphological and syntactic Natural Language Processing components, such as <a href="/facts/Part-of-speech_tagging/BYAPmH3r">morphological analyzers</a>, <a href="/facts/Part-of-speech_tagging/BYAPmH3r">Part-Of-Speech taggers</a> and <a href="/facts/Parsing/nRKB01S3">syntactic parsers</a>.</li> <li>In the testing phase, participants were provided with a <a href="/facts/Test_set/V3mMQb2G">test set</a> for the disambiguating subtask using the induced sense inventory from the training phase.</li> <li>In the evaluation phase, answers of to the testing phase were evaluated in a <a href="/facts/Supervised_learning/JlNFNPFt">supervised</a> an <a href="/facts/Unsupervised_learning/8a6f61MC">unsupervised</a> framework.</li></ol> <p>The unsupervised evaluation for WSI considered two types of evaluation V Measure (Rosenberg and Hirschberg, 2007), and paired F-Score (Artiles et al., 2009). This evaluation follows the supervised evaluation of SemEval-2007 <a href="/facts/Word_Sense_Induction/3pcwGFTo">WSI</a> task (Agirre and Soroa, 2007) </p> <h3>Senseval and SemEval tasks overview</h3> <p>The tables below reflects the workshop growth from Senseval to SemEval and gives an overview of which area of computational semantics was evaluated throughout the Senseval/SemEval workshops. </p> <table><tbody><tr><th>Workshop</th><th>No. of Tasks</th><th>Areas of study</th><th>Languages of Data Evaluated</th></tr><tr><td>Senseval-1</td><td>3</td><td>Word Sense Disambiguation (WSD) - Lexical Sample WSD tasks</td><td>English, French, Italian</td></tr><tr><td>Senseval-2</td><td>12</td><td>Word Sense Disambiguation (WSD) - Lexical Sample, All Words, Translation WSD tasks</td><td>Czech, Dutch, English, Estonian, Basque, Chinese, Danish, English, Italian, Japanese, Korean, Spanish, Swedish</td></tr><tr><td>Senseval-3</td><td>16 (incl. 2 cancelled)</td><td>Logic Form Transformation, Machine Translation (MT) Evaluation, Semantic Role Labelling, WSD</td><td>Basque, Catalan, Chinese, English, Italian, Romanian, Spanish</td></tr><tr><td>SemEval2007</td><td>19 (incl. 1 cancelled)</td><td>Cross-lingual, Frame Extraction, Information Extraction, Lexical Substitution, Lexical Sample, Metonymy, Semantic Annotation, Semantic Relations, Semantic Role Labelling, Sentiment Analysis, Time Expression, WSD</td><td>Arabic, Catalan, Chinese, English, Spanish, Turkish</td></tr><tr><td>SemEval2010</td><td>18 (incl. 1 cancelled)</td><td>Coreference, Cross-lingual, Ellipsis, Information Extraction, Lexical Substitution, Metonymy, Noun Compounds, Parsing, Semantic Relations, Semantic Role Labeling, Sentiment Analysis, Textual Entailment, Time Expressions, WSD</td><td>Catalan, Chinese, Dutch, English, French, German, Italian, Japanese, Spanish</td></tr><tr><td>SemEval2012</td><td>8</td><td>Common Sense Reasoning, Lexical Simplification, Relational Similarity, Spatial Role Labelling, Semantic Dependency Parsing, Semantic and Textual Similarity</td><td>Chinese, English</td></tr><tr><td>SemEval2013</td><td>14</td><td>Temporal Annotation, Sentiment Analysis, Spatial Role Labeling, Noun Compounds, Phrasal Semantics, Textual Similarity, Response Analysis, Cross-lingual Textual Entailment, BioMedical Texts, Cross and Multilingual WSD, Word Sense Induction, and Lexical Sample</td><td>Catalan, French, German, English, Italian, Spanish</td></tr><tr><td>SemEval2014</td><td>10</td><td>Compositional Distributional Semantic, Grammar Induction for Spoken Dialogue Systems, Cross-Level Semantic Similarity, Sentiment Analysis, L2 Writing Assistant, Supervised Semantic Parsing, Clinical Text Analysis, Semantic Dependency Parsing, Sentiment Analysis in Twitter, Multilingual Semantic Textual Similarity</td><td>English, Spanish, French, German, Dutch,</td></tr><tr><td>SemEval2015</td><td>18 (incl. 1 cancelled)</td><td>Text Similarity and Question Answering, Time and Space, Sentiment, Word Sense Disambiguation and Induction, Learning Semantic Relations</td><td>English, Spanish, Arabic, Italian</td></tr><tr><td>SemEval2016</td><td>14</td><td>Textual Similarity and Question Answering, Sentiment Analysis, Semantic Parsing, Semantic Analysis, Semantic Taxonomy</td><td></td></tr><tr><td>SemEval2017</td><td>12<a class="footnote-ref" id="fnref:15" href="#fn:15"><sup>15</sup></a></td><td>Semantic comparison for words and texts, Detecting sentiment, humor, and truth and Parsing semantic structures</td><td></td></tr><tr><td>SemEval2018</td><td>12<a class="footnote-ref" id="fnref:16" href="#fn:16"><sup>16</sup></a></td><td>Affect and Creative Language in Tweets, Coreference, Information Extraction, Lexical Semantics and Reading Comprehension and Reasoning</td><td></td></tr></tbody></table> <p>The Multilingual WSD task was introduced for the SemEval-2013 workshop.<a class="footnote-ref" id="fnref:17" href="#fn:17"><sup>17</sup></a> The task is aimed at evaluating Word Sense Disambiguation systems in a multilingual scenario using BabelNet as its sense inventory. Unlike similar task like crosslingual WSD or the <a href="/facts/Lexical_substitution/HTXUVBEm">multilingual lexical substitution</a> task, where no fixed sense inventory is specified, Multilingual WSD uses the <a href="/facts/BabelNet/3c8e9Gz7">BabelNet</a> as its sense inventory. Prior to the development of BabelNet, a bilingual <a href="/facts/Classic_monolingual_WSD/jR1LbJLV">lexical sample</a> WSD evaluation task was carried out in SemEval-2007 on Chinese-English bitexts.<a class="footnote-ref" id="fnref:18" href="#fn:18"><sup>18</sup></a> </p><p>The Cross-lingual WSD task was introduced in the SemEval-2007 evaluation workshop and re-proposed in the SemEval-2013 workshop .<a class="footnote-ref" id="fnref:19" href="#fn:19"><sup>19</sup></a> To facilitate the ease of integrating WSD systems into other <a href="/facts/Natural_Language_Processing/1hjMKsSN">Natural Language Processing</a> (NLP) applications, such as Machine Translation and multilingual <a href="/facts/Information_Retrieval/MFSmpeNw">Information Retrieval</a>, the cross-lingual WSD evaluation task was introduced a language-independent and knowledge-lean approach to WSD. The task is an unsupervised Word Sense Disambiguation task for English nouns by means of parallel corpora. It follows the lexical-sample variant of the Classic WSD task, restricted to only 20 polysemous nouns. </p><p>It is worth noting that the SemEval-2014 have only two tasks that were multilingual/crosslingual, i.e. (i) the <a href="http://alt.qcri.org/semeval2014/task5/">L2 Writing Assistant</a> task, which is a crosslingual WSD task that includes English, Spanish, German, French and Dutch and (ii) the <a href="http://www.cs.york.ac.uk/semeval-2013/task12/">Multilingual Semantic Textual Similarity</a> task that evaluates systems on English and Spanish texts. </p> <h3>Areas of evaluation</h3> <p>The major tasks in semantic evaluation include the following areas of <a href="/facts/Natural_language_processing/1hjMKsSN">natural language processing</a>. This list is expected to grow as the field progresses.<a class="footnote-ref" id="fnref:20" href="#fn:20"><sup>20</sup></a> </p><p>The following table shows the areas of studies that were involved in Senseval-1 through SemEval-2014 (S refers to Senseval and SE refers to SemEval, e.g. S1 refers to Senseval-1 and SE07 refers to SemEval2007): </p> <table><tbody><tr><th>Areas of Study</th><th>S1</th><th>S2</th><th>S3</th><th>SE07</th><th>SE10</th><th>SE12</th><th>SE13</th><th>SE14</th><th>SE15</th><th>SE16</th><th>SE17</th></tr><tr><td>Bioinfomatics / Clinical Text Analysis</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Common Sense Reasoning (COPA)</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td><a href="/facts/Coreference/UH4fUcND">Coreference</a> Resolution</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td><a href="/facts/Compound_(linguistics)/IbumXJgd">Noun Compounds</a> (<a href="/facts/Information_Extraction/DL9nMD2X">Information Extraction</a>)</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td><a href="/facts/Ellipsis_(linguistics)/xDJyjytB">Ellipsis</a></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Grammar Induction</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td><a href="/facts/Automatic_summarization/yTQYaEIe">Keyphrase Extraction</a> (<a href="/facts/Information_Extraction/DL9nMD2X">Information Extraction</a>)</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td><a href="/facts/Lexical_simplification/fc9tP41H">Lexical Simplification</a></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Lexical Substitution (Multilingual or Crosslingual)</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Lexical Complexity</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td><a href="/facts/Metonymy/EdwzPEyE">Metonymy</a> (<a href="/facts/Information_Extraction/DL9nMD2X">Information Extraction</a>)</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td><a href="/facts/Paraphrase/YQuQNPcn">Paraphrases</a></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td><a href="/facts/Question_Answering/1gplmc6L">Question Answering</a></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Relational Similarity</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Rumour and veracity</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Semantic Parsing</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Semantic Relation Identification</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td><a href="/facts/Semantic_role_labeling/mykIh41q">Semantic Role Labeling</a></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Semantic Similarity</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Semantic Similarity (Crosslingual)</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Semantic Similarity (Multilingual)</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td><a href="/facts/Sentiment_analysis/QRdywf6g">Sentiment Analysis</a></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Spatial Role Labelling</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Taxonomy Induction/Enrichment</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Textual Entailment</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Textual Entailment (Cross-lingual)</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td><a href="/facts/Temporal_annotation/7Sefma2a">Temporal annotation</a></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Twitter Analysis</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Word sense disambiguation (<a href="/facts/Word_sense_disambiguation/Mfcw7gsu">Lexical Sample</a>)</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Word sense disambiguation (<a href="/facts/Word_sense_disambiguation/Mfcw7gsu">All-Words</a>)</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Word sense disambiguation (Multilingual)</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Word sense disambiguation (Cross-lingual)</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td><a href="/facts/Word_sense_induction/3pcwGFTo">Word sense induction</a></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr></tbody></table> <h3>Type of Semantic Annotations</h3> <p>SemEval tasks have created many types of semantic annotations, each type with various schema. In SemEval-2015, the organizers have decided to group tasks together into several tracks. These tracks are by the type of semantic annotations that the task hope to achieve.<a class="footnote-ref" id="fnref:21" href="#fn:21"><sup>21</sup></a> Here lists the type of semantic annotations involved in the SemEval workshops: </p> <ol><li>Learning Semantic Relations</li> <li>Question and Answering</li> <li>Semantic Parsing</li> <li>Semantic Taxonomy</li> <li>Sentiment Analysis</li> <li>Text Similarity</li> <li>Time and Space</li> <li>Word Sense Disambiguation and Induction</li></ol> <p>A task and its track allocation is flexible; a task might develop into its own track, e.g. the taxonomy evaluation task in SemEval-2015 was under the <i>Learning Semantic Relations</i> track and in SemEval-2016, there is a dedicated track for <i>Semantic Taxonomy</i> with a new <i>Semantic Taxonomy Enrichment</i> task.<a class="footnote-ref" id="fnref:22" href="#fn:22"><sup>22</sup></a><a class="footnote-ref" id="fnref:23" href="#fn:23"><sup>23</sup></a> </p> <h2 id="see-also">See also</h2> <ul><li><a href="/facts/List_of_computer_science_awards/WeVUHhBs">List of computer science awards</a></li> <li><a href="/facts/Computational_semantics/gbHj755X">Computational semantics</a></li> <li><a href="/facts/Natural_language_processing/1hjMKsSN">Natural language processing</a></li> <li><a href="/facts/Word_sense/9IR1286V">Word sense</a></li> <li><a href="/facts/Word_sense_disambiguation/Mfcw7gsu">Word sense disambiguation</a></li> <li><a href="/facts/Word_Sense_Disambiguation/Mfcw7gsu">Different variants of WSD evaluations</a></li> <li><a href="/facts/Semantic_analysis_(computational)/s3RsmLMv">Semantic analysis (computational)</a></li></ul> <h2 id="external-links">External links</h2> <ul><li><a href="http://www.siglex.org/">Special Interest Group on the Lexicon (SIGLEX)</a> of the <a href="http://www.aclweb.org/">Association for Computational Linguistics</a> (ACL)</li> <li><a href="http://semeval2.fbk.eu/semeval2.php/">Semeval-2010</a> – Semantic Evaluation Workshop (endorsed by SIGLEX)</li> <li><a href="https://web.archive.org/web/20030622211418/http://www.senseval.org/">Senseval</a>[<i>usurped</i>] - international organization devoted to the evaluation of Word Sense Disambiguation Systems (endorsed by SIGLEX)</li> <li><a href="http://aclweb.org/aclwiki/index.php?title=SemEval_Portal">SemEval Portal</a> on the Wiki of the Association for Computational Linguistics</li> <li>Senseval / SemEval tasks: <ul><li><a href="http://www.itri.brighton.ac.uk/events/senseval/ARCHIVE/index.html">Senseval-1</a> – the first evaluation exercise on word sense disambiguation systems; the lexical-sample task was evaluated on English, French and Italian</li> <li><a href="http://www.hipposmond.com/senseval2/">Senseval-2</a> – evaluated word sense disambiguation systems on three types of tasks (the all-words, lexical-sample and the translation task)</li> <li><a href="https://archive.today/20130415233904/http://www.senseval.org/senseval3/tasks.html">Senseval-3</a>[<i>usurped</i>] – included tasks for word sense disambiguation, as well as identification of semantic roles, multilingual annotations, logic forms, subcategorization acquisition.</li> <li><a href="http://nlp.cs.swarthmore.edu/semeval/tasks/index.php">SemEval-2007</a> – included tasks which were more elaborate than Senseval as it crosses the different areas of studies in Natural Language Processing</li> <li><a href="http://semeval2.fbk.eu/semeval2.php?location=tasks-short">SemEval-2010</a> – added tasks that were from new areas of studies in computational semantics, viz., Coreference, Ellipsis, Keyphrase Extraction, Noun Compounds and Textual Entailment.</li> <li><a href="http://www.cs.york.ac.uk/semeval-2012/">SemEval-2012</a> – co-located with the first *SEM conference and the Semantic Similarity Task was being promoted as the *Sem Shared Task</li> <li><a href="http://www.cs.york.ac.uk/semeval-2013/">SemEval-2013</a> – SemEval moved from 2–3 years cycle to an annual workshop</li> <li><a href="http://alt.qcri.org/semeval2014/">SemEval-2014</a> – the first time SemEval is located in a non-ACL event in COLING</li> <li><a href="http://alt.qcri.org/semeval2015/">SemEval-2015</a> – the first SemEval with tasks categorized into various tracks</li> <li><a href="http://alt.qcri.org/semeval2016/">SemEval-2016</a> – the second SemEval without a WSD task (the first was in SemEval-2012)</li> <li><a href="http://ixa2.si.ehu.es/starsem/">*SEM</a> – conference for SemEval-related papers other than task systems.</li></ul></li> <li><a href="http://www-nlpir.nist.gov/related_projects/muc/muc_sw/muc_sw_manual.html">Message Understanding Conferences (MUCs)</a></li> <li><a href="http://babelnet.org">BabelNet</a></li> <li><a href="http://compling.hss.ntu.edu.sg/omw/">Open Multilingual WordNet</a> – Compilation of WordNets with Open licenses</li></ul> <h2 id="references">References</h2> <ol> <li id="fn:1"><p>Blackburn, P., and Bos, J. (2005), Representation and Inference for Natural Language: A First Course in Computational Semantics, CSLI Publications. ISBN 1-57586-496-7. <a href="/wiki/ISBN_(identifier)" target="_blank">/wiki/ISBN_(identifier)</a> <a href="#fnref:1" class="footnote-back-ref">↩</a></p></li> <li id="fn:2"><p>Navigli, R (2009). "Word sense disambiguation". ACM Computing Surveys. 41 (2): 1–69. doi:10.1145/1459352.1459355. S2CID 461624. <a href="/wiki/Doi_(identifier)" target="_blank">/wiki/Doi_(identifier)</a> <a href="#fnref:2" class="footnote-back-ref">↩</a></p></li> <li id="fn:3"><p>Palmer, M., Ng, H.T., & Hoa, T.D. (2006), Evaluation of WSD systems, in Eneko Agirre & Phil Edmonds (eds.), Word Sense Disambiguation: Algorithms and Applications, Text, Speech and Language Technology, vol. 33. Amsterdam: Springer, 75–106. <a href="https://books.google.com/books?id=GLck75U20pAC" target="_blank">https://books.google.com/books?id=GLck75U20pAC</a> <a href="#fnref:3" class="footnote-back-ref">↩</a></p></li> <li id="fn:4"><p>Resnik, P. (2006), WSD in NLP applications, in Eneko Agirre & Phil Edmonds (eds.), Word Sense Disambiguation: Algorithms and Applications. Dordrecht: Springer, 299–338. <a href="#fnref:4" class="footnote-back-ref">↩</a></p></li> <li id="fn:5"><p>Yarowsky, D. (1992), Word-sense disambiguation using statistical models of Roget’s categories trained on large corpora. Proceedings of the 14th Conference on Computational Linguistics, 454–60. doi:10.3115/992133.992140 <a href="/wiki/Doi_(identifier)" target="_blank">/wiki/Doi_(identifier)</a> <a href="#fnref:5" class="footnote-back-ref">↩</a></p></li> <li id="fn:6"><p>Palmer, M., & Light, M. (1999), Tagging with Lexical Semantics: Why, What, and How?| ACL SIGLEX workshop on tagging text with lexical semantics: what, why, and how? Archived 2010-07-15 at the Wayback Machine Natural Language Engineering 5(2): i–iv. <a href="http://www.ims.uni-stuttgart.de/~light/tueb_html/semtag_ws.html" target="_blank">http://www.ims.uni-stuttgart.de/~light/tueb_html/semtag_ws.html</a> <a href="#fnref:6" class="footnote-back-ref">↩</a></p></li> <li id="fn:7"><p>Ng, H.T. (1997), Getting serious about word sense disambiguation. Proceedings of the ACL SIGLEX Workshop on Tagging Text with Lexical Semantics: Why, What, and How? 1–7. <a href="#fnref:7" class="footnote-back-ref">↩</a></p></li> <li id="fn:8"><p>Philip Resnik and Jimmy Lin (2010). Evaluation of NLP Systems. In Alexander Clark, Chris Fox, and Shalom Lappin, editors. The Handbook of Computational Linguistics and Natural Language Processing. Wiley-Blackwellis. 11:271 <a href="#fnref:8" class="footnote-back-ref">↩</a></p></li> <li id="fn:9"><p>Adam Kilgarriff and Martha Palmer (ed. 2000). Special Issue of Computers and the Humanities, SENSEVAL98:Evaluating Word Sense Disambiguation Systems. Kluwer, 34: 1–2. <a href="#fnref:9" class="footnote-back-ref">↩</a></p></li> <li id="fn:10"><p>Scott Cotton, Phil Edmonds, Adam Kilgarriff, and Martha Palmer (ed. 2001). SENSEVAL-2: Second International Workshop on Evaluating Word Sense Disambiguation Systems. SIGLEX Workshop, ACL03, Toulouse, France. <a href="#fnref:10" class="footnote-back-ref">↩</a></p></li> <li id="fn:11"><p>SIGLEX: Message Board (2010) Retrieved on Aug 15, 2012, from http://www.clres.com/siglex/messdisp.php?id=111 <a href="http://www.clres.com/siglex/messdisp.php?id=111" target="_blank">http://www.clres.com/siglex/messdisp.php?id=111</a> <a href="#fnref:11" class="footnote-back-ref">↩</a></p></li> <li id="fn:12"><p>SemEval 3 Google Group post. Retrieved on Aug 15, 2012, from https://groups.google.com/forum/?fromgroups#!topic/semeval3/8YXMvVlH-CM%5B1-25%5D <a href="https://groups.google.com/forum/?fromgroups#!topic/semeval3/8YXMvVlH-CM%5B1-25%5D" target="_blank">https://groups.google.com/forum/?fromgroups#!topic/semeval3/8YXMvVlH-CM%5B1-25%5D</a> <a href="#fnref:12" class="footnote-back-ref">↩</a></p></li> <li id="fn:13"><p>Language Resources and Evaluation Volume 43, Number 2[dead link] <a href="http://springerlink.com/1574-020x/43/2/" target="_blank">http://springerlink.com/1574-020x/43/2/</a> <a href="#fnref:13" class="footnote-back-ref">↩</a></p></li> <li id="fn:14"><p>Kilgarriff, A. (1998). SENSEVAL: An Exercise in Evaluating Word Sense Disambiguation Programs. In Proc. LREC, Granada, May 1998. Pp 581--588 <a href="#fnref:14" class="footnote-back-ref">↩</a></p></li> <li id="fn:15"><p>"Tasks < SemEval-2017". alt.qcri.org. Retrieved 2018-05-04. <a href="http://alt.qcri.org/semeval2017/index.php?id=tasks" target="_blank">http://alt.qcri.org/semeval2017/index.php?id=tasks</a> <a href="#fnref:15" class="footnote-back-ref">↩</a></p></li> <li id="fn:16"><p>"Tasks < SemEval-2018". alt.qcri.org. Retrieved 2018-05-04. <a href="http://alt.qcri.org/semeval2018/index.php?id=tasks" target="_blank">http://alt.qcri.org/semeval2018/index.php?id=tasks</a> <a href="#fnref:16" class="footnote-back-ref">↩</a></p></li> <li id="fn:17"><p>Navigli, R., Jurgens, D., & Vannella, D. (2013, June). Semeval-2013 task 12: Multilingual word sense disambiguation. In Proceedings of the 7th International Workshop on Semantic Evaluation (SemEval 2013), in conjunction with the Second Joint Conference on Lexical and Computational Semantics (* SEM 2013) (pp. 222-231). <a href="#fnref:17" class="footnote-back-ref">↩</a></p></li> <li id="fn:18"><p>Peng Jin, Yunfang Wu and Shiwen Yu. SemEval-2007 task 05: multilingual Chinese-English lexical sample. Proceedings of the 4th International Workshop on Semantic Evaluations, p.19-23, June 23–24, 2007, Prague, Czech Republic. <a href="http://aclweb.org/anthology-new/S/S07/S07-1004.pdf" target="_blank">http://aclweb.org/anthology-new/S/S07/S07-1004.pdf</a> <a href="#fnref:18" class="footnote-back-ref">↩</a></p></li> <li id="fn:19"><p>Lefever, E., & Hoste, V. (2013, June). Semeval-2013 task 10: Cross-lingual word sense disambiguation. In Second joint conference on lexical and computational semantics (Vol. 2, pp. 158-166). <a href="#fnref:19" class="footnote-back-ref">↩</a></p></li> <li id="fn:20"><p>SemEval Portal (n.d.). In ACLwiki. Retrieved August 12, 2010 from http://aclweb.org/aclwiki/index.php?title=SemEval_Portal <a href="http://aclweb.org/aclwiki/index.php?title=SemEval_Portal" target="_blank">http://aclweb.org/aclwiki/index.php?title=SemEval_Portal</a> <a href="#fnref:20" class="footnote-back-ref">↩</a></p></li> <li id="fn:21"><p>SemEval-2015 website. Retrieved Nov 14, 2014 http://alt.qcri.org/semeval2015/index.php?id=tasks <a href="http://alt.qcri.org/semeval2015/index.php?id=tasks" target="_blank">http://alt.qcri.org/semeval2015/index.php?id=tasks</a> <a href="#fnref:21" class="footnote-back-ref">↩</a></p></li> <li id="fn:22"><p>Georgeta Bordea, Paul Buitelaar, Stefano Faralli and Roberto Navigli. 2015. Semeval-2015 task 17: Taxonomy Extraction Evaluation (TExEval). In Proceedings of the 9th International Workshop on Semantic Evaluation. Denver, USA. <a href="#fnref:22" class="footnote-back-ref">↩</a></p></li> <li id="fn:23"><p>SemEval-2016 website. Retrieved Jun 4 2015 http://alt.qcri.org/semeval2016/ <a href="http://alt.qcri.org/semeval2016/" target="_blank">http://alt.qcri.org/semeval2016/</a> <a href="#fnref:23" class="footnote-back-ref">↩</a></p></li> </ol>