Parallel sense-annotated corpus ELEXIS-WSD 1.2

  Authors

Čibej, Jaka ; et al.show everyone

Čibej, Jaka ; Krek, Simon ; Tiberius, Carole ; Martelli, Federico ; Navigli, Roberto ; Kallas, Jelena ; Gantar, Polona ; Koeva, Svetla ; Nimb, Sanni ; Sandford Pedersen, Bolette ; Olsen, Sussi ; Langemets, Margit ; Koppel, Kristina ; Üksik, Tiiu ; Dobrovoljc, Kaja ; Ureña-Ruiz, Rafael ; Sancho-Sánchez, José-Luis ; Lipp, Veronika ; Váradi, Tamás ; Győrffy, András ; Simon, László ; Quochi, Valeria ; Monachini, Monica ; Frontini, Francesca ; Tempelaars, Rob ; Costa, Rute ; Salgado, Ana ; Munda, Tina ; Kosem, Iztok ; Roblek, Rebeka ; Kamenšek, Urška ; Zaranšek, Petra ; Zgaga, Karolina ; Ponikvar, Primož ; Terčon, Luka ; Jensen, Jonas ; Flörke, Ida ; Lorentzen, Henrik ; Troelsgård, Thomas ; Blagoeva, Diana ; Hristov, Dimitar ; Kolkovska, Sia

  Item identifier

http://hdl.handle.net/11356/2022

 Project URL

https://elex.is/

 Referenced by

https://elex.link/elex2021/wp-content/uploads/2021/08/eLex_2021_22_pp377-395.pdf

 Date issued

2025-04-04

 Type

corpus, text

 Size

20 files, 345092 tokens, 20240 sentences

 Language(s)

Bulgarian , Danish , Dutch , English , Estonian , Hungarian , Italian , Portuguese , Slovenian , Spanish

 Description

ELEXIS-WSD is a parallel sense-annotated corpus in which content words (nouns, adjectives, verbs, and adverbs) have been assigned senses. Version 1.2 contains sentences for 10 languages: Bulgarian, Danish, English, Spanish, Estonian, Hungarian, Italian, Dutch, Portuguese, and Slovene. The corpus was compiled by automatically extracting a set of sentences from WikiMatrix (Schwenk et al., 2019), a large open-access collection of parallel sentences derived from Wikipedia, using an automatic approach based on multilingual sentence embeddings. The sentences were manually validated according to specific formal, lexical and semantic criteria (e.g. by removing incorrect punctuation, morphological errors, notes in square brackets and etymological information typically provided in Wikipedia pages). To obtain a satisfactory semantic coverage, we filtered out sentences with less than 5 words and less than 2 polysemous words were filtered out. Subsequently, in order to obtain datasets in the other nine target languages, for each selected sentence in English, the corresponding WikiMatrix translation into each of the other languages was retrieved. If no translation was available, the English sentence was translated manually. The resulting corpus is comprised of 2,024 sentences for each language. The sentences were tokenized, lemmatized, and tagged with UPOS tags using UDPipe v2.6 (https://lindat.mff.cuni.cz/services/udpipe/). Senses were annotated using LexTag (https://elexis.babelscape.com/): each content word (noun, verb, adjective, and adverb) was assigned a sense from among the available senses from the sense inventory selected for the language (see below) or BabelNet. Sense inventories were also updated with new senses during annotation. Dependency relations were added with UDPipe 2.15 in version 1.2. List of sense inventories BG: Dictionary of Bulgarian DA: DanNet – The Danish WordNet EN: Open English WordNet ES: Spanish Wiktionary ET: The EKI Combined Dictionary of Estonian HU: The Explanatory Dictionary of the Hungarian Language IT: PSC + Italian WordNet NL: Open Dutch WordNet PT: Portuguese Academy Dictionary (DACL) SL: Digital Dictionary Database of Slovene The corpus is available in the CoNLL-U tab-separated format. In order, the columns contain the token ID, its form, its lemma, its UPOS-tag, its XPOS-tag (if available), its morphological features (FEATS), the head of the dependency relation (HEAD), the type of dependency relation (DEPREL); the ninth column (DEPS) is empty; the final MISC column contains the following: the token's whitespace information (whether the token is followed by a whitespace or not; e.g. SpaceAfter=No), the ID of the sense assigned to the token, the index of the multiword expression (if the token is part of an annotated multiword expression), and the index and type of the named entity annotation (currently only available in elexis-wsd-sl). Each language has a separate sense inventory containing all the senses (and their definitions) used for annotation in the corpus. Not all the senses from the sense inventory are necessarily included in the corpus annotations: for instance, all occurrences of the English noun "bank" in the corpus might be annotated with the sense of "financial institution", but the sense inventory also contains the sense "edge of a river" as well as all other possible senses to disambiguate between. For more information, please refer to 00README.txt. Updates in version 1.2: - Several tokenization errors with multiword tokens were fixed in all subcorpora (e.g. the order of subtokens was incorrect in many cases; the issue has now been resolved). - XPOS, FEATS, HEAD, and DEPREL columns were added automatically with UDPipe (except for elexis-wsd-sl and elexis-wsd-et; for Slovene, all columns were manually validated; for Estonian, HEAD and DEPREL were manually validated; all other languages contain automatic tags in these columns – for more information on the models used and their performance, see 00README.txt). - The entry now includes lists of potential errors in automatically assigned XPOS and FEATS values. In previous versions, only UPOS tags were manually annotated, while the XPOS and FEATS columns were left empty. XPOS and FEATS have now been added automatically through UDPipe. The list of potential errors contains the list of lines in the corpus in which the XPOS and FEATS columns are potentially incorrect because the manually validated UPOS tag differs from the automatically assigned UPOS tag, which indicates that the automatically assigned XPOS and FEATS columns are probably incorrect. This is meant as a reference for future validation efforts. - For Slovene, named entity annotations were added based on the annotations from the SUK 1.1 Training Corpus of Slovene (http://hdl.handle.net/11356/1959).

 Publisher

Jožef Stefan Institute

 Acknowledgement

European Union EC/H2020/731015 "ELEXIS - European Lexicographic Infrastructure"

Jožef Stefan Institute CLARIN "CLARIN.SI"

COST CA21167 "Universality, Diversity and Idiosyncrasy in Language Technology (UniDive)"

ARRS (Slovenian Research Agency) P6-0411 "Language Resources and Technologies for Slovene"

 Subject(s)

word sense disambiguation parallel corpus sense annotation multilingual

 Collection(s)

CLARIN.SI ELEXIS

 

This item is replaced by a newer submission:

http://hdl.handle.net/11356/2029

List all versions  

Show full item record