Dataset of annotated collocation-distractor pairs COLLDIST

Name: Dataset of annotated collocation-distractor pairs COLLDIST
License: https://creativecommons.org/licenses/by/4.0/

Kosem, Iztok; Arhar Holdt, Špela; Zgaga, Karolina; Arčon, Tjaša

Dataset of annotated collocation-distractor pairs COLLDIST

CLARIN.SI data & tools

Authors: Kosem, Iztok ; Arhar Holdt, Špela ; Zgaga, Karolina and Arčon, Tjaša

Item identifier: http://hdl.handle.net/11356/2076

Project URL: https://www.cjvt.si/llm4dh/en/

Referenced by: https://elex.link/elex2025/wp-content/uploads/eLex2025-37-KosemArhar-Holdt.pdf

Date issued: 2025-12-23

Type: lexicalConceptualResource, text

Size: 59598 entries

Language(s): Slovenian

Description: The dataset contains 59,598 collocation-distractor pairs for 2,856 headwords. Distractor is defined as an incorrect answer/alternative to collocation, which can be similar to collocation meaning and/or form. Headwords and their collocations were obtained from the Collocations Dictionary of Modern Slovene (http://hdl.handle.net/11356/1933), which is part of the Dictionary Database of Slovene (the database is available via API: https://wiki.cjvt.si/books/digital-dictionary-database-of-slovene). The criteria for selecting the collocations were that they had to be manually validated and assigned under one of the senses of the headword. The distractors were obtained with the gpt-4o-2024-08-06 (https://platform.openai.com/docs/models/gpt-4o) model, using the following prompt: “We are preparing a language game where the player will be given a headword, a collocation (combination of the headword and another word) and a distractor (a collocation that has the same headword, but the other word is not a collocate of the main word). For example "huge victory" is a collocation of "victory", but "rotten victory" is not so "rotten is a good distractor. The rules for forming distractors are the following: 1. Distractor has to be a single word. 2. Distractor has to have the same part of speech as the word being replaced (e.g. if the word next to the headword is a noun, the distractor should also be a noun). 3. Distractor should not include sensitive vocabulary, e.g. related to minorities, nationalities, religion, sexual content and similar. 4. Distractor has to be a word that is frequent in the Slovene language. 5. Distractor has to be a word that is completely unlikely to occur with the headword. Return the distractor in the same format as the examples below: Example: hiter - hitre rešitve (hiter + rešitve) - hitre težave (hiter + težava) Example: obljuba - držati obljubo (držati + obljuba) - najti obljubo (najti + obljuba). This is the headword: {headword}. This is the collocation: {collocation} ({all_collocation_parts}). The distractor has to be a collocation that contains the headword {headword} but is unlikely to occur with it. Only return the distractor in the correct format with the given headword {headword}. No explanations, no other text.” The evaluation was conducted in two parts, automatic and manual. Five decisions were used: good, good-possible_collocation, bad-wrong_headword, bad-same_as_collocation, bad-grammar_problem, bad-collocation. The automatic evaluation used the frequency information from the data warehouse containing all the collocations from the Gigafida 2.0 corpus. If the distractor was found there, it was considered bad (bad-collocation), otherwise good. We also identified the distractors that were the same as the collocations provided (bad-same_as_collocation), and the distractors where the headword was not kept (bad-wrong_headword). The remaining distractors were manually evaluated and those with grammatical problems (collocate and headword not matching in case, number, gender etc.) were labelled as bad (bad-grammar_problem). All the others were considered good (45,772 out of 59,598, or 77 %), however we also annotated those that were deemed as possible collocations (i.e. the combination sounded as possible in the language, e.g. to party loudly) (good-possible_collocation). The dataset also includes the information on the frequency and logDice statistics of collocations and the distractors in the Gigafida 2.0 reference corpus of Slovene (http://hdl.handle.net/11356/1320).