"Choice of plausible alternatives" datasets in South Slavic dialects DIALECT-COPA

Name: "Choice of plausible alternatives" datasets in South Slavic dialects DIALECT-COPA
License: https://creativecommons.org/licenses/by-sa/4.0/

Ljubešić, Nikola; Kuzman, Taja; Rupnik, Peter; Milosavljević, Stefan; Galant, Nada; Benčina, Sonja; Čibej, Jaka

"Choice of plausible alternatives" datasets in South Slavic dialects DIALECT-COPA

CLARIN.SI data & tools

Avtorji: Ljubešić, Nikola ; et al.prikaži vse
Ljubešić, Nikola ; Kuzman, Taja ; Rupnik, Peter ; Milosavljević, Stefan ; Galant, Nada ; Benčina, Sonja ; Čibej, Jaka

Identifikator vnosa: http://hdl.handle.net/11356/1766

URL projekta: https://sites.google.com/view/vardial-2024/shared-tasks/dialect-copa

Dokumentirano v: https://aclanthology.org/2024.vardial-1.7/

Datum objave: 2024-04-26

Vrsta: corpus, text

Velikost: 1500 items, 6 files, 279 kb

Jezik(i): Chakavian , Croatian , Serbian , Slovenian

Opis: The DIALECT-COPA datasets comprise Choice of Plausible Alternatives (COPA) datasets for three South Slavic dialects: (1) COPA-SL-CER for the Cerkno dialect of Slovenian, spoken in the Slovenian Littoral region, specifically from the town of Idrija; (2) COPA-HR-CKM for the Chakavian dialect of Croatian from northern Adriatic, specifically from the town of Žminj; (3) COPA-SR-TOR for the Torlak dialect from southeastern Serbia, specifically from the town of Lebane. The datasets were translated from the English COPA dataset (https://people.ict.usc.edu/~gordon/copa.html) by native dialect speakers, following the XCOPA dataset translation methodology (https://arxiv.org/abs/2005.00333). A novelty in the DIALECT-COPA translation approach is that both English and the corresponding standard South Slavic language were at disposal to the translator during the translation process. Each instance consists of a premise (My body cast a shadow over the grass), a question (What is the cause? / What happened as a result?), and two choices (The sun was rising; The grass was cut), with a label encoding which of the choices is more plausible given the annotator or translator (The sun was rising). The datasets follow the same format as the Croatian COPA-HR dataset (http://hdl.handle.net/11356/1404), the Macedonian COPA-MK dataset (http://hdl.handle.net/11356/1687) and the Serbian COPA-SR dataset (http://hdl.handle.net/11356/1708). Each dataset is split into training (400 instances) and validation (100 instances) JSONL files. The test split (500 instances), which is usually a part of the COPA datasets, has been withheld and can be shared upon request. The reason for this is to prevent its inclusion of the test instances in the training data of future large language models, which would invalidate the benchmark measurements. The DIALECT-COPA datasets are published as part of the DIALECT-COPA shared task at the VarDial 2024 workshop where they were used as gold data for evaluation of the performance of large language models on South Slavic dialects (https://sites.google.com/view/vardial-2024/shared-tasks/dialect-copa).