"Choice of plausible alternatives" datasets in South Slavic dialects DIALECT-COPA

  Authors

Ljubešić, Nikola ; et al.show everyone

Ljubešić, Nikola ; Kuzman, Taja ; Rupnik, Peter ; Milosavljević, Stefan ; Galant, Nada ; Benčina, Sonja ; Čibej, Jaka

  Item identifier

http://hdl.handle.net/11356/1766

 Project URL

https://sites.google.com/view/vardial-2024/shared-tasks/dialect-copa

 Referenced by

https://aclanthology.org/2024.vardial-1.7/

 Date issued

2024-04-26

 Type

corpus, text

 Size

1500 items, 6 files, 279 kb

 Language(s)

Chakavian , Croatian , Serbian , Slovenian

 Description

The DIALECT-COPA datasets comprise Choice of Plausible Alternatives (COPA) datasets for three South Slavic dialects: (1) COPA-SL-CER for the Cerkno dialect of Slovenian, spoken in the Slovenian Littoral region, specifically from the town of Idrija; (2) COPA-HR-CKM for the Chakavian dialect of Croatian from northern Adriatic, specifically from the town of Žminj; (3) COPA-SR-TOR for the Torlak dialect from southeastern Serbia, specifically from the town of Lebane. The datasets were translated from the English COPA dataset (https://people.ict.usc.edu/~gordon/copa.html) by native dialect speakers, following the XCOPA dataset translation methodology (https://arxiv.org/abs/2005.00333). A novelty in the DIALECT-COPA translation approach is that both English and the corresponding standard South Slavic language were at disposal to the translator during the translation process. Each instance consists of a premise (My body cast a shadow over the grass), a question (What is the cause? / What happened as a result?), and two choices (The sun was rising; The grass was cut), with a label encoding which of the choices is more plausible given the annotator or translator (The sun was rising). The datasets follow the same format as the Croatian COPA-HR dataset (http://hdl.handle.net/11356/1404), the Macedonian COPA-MK dataset (http://hdl.handle.net/11356/1687) and the Serbian COPA-SR dataset (http://hdl.handle.net/11356/1708). Each dataset is split into training (400 instances) and validation (100 instances) JSONL files. The test split (500 instances), which is usually a part of the COPA datasets, has been withheld and can be shared upon request. The reason for this is to prevent its inclusion of the test instances in the training data of future large language models, which would invalidate the benchmark measurements. The DIALECT-COPA datasets are published as part of the DIALECT-COPA shared task at the VarDial 2024 workshop where they were used as gold data for evaluation of the performance of large language models on South Slavic dialects (https://sites.google.com/view/vardial-2024/shared-tasks/dialect-copa).

 Publisher

Jožef Stefan Institute

 Acknowledgement

ARRS (Slovenian Research Agency) P6-0411 "Language Resources and Technologies for Slovene"

ARRS (Slovenian Research Agency) J7-4642 "MEZZANINE"

Jožef Stefan Institute CLARIN "CLARIN.SI"

 Subject(s)

commonsense reasoning manual translation

 Collection(s)

CLARIN.SI data & tools

Show full item record