Choice of plausible alternatives dataset in Croatian COPA-HR

Name: Choice of plausible alternatives dataset in Croatian COPA-HR
License: https://creativecommons.org/licenses/by-sa/4.0/

Description: The COPA-HR dataset (Choice of plausible alternatives in Croatian) is a translation of the English COPA dataset (https://people.ict.usc.edu/~gordon/copa.html) by following the XCOPA dataset translation methodology (https://arxiv.org/abs/2005.00333). The dataset consists of 1000 premises (My body cast a shadow over the grass), each given a question (What is the cause?), and two choices (The sun was rising; The grass was cut), with a label encoding which of the choices is more plausible given the annotator or translator (The sun was rising). The observed agreement of the English annotator and the Croatian translator is perfect on the training and the validation dataset, with one different label (agreement of 99.8%) on the test dataset. The current state-of-the-art on this dataset is held by the BERTić model (https://huggingface.co/CLASSLA/bcms-bertic), achieving an accuracy of 66% (50% is random).