Slovene Natural Language Inference Dataset SI-NLI

  Authors

Klemen, Matej ; Žagar, Aleš ; Čibej, Jaka and Robnik-Šikonja, Marko

  Item identifier

http://hdl.handle.net/11356/1707

 Date issued

2022-11-13

 Type

corpus, text

 Size

5937 units

 Language(s)

Slovenian

 Description

SI-NLI (Slovene Natural Language Inference Dataset) contains 5,937 human-created Slovene sentence pairs (premise and hypothesis) that are manually labeled with the labels "entailment", "contradiction", and "neutral". We created the dataset using sentences that appear in the Slovenian reference corpus ccKres (http://hdl.handle.net/11356/1034). Annotators were tasked to modify the hypothesis in a candidate pair in a way that reflects one of the labels. The dataset is balanced since the annotators created three modifications (entailment, contradiction, neutral) for each candidate sentence pair. The dataset is split into train, validation, and test sets, with sizes of 4,392, 547, and 998. We used Slovenian pre-trained language models to create splits, thereby ensuring that difficult and easy instances are evenly distributed in all three subsets. The dataset is released in a tabular TSV format. The README.txt file contains a description of the attributes. Only the hypothesis and premise are given in the test set (i.e. no annotations) since SI-NLI is integrated into the Slovene evaluation framework SloBENCH (https://slobench.cjvt.si/). If you use the dataset to train your models, please consider submitting the test set predictions to SloBENCH to get the evaluation score and see how it compares to others.

 Publisher

Faculty of Computer and Information Science, University of Ljubljana

 Acknowledgement

Jožef Stefan Institute CLARIN "CLARIN.SI"

 Subject(s)

natural language inference

 Collection(s)

CLARIN.SI data & tools

Show full item record