Pragmatics understanding benchmark for Czech, Slovenian and Croatian PragMega CzeSloCro

 
CLARIN.SI data & tools
  Authors
Vintar, Špela ; Brglez, Mojca ; Potočnjak, Mirna ; Žižkova, Hana and Sangawa Hmeljak, Nina
  Item identifier
http://hdl.handle.net/11356/2261
 Referenced by
https://doi.org/10.63317/4bpncy453r9k
 Date issued
2026-07-21
 Type
corpus, text
 Size
420 items
 Language(s)
Croatian , Czech , English , Slovenian
 Description
PragMega CzeSloCro is a translation and adaptation of a section of the PragMega dataset (Floyd et al., 2026) into Czech, Slovenian and Croatian. The original dataset was manually crafted by psychologists and aimed at discovering whether "pragmatic inferencing" is a result of a single cognitive skill or, on the contrary, of different dissociable skills depending on the type of phenomena encountered. A Slovenian version of the dataset was created first, for which we selected three tasks: Irony, Metaphor, and Humour. These consist of 50, 30, and 25 examples, respectively, or 105 examples in total. The Slovenian benchmark is described in Brglez & Vintar (2026) and is implemented as SloPragMega on the SloBench (https://slobench.cjvt.si) platform. The dataset was then translated into Croatian and Czech by students of Digital Linguistics within a student project, then thoroughly revised by two professional linguists. Due to the highly nuanced and culturally specific nature of the dataset, some tasks were completely rewritten or replaced by more naturally sounding examples in the respective language. For each language, the dataset is divided into 3 subfolders (Metaphor, Irony, Humor), which contain the following files: - stim.csv: The actual localised tasks with possible answers, - stim_en.csv: The original English tasks with possible answers, - stimOrder.csv: Template to create a randomized test with the order of questions and answers reshuffled, - keys.csv: Solutions for the shuffled tasks. References: Brglez, M., & Vintar, S. (2026). From Polyester Girlfriends to Blind Mice: Creating the First Pragmatics Understanding Benchmarks for Slovene. In The Fourth Workshop on Resources and Representations for Under-Resourced Languages and Domains (RESOURCEFUL 2026) (pp. 44–54). European Language Resources Association (ELRA). https://doi.org/10.63317/4bpncy453r9k Floyd, S., Gibson, E., Fedorenko, E., & Poliak, M. (2026, January 14). PragMega. https://doi.org/10.17605/OSF.IO/DPGE6
 Publisher
Jožef Stefan Institute
 Subject(s)
pragmatics benchmark metaphor irony humour
 Collection(s)
CLARIN.SI data & tools
Show full item record
 

 Files in this item

This item is
Publicly Available
and licensed under:
Creative Commons - Attribution-ShareAlike 4.0 International (CC BY-SA 4.0)
Distributed under Creative Commons Attribution Required Share Alike
Icon
Name
PragMega-EN-SL-HR-CZ.zip
Size
152.98 KB
Format
application/zip
Description
Dataset in CSV format
MD5
d2dfa62d790c252d375d9dce43f02b28
 Download file  Preview
 File Preview  
  • PragMega
    • SL
      • HumorText
        • stim_en.csv-1 B
        • stimOrder.csv-1 B
        • stim.csv-1 B
        • keys.csv-1 B
      • MetaphorText
        • stim_en.csv-1 B
        • stimOrder.csv-1 B
        • stim.csv-1 B
        • keys.csv-1 B
      • IronyText
        • stim_en.csv-1 B
        • stimOrder.csv-1 B
        • stim.csv-1 B
        • keys.csv-1 B
    • HR
      • HumorText
        • stim_en.csv-1 B
        • stimOrder.csv-1 B
        • stim.csv-1 B
        • keys.csv-1 B
      • MetaphorText
        • stim_en.csv-1 B
        • stimOrder.csv-1 B
        • stim.csv-1 B
        • keys.csv-1 B
      • IronyText
        • stim_en.csv-1 B
        • stimOrder.csv-1 B
        • stim.csv-1 B
        • keys.csv-1 B
    • PragMegaOrig
      • HumorText
        • keys copy.csv-1 B
        • stimOrder.csv-1 B
        • stim.csv-1 B
        • keys.csv-1 B
      • MetaphorText
        • stimOrder.csv-1 B
        • stim.csv-1 B
        • keys.csv-1 B
      • IronyText
        • stimOrder.csv-1 B
        • stim.csv-1 B
        • keys.csv-1 B
    • CZ
      • HumorText
        • stim_en.csv-1 B
        • stimOrder.csv-1 B
        • stim.csv-1 B
        • keys.csv-1 B
      • MetaphorText
        • stim_en.csv-1 B
        • stimOrder.csv-1 B
        • stim.csv-1 B
        • keys.csv-1 B
      • IronyText
        • stim_en.csv-1 B
        • stimOrder.csv-1 B
        • stim.csv-1 B
        • keys.csv-1 B
  • __MACOSX
    • PragMega
      • ._PragMegaOrig-1 B
      • ._SL-1 B
      • SL
        • HumorText
          • ._stim.csv-1 B
          • ._stimOrder.csv-1 B
          • ._keys.csv-1 B
          • ._stim_en.csv-1 B
        • ._HumorText-1 B
        • MetaphorText
          • ._stim.csv-1 B
          • ._stimOrder.csv-1 B
          • ._keys.csv-1 B
          • ._stim_en.csv-1 B
        • IronyText
          • ._stim.csv-1 B
          • ._stimOrder.csv-1 B
          • ._keys.csv-1 B
          • ._stim_en.csv-1 B
        • ._MetaphorText-1 B
        • ._IronyText-1 B
      • ._HR-1 B
      • HR
        • HumorText
          • ._stim.csv-1 B
          • ._stimOrder.csv-1 B
          • ._keys.csv-1 B
          • ._stim_en.csv-1 B
        • ._HumorText-1 B
        • MetaphorText
          • ._stim.csv-1 B
          • ._stimOrder.csv-1 B
          • ._keys.csv-1 B
          • ._stim_en.csv-1 B
        • IronyText
          • ._stim.csv-1 B
          • ._stimOrder.csv-1 B
          • ._keys.csv-1 B
          • ._stim_en.csv-1 B
        • ._MetaphorText-1 B
        • ._IronyText-1 B
      • ._CZ-1 B
      • PragMegaOrig
        • HumorText
          • ._keys copy.csv-1 B
          • ._stim.csv-1 B
          • ._stimOrder.csv-1 B
          • ._keys.csv-1 B
        • ._HumorText-1 B
        • MetaphorText
          • ._stim.csv-1 B
          • ._stimOrder.csv-1 B
          • ._keys.csv-1 B
        • IronyText
          • ._stim.csv-1 B
          • ._stimOrder.csv-1 B
          • ._keys.csv-1 B
        • ._MetaphorText-1 B
        • ._IronyText-1 B
      • CZ
        • HumorText
          • ._stim.csv-1 B
          • ._stimOrder.csv-1 B
          • ._keys.csv-1 B
          • ._stim_en.csv-1 B
        • ._HumorText-1 B
        • MetaphorText
          • ._stim.csv-1 B
          • ._stimOrder.csv-1 B
          • ._keys.csv-1 B
          • ._stim_en.csv-1 B
        • IronyText
          • ._stim.csv-1 B
          • ._stimOrder.csv-1 B
          • ._keys.csv-1 B
          • ._stim_en.csv-1 B
        • ._MetaphorText-1 B
        • ._IronyText-1 B
    • ._PragMega-1 B

Partners

Partners

Repository

This platform runs under the software developed for the LINDAT/CLARIAH-CZ repository for linguistics, available on GitHub

CLARIN.SI is supported by the Ministry of Education, Science and Sport of the Republic of Slovenia
under the Programme of "Research Infrastructures".