A Multilingual Benchmark for Evaluating Metalinguistic Knowledge WALS-Bench 1.0

  Avtorji

Arčon, Tjaša ; Klemen, Matej ; Robnik-Šikonja, Marko ; Dobrovoljc, Kaja and Terčon, Luka

  Identifikator vnosa

http://hdl.handle.net/11356/2083

 Dokumentirano v

https://arxiv.org/abs/2602.02182

 Datum objave

2026-03-03

 Vrsta

corpus, text

 Velikost

192 entries, 76475 entries

 Jezik(i)

English , Multiple languages

 Opis

This is a large-scale multilingual benchmark for evaluating metalinguistic knowledge (i.e. explicit knowledge about the structure of languages) in large language models using grammatical features from the World Atlas of Language Structures (WALS). The benchmark covers 192 linguistic features across 12 linguistic domains and 2,660 languages and is available in two formats (jsonl files): - Format 1 (192-question version): One question per feature, under which all languages with a corresponding ground truth value for that feature are listed. - Format 2 (76,475-question version): One question per feature-language pair with a corresponding ground truth value, fully expanded across all languages. The original WALS data is licensed under CC BY 4.0. The data has been adapted for use in this benchmark. Source: Dryer, Matthew S. & Haspelmath, Martin (eds.). World Atlas of Language Structures Online. Max Planck Institute for Evolutionary Anthropology. https://wals.info

 Izdajatelj

Faculty of Computer and Information Science, University of Ljubljana

 Zahvala

ARIS (Slovenian Research and Innovation Agency) GC-0002 "LLM4DH: Large Language Models for Digital Humanities"

The Slovenian Research and Innovation Agency (ARIS) P6-0411 "Language Resources and Technologies for Slovene"

European Union HORIZON-WIDERA-2023-TALENTS-01-01 101186647 "EU Era Chair (AI4DH)"

 Ključne besede

metalinguistic benchmark large-scale multilingual evaluation large language models low-resource languages WALS

 Zbirke

CLARIN.SI data & tools

Prikaži polni zapis vnosa