A Multilingual Benchmark for Evaluating Metalinguistic Knowledge WALS-Bench 1.0

  Authors

Arčon, Tjaša ; Klemen, Matej ; Robnik-Šikonja, Marko ; Dobrovoljc, Kaja and Terčon, Luka

  Item identifier

http://hdl.handle.net/11356/2083

 Referenced by

https://arxiv.org/abs/2602.02182

 Date issued

2026-03-03

 Type

corpus, text

 Size

192 entries, 76475 entries

 Language(s)

English , Multiple languages

 Description

This is a large-scale multilingual benchmark for evaluating metalinguistic knowledge (i.e. explicit knowledge about the structure of languages) in large language models using grammatical features from the World Atlas of Language Structures (WALS). The benchmark covers 192 linguistic features across 12 linguistic domains and 2,660 languages and is available in two formats (jsonl files): - Format 1 (192-question version): One question per feature, under which all languages with a corresponding ground truth value for that feature are listed. - Format 2 (76,475-question version): One question per feature-language pair with a corresponding ground truth value, fully expanded across all languages. The original WALS data is licensed under CC BY 4.0. The data has been adapted for use in this benchmark. Source: Dryer, Matthew S. & Haspelmath, Martin (eds.). World Atlas of Language Structures Online. Max Planck Institute for Evolutionary Anthropology. https://wals.info

 Publisher

Faculty of Computer and Information Science, University of Ljubljana

 Acknowledgement

ARIS (Slovenian Research and Innovation Agency) GC-0002 "LLM4DH: Large Language Models for Digital Humanities"

The Slovenian Research and Innovation Agency (ARIS) P6-0411 "Language Resources and Technologies for Slovene"

European Union HORIZON-WIDERA-2023-TALENTS-01-01 101186647 "EU Era Chair (AI4DH)"

 Subject(s)

metalinguistic benchmark large-scale multilingual evaluation large language models low-resource languages WALS

 Collection(s)

CLARIN.SI data & tools

Show full item record