A Multilingual Benchmark for Evaluating Metalinguistic Knowledge WALS-Bench 1.0

Name: A Multilingual Benchmark for Evaluating Metalinguistic Knowledge WALS-Bench 1.0
License: https://creativecommons.org/licenses/by/4.0/

Arčon, Tjaša; Klemen, Matej; Robnik-Šikonja, Marko; Dobrovoljc, Kaja; Terčon, Luka

Show simple item record

dc.contributor.author	Arčon, Tjaša
dc.contributor.author	Klemen, Matej
dc.contributor.author	Robnik-Šikonja, Marko
dc.contributor.author	Dobrovoljc, Kaja
dc.contributor.author	Terčon, Luka
dc.date.accessioned	2026-03-03T07:58:22Z
dc.date.available	2026-03-03T07:58:22Z
dc.date.issued	2026-03-03
dc.identifier.uri	http://hdl.handle.net/11356/2083
dc.description	This is a large-scale multilingual benchmark for evaluating metalinguistic knowledge (i.e. explicit knowledge about the structure of languages) in large language models using grammatical features from the World Atlas of Language Structures (WALS). The benchmark covers 192 linguistic features across 12 linguistic domains and 2,660 languages and is available in two formats (jsonl files): - Format 1 (192-question version): One question per feature, under which all languages with a corresponding ground truth value for that feature are listed. - Format 2 (76,475-question version): One question per feature-language pair with a corresponding ground truth value, fully expanded across all languages. The original WALS data is licensed under CC BY 4.0. The data has been adapted for use in this benchmark. Source: Dryer, Matthew S. & Haspelmath, Martin (eds.). World Atlas of Language Structures Online. Max Planck Institute for Evolutionary Anthropology. https://wals.info
dc.language.iso	eng
dc.language.iso	mul
dc.publisher	Faculty of Computer and Information Science, University of Ljubljana
dc.relation.isreferencedby	https://arxiv.org/abs/2602.02182
dc.rights	Creative Commons - Attribution 4.0 International (CC BY 4.0)
dc.rights.uri	https://creativecommons.org/licenses/by/4.0/
dc.rights.label	PUB
dc.subject	metalinguistic benchmark
dc.subject	large-scale multilingual evaluation
dc.subject	large language models
dc.subject	low-resource languages
dc.subject	WALS
dc.title	A Multilingual Benchmark for Evaluating Metalinguistic Knowledge WALS-Bench 1.0
dc.type	corpus
metashare.ResourceInfo#ContentInfo.mediaType	text
has.files	yes
branding	CLARIN.SI data & tools
contact.person	Tjaša Arčon tjasa.arcon@fri.uni-lj.si Faculty of Computer and Information Science, University of Ljubljana
sponsor	ARIS (Slovenian Research and Innovation Agency) GC-0002 LLM4DH: Large Language Models for Digital Humanities nationalFunds
sponsor	The Slovenian Research and Innovation Agency (ARIS) P6-0411 Language Resources and Technologies for Slovene nationalFunds
sponsor	European Union HORIZON-WIDERA-2023-TALENTS-01-01 101186647 EU Era Chair (AI4DH) euFunds
size.info	192 entries
size.info	76475 entries
files.count	1
files.size	1836339