Ukrainian web corpus MaCoCu-uk 1.0

  Avtorji

Bañón, Marta ; et al.prikaži vse

Bañón, Marta ; Chichirau, Malina ; Esplà-Gomis, Miquel ; Forcada, Mikel L. ; Galiano-Jiménez, Aarón ; García-Romero, Cristian ; Kuzman, Taja ; Ljubešić, Nikola ; van Noord, Rik ; Pla Sempere, Leopoldo ; Ramírez-Sánchez, Gema ; Rupnik, Peter ; Suchomel, Vít ; Toral, Antonio ; Zaragoza-Bernabeu, Jaume

  Identifikator vnosa

http://hdl.handle.net/11356/1838

 URL projekta

https://macocu.eu/

 Dokumentirano v

https://hdl.handle.net/11370/685514a8-947e-44f9-83cf-90356c5f1684

 Datum objave

2023-05-24

 Vrsta

corpus, text

 Velikost

21471613 texts, 6181945683 words

 Jezik(i)

Ukrainian

 Opis

The Ukrainian web corpus MaCoCu-uk 1.0 was built by crawling the ".ua" and ".укр" internet top-level domains in 2022, extending the crawl dynamically to other domains as well. The crawler is available at https://github.com/macocu/MaCoCu-crawler. Considerable effort was devoted into cleaning the extracted text to provide a high-quality web corpus. This was achieved by removing boilerplate (https://corpus.tools/wiki/Justext) and near-duplicated paragraphs (https://corpus.tools/wiki/Onion), discarding very short texts as well as texts that are not in the target language. The dataset is characterized by extensive metadata which allows filtering the dataset based on text quality and other criteria (https://github.com/bitextor/monotextor), making the corpus highly useful for corpus linguistics studies, as well as for training language models and other language technologies. In XML format, each document is accompanied by the following metadata: title, crawl date, url, domain, file type of the original document, distribution of languages inside the document, and a fluency score based on a language model. The text of each document is divided into paragraphs that are accompanied by metadata on the information whether a paragraph is a heading or not, metadata on the paragraph quality (labels, such as “short” or “good”, assigned based on paragraph length, URL and stopword density via the jusText tool - https://corpus.tools/wiki/Justext) and fluency (score between 0 and 1, assigned with the Monocleaner tool - https://github.com/bitextor/monocleaner), the automatically identified language of the text in the paragraph, and information whether the paragraph contains sensitive information (identified via the Biroamer tool - https://github.com/bitextor/biroamer). The corpus can be easily read with the prevert parser (https://pypi.org/project/prevert/). Notice and take down: Should you consider that our data contains material that is owned by you and should therefore not be reproduced here, please: (1) Clearly identify yourself, with detailed contact data such as an address, telephone number or email address at which you can be contacted. (2) Clearly identify the copyrighted work claimed to be infringed. (3) Clearly identify the material that is claimed to be infringing and information reasonably sufficient in order to allow us to locate the material. (4) Please write to the contact person for this resource whose email is available in the full item record. We will comply with legitimate requests by removing the affected sources from the next release of the corpus. This action has received funding from the European Union's Connecting Europe Facility 2014-2020 - CEF Telecom, under Grant Agreement No. INEA/CEF/ICT/A2020/2278341. This communication reflects only the author’s view. The Agency is not responsible for any use that may be made of the information it contains. A newer version of the corpus is available as part of the MaCoCu-Genre corpora collection at http://hdl.handle.net/11356/1969. The main novelty of the MaCoCu-Genre version is that the texts have been automatically annotated with genre categories. Additionally, the corpus underwent additional post-processing and has been transformed to the JSONL format.

 Izdajatelj

Jožef Stefan Institute

 

Prompsit

 

Rijksuniversiteit Groningen

 

Universitat d'Alacant

 Zahvala

Connecting Europe Facility (CEF) Telecom INEA/CEF/ICT/A2020/2278341 "MaCoCu - Massive collection and curation of monolingual and bilingual data: focus on under-resourced languages"

ARRS (Slovenian Research Agency) P6-0411 "Language Resources and Technologies for Slovene"

Jožef Stefan Institute CLARIN "CLARIN.SI"

 Ključne besede

web corpus

 Zbirke

CLARIN.SI data & tools

Prikaži polni zapis vnosa