Slovenian RoBERTa contextual embeddings model: SloBERTa 2.0

  Avtorji

Ulčar, Matej and Robnik-Šikonja, Marko

  Identifikator vnosa

http://hdl.handle.net/11356/1397

 URL projekta

https://rsdo.slovenscina.eu/en/semantic-resources-and-technologies

 Datum objave

2021-01-17

 Vrsta

toolService

 Jezik(i)

Slovenian

 Opis

The monolingual Slovene RoBERTa (A Robustly Optimized Bidirectional Encoder Representations from Transformers) model is a state-of-the-art model representing words/tokens as contextually dependent word embeddings, used for various NLP tasks. Word embeddings can be extracted for every word occurrence and then used in training a model for an end task, but typically the whole RoBERTa model is fine-tuned end-to-end. SloBERTa model is closely related to French Camembert model https://camembert-model.fr/. The corpora used for training the model have 3.47 billion tokens in total. The subword vocabulary contains 32,000 tokens. The scripts and programs used for data preparation and training the model are available on https://github.com/clarinsi/Slovene-BERT-Tool Compared with the previous version (1.0), this version was trained for further 61 epochs (v1.0 37 epochs, v2.0 98 epochs), for a total of 200,000 iterations/updates. The released model here is a pytorch neural network model, intended for usage with the transformers library https://github.com/huggingface/transformers (sloberta.2.0.transformers.tar.gz) or fairseq library https://github.com/pytorch/fairseq (sloberta.2.0.fairseq.tar.gz)

 Izdajatelj

Faculty of Computer and Information Science, University of Ljubljana

 Zahvala

European Union EC/H2020/825153 "EMBEDDIA - Cross-Lingual Embeddings for Less-Represented Languages in European News Media"

Ministry of Culture C3340-20-278001 "Development of Slovene in a Digital Environment"

 Ključne besede

BERT RoBERTa word embeddings language model contextual embeddings

 Zbirke

CLARIN.SI data & tools

 Druge različice

Seznam vseh različic  

Prikaži polni zapis vnosa