2026-06-25T12:16:40Zhttp://www.clarin.si/repository/oai/request

oai:www.clarin.si:11356/13872022-12-06T15:24:02Zhdl_11356_1023hdl_11356_1024

Slovenian RoBERTa contextual embeddings model: SloBERTa 1.0 Ulčar, Matej Robnik-Šikonja, Marko BERT RoBERTa word embeddings language model contextual embeddings The monolingual Slovene RoBERTa (A Robustly Optimized Bidirectional Encoder Representations from Transformers) model is a state-of-the-art model representing words/tokens as contextually dependent word embeddings, used for various NLP tasks. Word embeddings can be extracted for every word occurrence and then used in training a model for an end task, but typically the whole RoBERTa model is fine-tuned end-to-end. SloBERTa model is closely related to French Camembert model https://camembert-model.fr/. The corpora used for training the model have 3.47 billion tokens in total. The subword vocabulary contains 32,000 tokens. The scripts and programs used for data preparation and training the model are available on https://github.com/clarinsi/Slovene-BERT-Tool The released model here is a pytorch neural network model, intended for usage with the transformers library https://github.com/huggingface/transformers. 2020-12-29 toolService http://hdl.handle.net/11356/1387 slv info:eu-repo/grantAgreement/EC/H2020/825153 http://hdl.handle.net/11356/1397 The MIT License (MIT) https://opensource.org/licenses/mit-license.php PUB text/plain; charset=utf-8 application/octet-stream application/octet-stream application/octet-stream text/plain downloadable_files_count: 4 Faculty of Computer and Information Science, University of Ljubljana https://rsdo.slovenscina.eu/en/semantic-resources-and-technologies