Slovenian RoBERTa contextual embeddings model: SloBERTa 1.0

Ulčar, Matej; Robnik-Šikonja, Marko

Slovenian RoBERTa contextual embeddings model: SloBERTa 1.0

CLARIN.SI data & tools

Authors: Ulčar, Matej ; Robnik-Šikonja, Marko

Item identifier: http://hdl.handle.net/11356/1387

Project URL: https://slovenscina.eu/semanticni-viri-tehnologije

Date issued: 2020-12-29

Type: toolService

Language(s): Slovenian

Description: The monolingual Slovene RoBERTa (A Robustly Optimized Bidirectional Encoder Representations from Transformers) model is a state-of-the-art model representing words/tokens as contextually dependent word embeddings, used for various NLP tasks. Word embeddings can be extracted for every word occurrence and then used in training a model for an end task, but typically the whole RoBERTa model is fine-tuned end-to-end. SloBERTa model is closely related to French Camembert model https://camembert-model.fr/. The corpora used for training the model have 3.47 billion tokens in total. The subword vocabulary contains 32,000 tokens. The scripts and programs used for data preparation and training the model are available on https://github.com/clarinsi/Slovene-BERT-Tool The released model here is a pytorch neural network model, intended for usage with the transformers library https://github.com/huggingface/transformers.

Publisher: Faculty of Computer and Information Science, University of Ljubljana

Subject(s): BERT RoBERTa word embeddings language model contextual embeddings

Collection(s): CLARIN.SI data & tools

Show full item record

Files in this item

Download all files in item (423.5 MB)

This item is

Publicly Available

and licensed under:
Creative Commons - Attribution-ShareAlike 4.0 International (CC BY-SA 4.0)

Name: config.json
Size: 520 bytes
Format: Unknown
Description: Configuration file, describing the model's architecture
MD5: 00189fa49a298a01e689fafbffff9fb5

Download file

Name: pytorch_model.bin
Size: 422.32 MB
Format: Unknown
Description: SloBERTa model
MD5: e84c8114553e35003c8b5777534c8d4a

Download file

Name: sentencepiece.bpe.model
Size: 781.26 KB
Format: Unknown
Description: Sentencepiece model tokenizer
MD5: e4acd9c398b3fc68e1b2bc0f27d0d409

Download file

Name: dict.txt
Size: 424.06 KB
Format: Text file
Description: Sentencepiece subword token vocabulary
MD5: c3988aad6b209cf7f61667bbf2aba173

Download file Preview

File Preview

999 #fairseq:overwrite ~~999 #fairseq:overwrite~~ 999 #fairseq:overwrite ▁p 999 ▁s 999 je 999 na 999 ni 999 ra 999 ▁v 999 re 999 ▁d 999 st 999 ▁i 999 ne 999 ▁z 999 ko 999 no 999 ▁po 999 ▁o 999 ▁t 999 li 999 ja 999 ri 999 ▁na 999 la 999 ▁k 999 lo 999 me 999 ▁in 999 le 999 ro 999 va 999 ▁za 999 ▁je 999 ve 999 te 999 ▁b 999 ti 999 mo 999 ga 999 vi 999 di 999 ka 999 ma 999 ▁se 999 jo 999 ji 999 vo 999 nje 999 ci 999 da 999 to 999 go 999 ▁pre 999 po 999 ta 999 mi 999 ▁pri 999 se 999 ▁u 999 ke 999 ki 999 ▁da 999 ▁ko 999 de 999 ▁do 999 ▁te 999 ▁so 999 če 999 ▁ne 999 ce 999 ▁iz 999 či 999 ju 999 sti 999 ▁bi 999 ▁a 999 ▁pa 999 ▁ra 999 ▁P 999 ▁ki 999 ▁ka 999 ▁od 999 ▁ob 999 nih 999 ▁de 999 ▁S 999 ča 999 ru 999 ▁ve 999 ▁bo 999 do 999 sta 999 ▁š 999 nja 999 ▁( 999 ▁me 999 ▁1 999 že 999 ▁mo 999 ▁raz 999 ku 999 ▁tu 999 bi 999 lja 999 ▁e 999 ▁V 999 ▁2 999 ▁pro 999 ▁le 999 lje 999 ns 999 be 999 sa 999 sto 999 zi 999 za 999 ▁N 999 ▁T 999 ▁ni 999 pa 999 bo 999 ▁ta 999 pi 999 ▁K 999 ▁tudi 999 . . .

Slovenian RoBERTa contextual embeddings model: SloBERTa 1.0

Files in this item

Partners

Partners

Repository