Show simple item record

 
dc.contributor.author Bañón, Marta
dc.contributor.author Chichirau, Malina
dc.contributor.author Esplà-Gomis, Miquel
dc.contributor.author Forcada, Mikel L.
dc.contributor.author Galiano-Jiménez, Aarón
dc.contributor.author García-Romero, Cristian
dc.contributor.author Kuzman, Taja
dc.contributor.author Ljubešić, Nikola
dc.contributor.author van Noord, Rik
dc.contributor.author Pla Sempere, Leopoldo
dc.contributor.author Ramírez-Sánchez, Gema
dc.contributor.author Rupnik, Peter
dc.contributor.author Suchomel, Vít
dc.contributor.author Toral, Antonio
dc.contributor.author Zaragoza-Bernabeu, Jaume
dc.date.accessioned 2023-04-25T13:49:18Z
dc.date.available 2023-04-25T13:49:18Z
dc.date.issued 2023-04-26
dc.identifier.uri http://hdl.handle.net/11356/1814
dc.description The Croatian-English parallel corpus MaCoCu-hr-en 2.0 was built by crawling the “.hr” internet top-level domain in 2021 and 2022, extending the crawl dynamically to other domains as well. All the crawling process was carried out by the MaCoCu crawler (https://github.com/macocu/MaCoCu-crawler). Websites containing documents in both target languages were identified and processed using the tool Bitextor (https://github.com/bitextor/bitextor). Considerable effort was devoted into cleaning the extracted text to provide a high-quality parallel corpus. This was achieved by removing boilerplate and near-duplicated paragraphs and documents that are not in one of the targeted languages. Document and segment alignment as implemented in Bitextor were carried out, and Bifixer (https://github.com/bitextor/bifixer) and BicleanerAI (https://github.com/bitextor/bicleaner-ai) were used for fixing, cleaning, and deduplicating the final version of the corpus. The corpus is available in three formats: two sentence-level formats, TXT and TMX, and a document-level TXT format. TMX is an XML-based format and TXT is a tab-separated format. They both consist of pairs of source and target segments (one or several sentences) and additional metadata. The following metadata is included in both sentence-level formats: - source and target document URL; - paragraph ID which includes information on the position of the sentence in the paragraph and in the document (e.g., “p35:77s1/3” which means “paragraph 35 out of 77, sentence 1 out of 3”); - quality score as provided by the tool Bicleaner AI (a likelihood of a pair of sentences being mutual translations, provided with a score between 0 and 1); - similarity score as provided by the sentence alignment tool Bleualign (value between 0 and 1); - personal information identification (“biroamer-entities-detected”): segments containing personal information are flagged, so final users of the corpus can decide whether to use these segments; - translation direction and machine translation identification (“translation-direction”): the source segment in each segment pair was identified by using a probabilistic model (https://github.com/RikVN/TranslationDirection), which also determines if the translation has been produced by a machine-translation system; - a DSI class (“dsi”): information whether the segment is connected to any of Digital Service Infrastructure (DSI) classes (e.g., cybersecurity, e-health, e-justice, open-data-portal), defined by the Connecting Europe Facility (https://github.com/RikVN/DSI); - English language variant: the language variant of English (British or American, using a lexicon-based English variety classifier - https://pypi.org/project/abclf/) was identified on document and domain level. Furthermore, the sentence-level TXT format provides additional metadata: - web domain of the text; - source and target document title; - the date when the original file was retrieved; - the original type of the file (e.g., “html”), from which the sentence was extracted; - paragraph quality (labels, such as “short” or “good”, assigned based on paragraph length, URL and stopword density via the jusText tool - https://corpus.tools/wiki/Justext); - information whether the sentence is a heading or not in the original document. The document-level TXT format provides pairs of documents identified to contain parallel data. In addition to the parallel documents (in base64 format), the corpus includes the following metadata: source and target document URL, a DSI category and the English language variant (British or American). As opposed to the previous version, this version has more accurate metadata on languages of the texts, which was achieved by using Google's Compact Language Detector 2 (CLD2) (https://github.com/CLD2Owners/cld2), a high-performance language detector supporting many languages. Other tools, used for web corpora creation and curation, have been updated as well, resulting in an even cleaner corpus. The new version also provides additional metadata, such as the position of the sentence in the paragraph and document, and information whether the sentence is related to a DSI. Moreover, the corpus is now also provided in a document-level format. Notice and take down: Should you consider that our data contains material that is owned by you and should therefore not be reproduced here, please: (1) Clearly identify yourself, with detailed contact data such as an address, telephone number or email address at which you can be contacted. (2) Clearly identify the copyrighted work claimed to be infringed. (3) Clearly identify the material that is claimed to be infringing and information reasonably sufficient in order to allow us to locate the material. (4) Please write to the contact person for this resource whose email is available in the full item record. We will comply with legitimate requests by removing the affected sources from the next release of the corpus. This action has received funding from the European Union's Connecting Europe Facility 2014-2020 - CEF Telecom, under Grant Agreement No. INEA/CEF/ICT/A2020/2278341. This communication reflects only the author’s view. The Agency is not responsible for any use that may be made of the information it contains.
dc.language.iso hrv
dc.language.iso eng
dc.publisher Jožef Stefan Institute
dc.publisher Prompsit
dc.publisher Rijksuniversiteit Groningen
dc.publisher Universitat d'Alacant
dc.relation.isreferencedby https://hdl.handle.net/11370/685514a8-947e-44f9-83cf-90356c5f1684
dc.relation.replaces http://hdl.handle.net/11356/1522
dc.rights CC0-No Rights Reserved
dc.rights.uri https://creativecommons.org/publicdomain/zero/1.0/
dc.rights.label PUB
dc.source.uri https://macocu.eu/
dc.subject web corpus
dc.subject parallel corpus
dc.subject multilingual
dc.title Croatian-English parallel corpus MaCoCu-hr-en 2.0
dc.type corpus
metashare.ResourceInfo#ContentInfo.mediaType text
has.files yes
branding CLARIN.SI data & tools
contact.person Miquel Esplà-Gomis mespla@dlsi.ua.es Universitat d’Alacant
sponsor Connecting Europe Facility (CEF) Telecom INEA/CEF/ICT/A2020/2278341 MaCoCu - Massive collection and curation of monolingual and bilingual data: focus on under-resourced languages Other
sponsor ARRS (Slovenian Research Agency) P6-0411 Language Resources and Technologies for Slovene nationalFunds
sponsor Jožef Stefan Institute CLARIN CLARIN.SI nationalFunds
size.info 2266004 entries
size.info 99500221 words
size.info 332271 texts
files.count 3
files.size 2602028904


 Files in this item

This item is
Publicly Available
and licensed under:
CC0-No Rights Reserved
Icon
Name
MaCoCu-hr-en.doc.txt.gz
Size
1.12 GB
Format
application/gzip
Description
Corpus in document-level TXT format
MD5
8f8edebcd478a6a1d5ffbcca2ed60683
 Download file
Icon
Name
MaCoCu-hr-en.sent.txt.gz
Size
690.51 MB
Format
application/gzip
Description
Corpus in sentence-level TXT format
MD5
d5c275d19790ba5b6c0c85c56d85e64b
 Download file
Icon
Name
MaCoCu-hr-en.tmx.gz
Size
648.19 MB
Format
application/gzip
Description
Corpus in sentence-level TMX format
MD5
9d0e356243752c93eeed65d5cb0b6152
 Download file

Show simple item record