{"id":5162,"date":"2021-06-28T06:52:37","date_gmt":"2021-06-28T06:52:37","guid":{"rendered":"https:\/\/www.clarin.si\/info\/?page_id=5162"},"modified":"2021-06-28T06:54:18","modified_gmt":"2021-06-28T06:54:18","slug":"macocu-massive-collection-and-curation-of-monolingual-and-bilingual-data","status":"publish","type":"page","link":"https:\/\/www.clarin.si\/info\/macocu-massive-collection-and-curation-of-monolingual-and-bilingual-data\/","title":{"rendered":"MaCoCu: Massive Collection and Curation of Monolingual and Bilingual Data"},"content":{"rendered":"

MaCoCu<\/h1>\n

The aim of MaCoCu, a CEF<\/a>-funded project, is to collect, curate and enrich monolingual and parallel data from the Internet for 12 under-resourced languages of EU member states and candidate states: Albanian, Bosnian, Bulgarian, Croatian, Greek, Icelandic, Macedonian, Maltese, Montenegrin, Serbian, Slovenian, and Turkish. The collection of monolingual data is performed by Jo\u017eef Stefan Institute<\/a>, Ljubljana, Slovenia.<\/p>\n

Web crawling<\/h2>\n

We run a web crawler<\/a> to download the texts from the Web. The software we use is SpiderLing<\/a> developed by the\u00a0Natural Language Processing Centre<\/a> at Masaryk University, Czech Republic.<\/p>\n

What do we do with the downloaded data?<\/h2>\n

We are interested in a language use rather than the content of the downloaded texts. The retrieved text will be cleaned, de-duplicated and annotated with text type information. Text corpora<\/a> for computational linguistics research<\/a> and language models<\/a> for natural language processing<\/a> tasks will be built using the data.<\/p>\n

What if I don’t want my website to be crawled?<\/h2>\n

Our crawler adheres to the Robots exclusion standard<\/a>. You can restrict the access to some or all of the pages on your website by creating a robots.txt file. The user-agent identification of our crawler is MaCoCu<\/tt>. This is what to include in your robots.txt if you want to prevent our crawler from crawling your website:<\/p>\n

User-agent: MaCoCu\r\nDisallow: \/<\/pre>\n
Please note the crawler reads your robots.txt the first time it accesses your site so any changes will be effective the next time the crawler is run, not immediately.<\/p>\n
Contacts<\/h2>\n
    \n
  • V\u00edt Suchomel, vit.suchomel at sketchengine dot eu<\/li>\n
  • Nikola Ljube\u0161i\u0107, nljubesi at gmail dot com<\/li>\n<\/ul>\n\n
    \n    <\/div>\n\n","protected":false},"excerpt":{"rendered":"
    MaCoCu The aim of MaCoCu, a CEF-funded project, is to collect, curate and enrich monolingual and parallel data from the Internet for 12 under-resourced languages of EU member states and candidate states: Albanian, Bosnian, Bulgarian, Croatian, Greek, Icelandic, Macedonian, Maltese, Montenegrin, Serbian, Slovenian, and Turkish. The collection of monolingual data is performed by Jo\u017eef Stefan […]<\/p>\n","protected":false},"author":13,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"footnotes":""},"class_list":["post-5162","page","type-page","status-publish","hentry","has-post-title","has-post-date","has-post-category","has-post-tag","has-post-comment","has-post-author",""],"aioseo_notices":[],"aioseo_head":"\n\t\t\n\t\n\t\n\t\n\t\n\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t