Finnish web corpus fiWaC 1.0

Finnish web corpus fiWaC 1.0

CLARIN.SI data & tools

Authors: Ljubešić, Nikola ; Pirinen, Tommi and Toral, Antonio

Item identifier: http://hdl.handle.net/11356/1074

Project URL: https://cordis.europa.eu/project/id/324414

Date issued: 2016-09-20

Type: corpus, text

Size: 3712935412 tokens, 40 gb

Language(s): English , Finnish

Description: The Finnish web corpus fiWaC was built by crawling the .fi top-level domain in 2015 for both Finnish and English documents. The corpus was naively tokenised (via spaces), near-deduplicated on paragraph level and paragraph-shuffled. Each paragraph contains metadata on the URL and language identification. The Finnish (~1.7B tokens) and English (~2B tokens) parts of the corpus are organised in separate files.

Publisher: Jožef Stefan Institute

Subject(s): web corpus

Collection(s): CLARIN.SI data & tools

Show full item record

Files in this item

This item is

Publicly Available

and licensed under:
Creative Commons - Attribution-ShareAlike 4.0 International (CC BY-SA 4.0)

Name: fiWaC1.0.fi.01.gz
Size: 518.45 MB
Format: application/gzip
Description: Unknown
MD5: 54d2687ed77e11c81ca5c9cd0076fa8a

Name: fiWaC1.0.fi.02.gz
Size: 503.96 MB
Format: application/gzip
Description: Unknown
MD5: 8b1109a417f81d72183ad6a02e237713

Name: fiWaC1.0.fi.03.gz
Size: 509 MB
Format: application/gzip
Description: Unknown
MD5: 1a3cf0d4c86587d12be435d19c38005a

Name: fiWaC1.0.fi.04.gz
Size: 509.95 MB
Format: application/gzip
Description: Unknown
MD5: 7422d1836e106cff45dc2e6cccd3cd6d

Name: fiWaC1.0.fi.05.gz
Size: 504.25 MB
Format: application/gzip
Description: Unknown
MD5: 1048089a1b3805119241b632e6dbebf9

Name: fiWaC1.0.fi.06.gz
Size: 505.2 MB
Format: application/gzip
Description: Unknown
MD5: b63263b18c9a41bf5d26aad74aafdf59

Name: fiWaC1.0.fi.07.gz
Size: 507.24 MB
Format: application/gzip
Description: Unknown
MD5: 244cce429fcd85c9aa5dc339883bb6ae

Name: fiWaC1.0.fi.08.gz
Size: 509.71 MB
Format: application/gzip
Description: Unknown
MD5: a396a3b5f313e5ebcd7a1c6720dd4c7a

Name: fiWaC1.0.fi.09.gz
Size: 510.57 MB
Format: application/gzip
Description: Unknown
MD5: 2bcdf99dabc3822ac5251d1d33311e15

Name: fiWaC1.0.fi.10.gz
Size: 509.84 MB
Format: application/gzip
Description: Unknown
MD5: 1d3fec37c169645156c460c56c9d22af

Name: fiWaC1.0.fi.11.gz
Size: 508.63 MB
Format: application/gzip
Description: Unknown
MD5: 38866998c01a4b424ee2f1c4eb447974

Name: fiWaC1.0.fi.12.gz
Size: 509.91 MB
Format: application/gzip
Description: Unknown
MD5: c81f7156366c9dce543bd7d324384f20

Name: fiWaC1.0.fi.13.gz
Size: 509.88 MB
Format: application/gzip
Description: Unknown
MD5: 9707a1ba5bd664f013a405548c819eea

Name: fiWaC1.0.fi.14.gz
Size: 511.85 MB
Format: application/gzip
Description: Unknown
MD5: 4fd27832b816c026da6d150a7704a1de

Name: fiWaC1.0.fi.15.gz
Size: 513.11 MB
Format: application/gzip
Description: Unknown
MD5: f14fbcfd70fd288a28425710ae9a9036

Name: fiWaC1.0.fi.16.gz
Size: 513.21 MB
Format: application/gzip
Description: Unknown
MD5: 72f0d36848ffca67aaee94e65f84c426

Name: fiWaC1.0.fi.17.gz
Size: 511.61 MB
Format: application/gzip
Description: Unknown
MD5: 01d78b4462397626186c2355fe308c8e

Name: fiWaC1.0.fi.18.gz
Size: 83.05 MB
Format: application/gzip
Description: Unknown
MD5: 2d1e404952fd701b622c32a426ccd7b4

Name: fiWaC1.0.en.01.gz
Size: 352.26 MB
Format: application/gzip
Description: Unknown
MD5: 04cc97ebc2f3e1de2439a862b243c982

Name: fiWaC1.0.en.02.gz
Size: 346.61 MB
Format: application/gzip
Description: Unknown
MD5: d65b9495f8cf4b38de3366502c424f42

Name: fiWaC1.0.en.03.gz
Size: 345.92 MB
Format: application/gzip
Description: Unknown
MD5: 844b5b18161b9a43c841499c7706b3b0

Name: fiWaC1.0.en.04.gz
Size: 346.46 MB
Format: application/gzip
Description: Unknown
MD5: f6054dab1400845eb9e2aa36c54d6578

Name: fiWaC1.0.en.05.gz
Size: 345.97 MB
Format: application/gzip
Description: Unknown
MD5: a0792bde80d13e5d0c0259a1a367ce91

Name: fiWaC1.0.en.06.gz
Size: 345.65 MB
Format: application/gzip
Description: Unknown
MD5: fecd25346f95f7c9058f700201bd6427

Name: fiWaC1.0.en.07.gz
Size: 347.65 MB
Format: application/gzip
Description: Unknown
MD5: a52e89887465aad23a3654005a9bb2d3

Name: fiWaC1.0.en.08.gz
Size: 346.02 MB
Format: application/gzip
Description: Unknown
MD5: 958c03be7676ab745cbae86312d806f4

Name: fiWaC1.0.en.09.gz
Size: 345.27 MB
Format: application/gzip
Description: Unknown
MD5: 3dceca505227ace9214462bb60bd8fa8

Name: fiWaC1.0.en.10.gz
Size: 345.07 MB
Format: application/gzip
Description: Unknown
MD5: 3c96a8e250249d084b8884778e6d07b4

Name: fiWaC1.0.en.11.gz
Size: 346.79 MB
Format: application/gzip
Description: Unknown
MD5: 2be5f6f9342c1d00dd41a8671ef14ccb

Name: fiWaC1.0.en.12.gz
Size: 344.65 MB
Format: application/gzip
Description: Unknown
MD5: 89eeaf563833d417504d1478d06cf587

Name: fiWaC1.0.en.13.gz
Size: 345.25 MB
Format: application/gzip
Description: Unknown
MD5: 029d0c081bc7bd95626756cb6bda7169

Name: fiWaC1.0.en.14.gz
Size: 344.65 MB
Format: application/gzip
Description: Unknown
MD5: cc4b3cbee8eaf66063a70c90ce76d2d5

Name: fiWaC1.0.en.15.gz
Size: 343.31 MB
Format: application/gzip
Description: Unknown
MD5: b2619936bf77d2377e585426687df31f

Name: fiWaC1.0.en.16.gz
Size: 345.32 MB
Format: application/gzip
Description: Unknown
MD5: 0664c46efca2d835cae5f5e4fed7d6b8

Name: fiWaC1.0.en.17.gz
Size: 346.1 MB
Format: application/gzip
Description: Unknown
MD5: 0d82bea40c7f62d19e0a6806aecb6b95

Name: fiWaC1.0.en.18.gz
Size: 344.83 MB
Format: application/gzip
Description: Unknown
MD5: 08a4aca4bd0270d5a06ecdbfd48ac855

Name: fiWaC1.0.en.19.gz
Size: 343 MB
Format: application/gzip
Description: Unknown
MD5: e8a810ad2983f43f26d6aa15caa1e16a

Name: fiWaC1.0.en.20.gz
Size: 328.81 MB
Format: application/gzip
Description: Unknown
MD5: a4b0310e982c0a6b2f96f47b83738853