Agreement report for both annotation questions: hate speech type (vrsta) and target (tarča) for the data in the file IMSyPP_SI_anotacije_training-clarin.csv. Hate speech types (vrsta): 0 appropriate (ni sporni govor) 1 inappropriate (nespodobni govor) 2 offensive (žalitev) 3 violent (nasilje) Hate speech targets (tarča): 1 racism (ksenofobija in rasizem) 2 migrants (begunci/migranti) 3 islamophobia (islamofobija) 4 antisemitism (antisemitizem) 5 religion (druge religije) 6 homophobia (homofobija) 7 sexism (seksizem) 8 ideology (ideologija) 9 media (novinarji in mediji) 10 politics (politika/-i) 11 individual (posameznik) 12 other (drugo) Annotated instances a0 9997 / 10000 a1 9950 / 10000 a2 9929 / 10000 a3 9992 / 10000 a4 10000 / 10000 a5 10000 / 10000 a6 9998 / 10000 a7 9979 / 10000 a8 9973 / 10000 a9 9991 / 10000 ----------------- -----OVERALL----- ----------------- Annotated for vrsta : 99809 vrsta 0 ni sporni govor 60969 1 nespodobni govor 3819 2 žalitev 34258 3 nasilje 763 ----------------------------------- -----AGGREGATED INTER-AGREEMENT----- ----------------------------------- Aggregated inter-agreement on 'vrsta' Number of duplicates: 44831 ||0 ni sporni govor |1 nespodobni govor |2 žalitev |3 nasilje ||sum 0 ni sporni govor || 23217 | 0 | 0 | 0 || 23217 1 nespodobni govor || 782 | 846 | 0 | 0 || 1628 2 žalitev || 7618 | 964 | 10938 | 0 || 19520 3 nasilje || 0 | 18 | 228 | 220 || 466 ------------------------------------------------------------- sum || 31617 | 1828 | 11166 | 220 || 44831 Labels: ['0 ni sporni govor', '1 nespodobni govor', '2 žalitev', '3 nasilje'] [[23217 416 4573 0] [ 366 846 578 10] [ 3045 386 10938 106] [ 0 8 122 220]] Agreement accuracy: 0.786 Nominal Kripp. Alpha: 0.577 Interval Kripp. Alpha: 0.606 Ordinal Kripp. Alpha: 0.606 ------------------------ ----------------------------------- -----AGGREGATED SELF-AGREEMENT----- ----------------------------------- Aggregated self-agreement on 'vrsta' Number of duplicates: 4981 ||0 ni sporni govor |1 nespodobni govor |2 žalitev |3 nasilje ||sum 0 ni sporni govor || 2765 | 0 | 0 | 0 || 2765 1 nespodobni govor || 65 | 112 | 0 | 0 || 177 2 žalitev || 451 | 70 | 1471 | 0 || 1992 3 nasilje || 0 | 2 | 19 | 26 || 47 ------------------------------------------------------------- sum || 3281 | 184 | 1490 | 26 || 4981 Labels: ['0 ni sporni govor', '1 nespodobni govor', '2 žalitev', '3 nasilje'] [[2765 29 253 0] [ 36 112 43 1] [ 198 27 1471 9] [ 0 1 10 26]] Agreement accuracy: 0.878 Nominal Kripp. Alpha: 0.760 Interval Kripp. Alpha: 0.787 Ordinal Kripp. Alpha: 0.787 ------------------------ ----------------------------------- -----ANNOTATOR-SPECIFIC------------ ----------------------------------- Annotator-specific agreements on 'vrsta' ----------------------------------------- ------self-agreement---------- a0 annotator-specific self-agreement on vrsta Number of duplicates: 500 Agreement accuracy: 0.944 Nominal Kripp. Alpha: 0.895 Interval Kripp. Alpha: 0.907 Ordinal Kripp. Alpha: 0.908 ------------------------ a1 annotator-specific self-agreement on vrsta Number of duplicates: 493 Agreement accuracy: 0.874 Nominal Kripp. Alpha: 0.707 Interval Kripp. Alpha: 0.749 Ordinal Kripp. Alpha: 0.747 ------------------------ a2 annotator-specific self-agreement on vrsta Number of duplicates: 496 Agreement accuracy: 0.819 Nominal Kripp. Alpha: 0.641 Interval Kripp. Alpha: 0.676 Ordinal Kripp. Alpha: 0.692 ------------------------ a3 annotator-specific self-agreement on vrsta Number of duplicates: 499 Agreement accuracy: 0.842 Nominal Kripp. Alpha: 0.702 Interval Kripp. Alpha: 0.717 Ordinal Kripp. Alpha: 0.716 ------------------------ a4 annotator-specific self-agreement on vrsta Number of duplicates: 500 Agreement accuracy: 0.906 Nominal Kripp. Alpha: 0.807 Interval Kripp. Alpha: 0.809 Ordinal Kripp. Alpha: 0.809 ------------------------ a5 annotator-specific self-agreement on vrsta Number of duplicates: 500 Agreement accuracy: 0.882 Nominal Kripp. Alpha: 0.740 Interval Kripp. Alpha: 0.773 Ordinal Kripp. Alpha: 0.771 ------------------------ a6 annotator-specific self-agreement on vrsta Number of duplicates: 500 Agreement accuracy: 0.888 Nominal Kripp. Alpha: 0.785 Interval Kripp. Alpha: 0.802 Ordinal Kripp. Alpha: 0.803 ------------------------ a7 annotator-specific self-agreement on vrsta Number of duplicates: 498 Agreement accuracy: 0.896 Nominal Kripp. Alpha: 0.752 Interval Kripp. Alpha: 0.771 Ordinal Kripp. Alpha: 0.771 ------------------------ a8 annotator-specific self-agreement on vrsta Number of duplicates: 497 Agreement accuracy: 0.877 Nominal Kripp. Alpha: 0.759 Interval Kripp. Alpha: 0.789 Ordinal Kripp. Alpha: 0.789 ------------------------ a9 annotator-specific self-agreement on vrsta Number of duplicates: 498 Agreement accuracy: 0.853 Nominal Kripp. Alpha: 0.731 Interval Kripp. Alpha: 0.787 Ordinal Kripp. Alpha: 0.788 ------------------------ ----------------------------------------- ------inter-agreement---------- a0 annotator-specific inter-agreement on vrsta Number of duplicates: 8983 Agreement accuracy: 0.805 Nominal Kripp. Alpha: 0.620 Interval Kripp. Alpha: 0.645 Ordinal Kripp. Alpha: 0.645 ------------------------ a1 annotator-specific inter-agreement on vrsta Number of duplicates: 8943 Agreement accuracy: 0.784 Nominal Kripp. Alpha: 0.555 Interval Kripp. Alpha: 0.577 Ordinal Kripp. Alpha: 0.577 ------------------------ a2 annotator-specific inter-agreement on vrsta Number of duplicates: 8924 Agreement accuracy: 0.759 Nominal Kripp. Alpha: 0.530 Interval Kripp. Alpha: 0.568 Ordinal Kripp. Alpha: 0.573 ------------------------ a3 annotator-specific inter-agreement on vrsta Number of duplicates: 8976 Agreement accuracy: 0.788 Nominal Kripp. Alpha: 0.590 Interval Kripp. Alpha: 0.610 Ordinal Kripp. Alpha: 0.611 ------------------------ a4 annotator-specific inter-agreement on vrsta Number of duplicates: 8989 Agreement accuracy: 0.802 Nominal Kripp. Alpha: 0.607 Interval Kripp. Alpha: 0.632 Ordinal Kripp. Alpha: 0.631 ------------------------ a5 annotator-specific inter-agreement on vrsta Number of duplicates: 8975 Agreement accuracy: 0.789 Nominal Kripp. Alpha: 0.556 Interval Kripp. Alpha: 0.583 Ordinal Kripp. Alpha: 0.583 ------------------------ a6 annotator-specific inter-agreement on vrsta Number of duplicates: 8977 Agreement accuracy: 0.789 Nominal Kripp. Alpha: 0.593 Interval Kripp. Alpha: 0.615 Ordinal Kripp. Alpha: 0.615 ------------------------ a7 annotator-specific inter-agreement on vrsta Number of duplicates: 8967 Agreement accuracy: 0.781 Nominal Kripp. Alpha: 0.547 Interval Kripp. Alpha: 0.576 Ordinal Kripp. Alpha: 0.577 ------------------------ a8 annotator-specific inter-agreement on vrsta Number of duplicates: 8960 Agreement accuracy: 0.803 Nominal Kripp. Alpha: 0.615 Interval Kripp. Alpha: 0.646 Ordinal Kripp. Alpha: 0.647 ------------------------ a9 annotator-specific inter-agreement on vrsta Number of duplicates: 8968 Agreement accuracy: 0.756 Nominal Kripp. Alpha: 0.542 Interval Kripp. Alpha: 0.582 Ordinal Kripp. Alpha: 0.581 ------------------------ -------------------------------------------------------------------- tarča ----------------- -----OVERALL----- ----------------- 34216 annotated for tarča Annotated for tarča : 34216 tarča 1 ksenofobija in rasizem 1104 2 begunci/migranti 1012 3 islamofobija 527 4 antisemitizem 55 5 druge religije 172 6 homofobija 304 7 seksizem 772 8 ideologija 6236 9 novinarji in mediji 2516 10 politika/-i 10927 11 posameznik 7028 12 drugo 3563 ----------------------------------- -----AGGREGATED INTER-AGREEMENT----- ----------------------------------- Aggregated inter-agreement on 'tarča' Number of duplicates: 10999 ||1 ksenofobija in rasizem |2 begunci/migranti |3 islamofobija |4 antisemitizem |5 druge religije |6 homofobija |7 seksizem |8 ideologija |9 novinarji in mediji |10 politika/-i |11 posameznik |12 drugo ||sum 1 ksenofobija in rasizem || 229 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 || 229 2 begunci/migranti || 63 | 210 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 || 273 3 islamofobija || 23 | 43 | 123 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 || 189 4 antisemitizem || 4 | 1 | 1 | 9 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 || 15 5 druge religije || 1 | 0 | 3 | 1 | 19 | 0 | 0 | 0 | 0 | 0 | 0 | 0 || 24 6 homofobija || 3 | 0 | 2 | 0 | 1 | 76 | 0 | 0 | 0 | 0 | 0 | 0 || 82 7 seksizem || 5 | 2 | 7 | 0 | 0 | 6 | 124 | 0 | 0 | 0 | 0 | 0 || 144 8 ideologija || 33 | 31 | 25 | 3 | 11 | 12 | 23 | 1268 | 0 | 0 | 0 | 0 || 1406 9 novinarji in mediji || 9 | 5 | 2 | 0 | 1 | 2 | 8 | 85 | 660 | 0 | 0 | 0 || 772 10 politika/-i || 43 | 50 | 11 | 0 | 10 | 17 | 31 | 915 | 143 | 2620 | 0 | 0 || 3840 11 posameznik || 58 | 27 | 12 | 1 | 12 | 22 | 67 | 341 | 120 | 654 | 1289 | 0 || 2603 12 drugo || 72 | 30 | 7 | 4 | 21 | 8 | 20 | 219 | 47 | 280 | 398 | 316 || 1422 ------------------------------------------------------------- sum || 543 | 399 | 193 | 18 | 75 | 143 | 273 | 2828 | 970 | 3554 | 1687 | 316 || 10999 Labels: ['1 ksenofobija in rasizem', '2 begunci/migranti', '3 islamofobija', '4 antisemitizem', '5 druge religije', '6 homofobija', '7 seksizem', '8 ideologija', '9 novinarji in mediji', '10 politika/-i', '11 posameznik', '12 drugo'] [[ 229 38 12 2 1 1 2 20 5 23 31 41] [ 25 210 22 1 0 0 1 15 3 28 11 17] [ 11 21 123 0 2 2 4 15 2 7 6 3] [ 2 0 1 9 1 0 0 0 0 0 0 3] [ 0 0 1 0 19 0 0 3 1 6 5 2] [ 2 0 0 0 1 76 0 8 0 6 12 5] [ 3 1 3 0 0 6 124 13 4 17 41 9] [ 13 16 10 3 8 4 10 1268 40 508 170 96] [ 4 2 0 0 0 2 4 45 660 69 51 22] [ 20 22 4 0 4 11 14 407 74 2620 334 128] [ 27 16 6 1 7 10 26 171 69 320 1289 190] [ 31 13 4 1 19 3 11 123 25 152 208 316]] Agreement accuracy: 0.631 Nominal Kripp. Alpha: 0.537 Interval Kripp. Alpha: 0.645 Ordinal Kripp. Alpha: 0.538 ------------------------ ----------------------------------- -----AGGREGATED SELF-AGREEMENT----- ----------------------------------- Aggregated self-agreement on 'tarča' Number of duplicates: 1475 ||1 ksenofobija in rasizem |2 begunci/migranti |3 islamofobija |4 antisemitizem |5 druge religije |6 homofobija |7 seksizem |8 ideologija |9 novinarji in mediji |10 politika/-i |11 posameznik |12 drugo ||sum 1 ksenofobija in rasizem || 34 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 || 34 2 begunci/migranti || 7 | 38 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 || 45 3 islamofobija || 0 | 2 | 27 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 || 29 4 antisemitizem || 0 | 0 | 0 | 3 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 || 3 5 druge religije || 0 | 0 | 0 | 0 | 7 | 0 | 0 | 0 | 0 | 0 | 0 | 0 || 7 6 homofobija || 0 | 0 | 0 | 0 | 0 | 12 | 0 | 0 | 0 | 0 | 0 | 0 || 12 7 seksizem || 1 | 0 | 0 | 0 | 0 | 1 | 32 | 0 | 0 | 0 | 0 | 0 || 34 8 ideologija || 3 | 2 | 2 | 0 | 1 | 2 | 2 | 237 | 0 | 0 | 0 | 0 || 249 9 novinarji in mediji || 2 | 1 | 0 | 0 | 0 | 0 | 1 | 7 | 81 | 0 | 0 | 0 || 92 10 politika/-i || 3 | 3 | 1 | 0 | 0 | 2 | 2 | 82 | 7 | 378 | 0 | 0 || 478 11 posameznik || 2 | 0 | 0 | 0 | 0 | 1 | 5 | 21 | 10 | 77 | 195 | 0 || 311 12 drugo || 2 | 3 | 0 | 1 | 1 | 1 | 1 | 13 | 2 | 30 | 48 | 79 || 181 ------------------------------------------------------------- sum || 54 | 49 | 30 | 4 | 9 | 19 | 43 | 360 | 100 | 485 | 243 | 79 || 1475 Labels: ['1 ksenofobija in rasizem', '2 begunci/migranti', '3 islamofobija', '4 antisemitizem', '5 druge religije', '6 homofobija', '7 seksizem', '8 ideologija', '9 novinarji in mediji', '10 politika/-i', '11 posameznik', '12 drugo'] [[ 34 3 0 0 0 0 0 1 2 1 2 1] [ 4 38 1 0 0 0 0 0 1 2 0 3] [ 0 1 27 0 0 0 0 1 0 0 0 0] [ 0 0 0 3 0 0 0 0 0 0 0 0] [ 0 0 0 0 7 0 0 1 0 0 0 0] [ 0 0 0 0 0 12 1 2 0 0 1 1] [ 1 0 0 0 0 0 32 1 0 1 3 0] [ 2 2 1 0 0 0 1 237 5 44 10 6] [ 0 0 0 0 0 0 1 2 81 4 6 1] [ 2 1 1 0 0 2 1 38 3 378 43 9] [ 0 0 0 0 0 0 2 11 4 34 195 7] [ 1 0 0 1 1 0 1 7 1 21 41 79]] Agreement accuracy: 0.761 Nominal Kripp. Alpha: 0.702 Interval Kripp. Alpha: 0.840 Ordinal Kripp. Alpha: 0.777 ------------------------ ----------------------------------- -----ANNOTATOR-SPECIFIC------------ ----------------------------------- Annotator-specific agreements on 'tarča' ----------------------------------------- ------self-agreement---------- a0 annotator-specific self-agreement on tarča Number of duplicates: 200 Agreement accuracy: 0.885 Nominal Kripp. Alpha: 0.859 Interval Kripp. Alpha: 0.910 Ordinal Kripp. Alpha: 0.907 ------------------------ a1 annotator-specific self-agreement on tarča Number of duplicates: 87 Agreement accuracy: 0.793 Nominal Kripp. Alpha: 0.716 Interval Kripp. Alpha: 0.915 Ordinal Kripp. Alpha: 0.759 ------------------------ a2 annotator-specific self-agreement on tarča Number of duplicates: 94 Agreement accuracy: 0.830 Nominal Kripp. Alpha: 0.786 Interval Kripp. Alpha: 0.865 Ordinal Kripp. Alpha: 0.834 ------------------------ a3 annotator-specific self-agreement on tarča Number of duplicates: 181 Agreement accuracy: 0.652 Nominal Kripp. Alpha: 0.571 Interval Kripp. Alpha: 0.654 Ordinal Kripp. Alpha: 0.596 ------------------------ a4 annotator-specific self-agreement on tarča Number of duplicates: 153 Agreement accuracy: 0.804 Nominal Kripp. Alpha: 0.743 Interval Kripp. Alpha: 0.774 Ordinal Kripp. Alpha: 0.775 ------------------------ a5 annotator-specific self-agreement on tarča Number of duplicates: 111 Agreement accuracy: 0.667 Nominal Kripp. Alpha: 0.507 Interval Kripp. Alpha: 0.855 Ordinal Kripp. Alpha: 0.697 ------------------------ a6 annotator-specific self-agreement on tarča Number of duplicates: 201 Agreement accuracy: 0.831 Nominal Kripp. Alpha: 0.782 Interval Kripp. Alpha: 0.920 Ordinal Kripp. Alpha: 0.843 ------------------------ a7 annotator-specific self-agreement on tarča Number of duplicates: 92 Agreement accuracy: 0.707 Nominal Kripp. Alpha: 0.638 Interval Kripp. Alpha: 0.778 Ordinal Kripp. Alpha: 0.680 ------------------------ a8 annotator-specific self-agreement on tarča Number of duplicates: 156 Agreement accuracy: 0.692 Nominal Kripp. Alpha: 0.598 Interval Kripp. Alpha: 0.827 Ordinal Kripp. Alpha: 0.766 ------------------------ a9 annotator-specific self-agreement on tarča Number of duplicates: 200 Agreement accuracy: 0.720 Nominal Kripp. Alpha: 0.657 Interval Kripp. Alpha: 0.876 Ordinal Kripp. Alpha: 0.761 ------------------------ ----------------------------------------- ------inter-agreement---------- a0 annotator-specific inter-agreement on tarča Number of duplicates: 2418 Agreement accuracy: 0.676 Nominal Kripp. Alpha: 0.596 Interval Kripp. Alpha: 0.701 Ordinal Kripp. Alpha: 0.618 ------------------------ a1 annotator-specific inter-agreement on tarča Number of duplicates: 1812 Agreement accuracy: 0.665 Nominal Kripp. Alpha: 0.573 Interval Kripp. Alpha: 0.688 Ordinal Kripp. Alpha: 0.586 ------------------------ a2 annotator-specific inter-agreement on tarča Number of duplicates: 1930 Agreement accuracy: 0.644 Nominal Kripp. Alpha: 0.552 Interval Kripp. Alpha: 0.660 Ordinal Kripp. Alpha: 0.534 ------------------------ a3 annotator-specific inter-agreement on tarča Number of duplicates: 2492 Agreement accuracy: 0.601 Nominal Kripp. Alpha: 0.507 Interval Kripp. Alpha: 0.606 Ordinal Kripp. Alpha: 0.498 ------------------------ a4 annotator-specific inter-agreement on tarča Number of duplicates: 2352 Agreement accuracy: 0.658 Nominal Kripp. Alpha: 0.567 Interval Kripp. Alpha: 0.635 Ordinal Kripp. Alpha: 0.527 ------------------------ a5 annotator-specific inter-agreement on tarča Number of duplicates: 1882 Agreement accuracy: 0.588 Nominal Kripp. Alpha: 0.469 Interval Kripp. Alpha: 0.581 Ordinal Kripp. Alpha: 0.505 ------------------------ a6 annotator-specific inter-agreement on tarča Number of duplicates: 2500 Agreement accuracy: 0.659 Nominal Kripp. Alpha: 0.571 Interval Kripp. Alpha: 0.691 Ordinal Kripp. Alpha: 0.583 ------------------------ a7 annotator-specific inter-agreement on tarča Number of duplicates: 1807 Agreement accuracy: 0.605 Nominal Kripp. Alpha: 0.508 Interval Kripp. Alpha: 0.613 Ordinal Kripp. Alpha: 0.510 ------------------------ a8 annotator-specific inter-agreement on tarča Number of duplicates: 2335 Agreement accuracy: 0.599 Nominal Kripp. Alpha: 0.484 Interval Kripp. Alpha: 0.609 Ordinal Kripp. Alpha: 0.491 ------------------------ a9 annotator-specific inter-agreement on tarča Number of duplicates: 2470 Agreement accuracy: 0.612 Nominal Kripp. Alpha: 0.514 Interval Kripp. Alpha: 0.626 Ordinal Kripp. Alpha: 0.517 ------------------------