Agreement report for both annotation questions: hate speech type (vrsta) and target (tarča) for the data in the file IMSyPP_SI_anotacije_evaluation-clarin.csv. Hate speech types (vrsta): 0 appropriate (ni sporni govor) 1 inappropriate (nespodobni govor) 2 offensive (žalitev) 3 violent (nasilje) Hate speech targets (tarča): 1 racism (ksenofobija in rasizem) 2 migrants (begunci/migranti) 3 islamophobia (islamofobija) 4 antisemitism (antisemitizem) 5 religion (druge religije) 6 homophobia (homofobija) 7 sexism (seksizem) 8 ideology (ideologija) 9 media (novinarji in mediji) 10 politics (politika/-i) 11 individual (posameznik) 12 other (drugo) Annotated instances a0 2000/2000 a1 2000/2000 a2 2000/2000 a3 2000/2000 a4 2000/2000 a5 2000/2000 a6 2000/2000 a7 2000/2000 a8 2000/2000 a9 2000/2000 ----------------- -----OVERALL----- ----------------- Annotated for vrsta : 20000 0 ni sporni govor 13273 1 nespodobni govor 285 2 žalitev 6373 3 nasilje 69 ----------------------------------- -----AGGREGATED INTER-AGREEMENT----- ----------------------------------- Aggregated inter-agreement on 'vrsta' Number of duplicates: 9000 ||0 ni sporni govor |1 nespodobni govor |2 žalitev |3 nasilje ||sum 0 ni sporni govor || 5072 | 0 | 0 | 0 || 5072 1 nespodobni govor || 68 | 53 | 0 | 0 || 121 2 žalitev || 1765 | 90 | 1899 | 0 || 3754 3 nasilje || 9 | 3 | 30 | 11 || 53 ------------------------------------------------------------- sum || 6914 | 146 | 1929 | 11 || 9000 Labels: ['0 ni sporni govor', '1 nespodobni govor', '2 žalitev', '3 nasilje'] [[5072 39 934 6] [ 29 53 42 2] [ 831 48 1899 12] [ 3 1 18 11]] Agreement accuracy: 0.782 Nominal Kripp. Alpha: 0.522 Interval Kripp. Alpha: 0.536 Ordinal Kripp. Alpha: 0.538 ------------------------ ----------------------------------- -----AGGREGATED SELF-AGREEMENT----- ----------------------------------- Aggregated self-agreement on 'vrsta' Number of duplicates: 1000 ||0 ni sporni govor |1 nespodobni govor |2 žalitev |3 nasilje ||sum 0 ni sporni govor || 599 | 0 | 0 | 0 || 599 1 nespodobni govor || 7 | 5 | 0 | 0 || 12 2 žalitev || 81 | 1 | 303 | 0 || 385 3 nasilje || 1 | 0 | 2 | 1 || 4 ------------------------------------------------------------- sum || 688 | 6 | 305 | 1 || 1000 Labels: ['0 ni sporni govor', '1 nespodobni govor', '2 žalitev', '3 nasilje'] [[599 4 48 0] [ 3 5 0 0] [ 33 1 303 2] [ 1 0 0 1]] Agreement accuracy: 0.908 Nominal Kripp. Alpha: 0.803 Interval Kripp. Alpha: 0.812 Ordinal Kripp. Alpha: 0.811 ------------------------ ----------------------------------- -----ANNOTATOR-SPECIFIC------------ ----------------------------------- Annotator-specific agreements on 'vrsta' ----------------------------------------- ------self-agreement---------- a0 annotator-specific self-agreement on vrsta Number of duplicates: 100 Agreement accuracy: 0.980 Nominal Kripp. Alpha: 0.957 Interval Kripp. Alpha: 0.957 Ordinal Kripp. Alpha: 0.957 ------------------------ a1 annotator-specific self-agreement on vrsta Number of duplicates: 100 Agreement accuracy: 0.910 Nominal Kripp. Alpha: 0.819 Interval Kripp. Alpha: 0.819 Ordinal Kripp. Alpha: 0.819 ------------------------ a2 annotator-specific self-agreement on vrsta Number of duplicates: 100 Agreement accuracy: 0.870 Nominal Kripp. Alpha: 0.626 Interval Kripp. Alpha: 0.595 Ordinal Kripp. Alpha: 0.609 ------------------------ a3 annotator-specific self-agreement on vrsta Number of duplicates: 100 Agreement accuracy: 0.870 Nominal Kripp. Alpha: 0.742 Interval Kripp. Alpha: 0.807 Ordinal Kripp. Alpha: 0.803 ------------------------ a4 annotator-specific self-agreement on vrsta Number of duplicates: 100 Agreement accuracy: 0.890 Nominal Kripp. Alpha: 0.664 Interval Kripp. Alpha: 0.664 Ordinal Kripp. Alpha: 0.664 ------------------------ a5 annotator-specific self-agreement on vrsta Number of duplicates: 100 Agreement accuracy: 0.850 Nominal Kripp. Alpha: 0.610 Interval Kripp. Alpha: 0.622 Ordinal Kripp. Alpha: 0.616 ------------------------ a6 annotator-specific self-agreement on vrsta Number of duplicates: 100 Agreement accuracy: 0.930 Nominal Kripp. Alpha: 0.817 Interval Kripp. Alpha: 0.819 Ordinal Kripp. Alpha: 0.815 ------------------------ a7 annotator-specific self-agreement on vrsta Number of duplicates: 100 Agreement accuracy: 0.920 Nominal Kripp. Alpha: 0.834 Interval Kripp. Alpha: 0.847 Ordinal Kripp. Alpha: 0.845 ------------------------ a8 annotator-specific self-agreement on vrsta Number of duplicates: 100 Agreement accuracy: 0.900 Nominal Kripp. Alpha: 0.808 Interval Kripp. Alpha: 0.819 Ordinal Kripp. Alpha: 0.819 ------------------------ a9 annotator-specific self-agreement on vrsta Number of duplicates: 100 Agreement accuracy: 0.960 Nominal Kripp. Alpha: 0.921 Interval Kripp. Alpha: 0.919 Ordinal Kripp. Alpha: 0.919 ------------------------ ----------------------------------------- ------inter-agreement---------- a0 annotator-specific inter-agreement on vrsta Number of duplicates: 1800 Agreement accuracy: 0.818 Nominal Kripp. Alpha: 0.602 Interval Kripp. Alpha: 0.618 Ordinal Kripp. Alpha: 0.620 ------------------------ a1 annotator-specific inter-agreement on vrsta Number of duplicates: 1800 Agreement accuracy: 0.767 Nominal Kripp. Alpha: 0.523 Interval Kripp. Alpha: 0.538 Ordinal Kripp. Alpha: 0.540 ------------------------ a2 annotator-specific inter-agreement on vrsta Number of duplicates: 1800 Agreement accuracy: 0.766 Nominal Kripp. Alpha: 0.455 Interval Kripp. Alpha: 0.462 Ordinal Kripp. Alpha: 0.469 ------------------------ a3 annotator-specific inter-agreement on vrsta Number of duplicates: 1800 Agreement accuracy: 0.797 Nominal Kripp. Alpha: 0.585 Interval Kripp. Alpha: 0.599 Ordinal Kripp. Alpha: 0.601 ------------------------ a4 annotator-specific inter-agreement on vrsta Number of duplicates: 1800 Agreement accuracy: 0.789 Nominal Kripp. Alpha: 0.519 Interval Kripp. Alpha: 0.536 Ordinal Kripp. Alpha: 0.538 ------------------------ a5 annotator-specific inter-agreement on vrsta Number of duplicates: 1800 Agreement accuracy: 0.804 Nominal Kripp. Alpha: 0.537 Interval Kripp. Alpha: 0.545 Ordinal Kripp. Alpha: 0.543 ------------------------ a6 annotator-specific inter-agreement on vrsta Number of duplicates: 1800 Agreement accuracy: 0.743 Nominal Kripp. Alpha: 0.341 Interval Kripp. Alpha: 0.348 Ordinal Kripp. Alpha: 0.355 ------------------------ a7 annotator-specific inter-agreement on vrsta Number of duplicates: 1800 Agreement accuracy: 0.804 Nominal Kripp. Alpha: 0.541 Interval Kripp. Alpha: 0.553 Ordinal Kripp. Alpha: 0.557 ------------------------ a8 annotator-specific inter-agreement on vrsta Number of duplicates: 1800 Agreement accuracy: 0.767 Nominal Kripp. Alpha: 0.520 Interval Kripp. Alpha: 0.532 Ordinal Kripp. Alpha: 0.534 ------------------------ a9 annotator-specific inter-agreement on vrsta Number of duplicates: 1800 Agreement accuracy: 0.762 Nominal Kripp. Alpha: 0.516 Interval Kripp. Alpha: 0.538 Ordinal Kripp. Alpha: 0.538 ------------------------ -------------------------------------------------------------------- ----------------- -----OVERALL----- ----------------- 6430 annotated for tarča Annotated for tarča : 6430 1 ksenofobija in rasizem 125 2 begunci/migranti 68 3 islamofobija 21 4 antisemitizem 10 5 druge religije 15 6 homofobija 16 7 seksizem 68 8 ideologija 839 9 novinarji in mediji 682 10 politika/-i 2623 11 posameznik 1318 12 drugo 645 ----------------------------------- -----AGGREGATED INTER-AGREEMENT----- ----------------------------------- Aggregated inter-agreement on 'tarča' Number of duplicates: 1935 ||1 ksenofobija in rasizem |2 begunci/migranti |3 islamofobija |4 antisemitizem |5 druge religije |6 homofobija |7 seksizem |8 ideologija |9 novinarji in mediji |10 politika/-i |11 posameznik |12 drugo ||sum 1 ksenofobija in rasizem || 19 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 || 19 2 begunci/migranti || 6 | 8 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 || 14 3 islamofobija || 0 | 5 | 3 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 || 8 4 antisemitizem || 0 | 0 | 0 | 2 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 || 2 5 druge religije || 0 | 0 | 0 | 0 | 2 | 0 | 0 | 0 | 0 | 0 | 0 | 0 || 2 6 homofobija || 0 | 0 | 0 | 0 | 0 | 4 | 0 | 0 | 0 | 0 | 0 | 0 || 4 7 seksizem || 0 | 0 | 0 | 0 | 0 | 0 | 13 | 0 | 0 | 0 | 0 | 0 || 13 8 ideologija || 5 | 3 | 1 | 1 | 2 | 1 | 1 | 138 | 0 | 0 | 0 | 0 || 152 9 novinarji in mediji || 1 | 1 | 0 | 0 | 0 | 1 | 1 | 8 | 168 | 0 | 0 | 0 || 180 10 politika/-i || 6 | 4 | 0 | 0 | 1 | 0 | 2 | 147 | 38 | 595 | 0 | 0 || 793 11 posameznik || 13 | 0 | 1 | 1 | 0 | 1 | 3 | 86 | 32 | 138 | 211 | 0 || 486 12 drugo || 11 | 2 | 0 | 0 | 0 | 0 | 2 | 42 | 11 | 76 | 66 | 52 || 262 ------------------------------------------------------------- sum || 61 | 23 | 5 | 4 | 5 | 7 | 22 | 421 | 249 | 809 | 277 | 52 || 1935 Labels: ['1 ksenofobija in rasizem', '2 begunci/migranti', '3 islamofobija', '4 antisemitizem', '5 druge religije', '6 homofobija', '7 seksizem', '8 ideologija', '9 novinarji in mediji', '10 politika/-i', '11 posameznik', '12 drugo'] [[ 19 2 0 0 0 0 0 3 1 3 4 2] [ 4 8 2 0 0 0 0 1 1 2 0 0] [ 0 3 3 0 0 0 0 1 0 0 1 0] [ 0 0 0 2 0 0 0 0 0 0 0 0] [ 0 0 0 0 2 0 0 2 0 1 0 0] [ 0 0 0 0 0 4 0 1 0 0 0 0] [ 0 0 0 0 0 0 13 0 0 0 0 0] [ 2 2 0 1 0 0 1 138 4 73 48 14] [ 0 0 0 0 0 1 1 4 168 20 18 7] [ 3 2 0 0 0 0 2 74 18 595 69 33] [ 9 0 0 1 0 1 3 38 14 69 211 21] [ 9 2 0 0 0 0 2 28 4 43 45 52]] Agreement accuracy: 0.628 Nominal Kripp. Alpha: 0.503 Interval Kripp. Alpha: 0.503 Ordinal Kripp. Alpha: 0.423 ------------------------ ----------------------------------- -----AGGREGATED SELF-AGREEMENT----- ----------------------------------- Aggregated self-agreement on 'tarča' Number of duplicates: 306 ||1 ksenofobija in rasizem |2 begunci/migranti |3 islamofobija |4 antisemitizem |5 druge religije |6 homofobija |7 seksizem |8 ideologija |9 novinarji in mediji |10 politika/-i |11 posameznik |12 drugo ||sum 1 ksenofobija in rasizem || 4 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 || 4 2 begunci/migranti || 2 | 4 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 || 6 3 islamofobija || 0 | 0 | 2 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 || 2 4 antisemitizem || 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 || 0 5 druge religije || 0 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 || 1 6 homofobija || 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 || 0 7 seksizem || 0 | 0 | 0 | 0 | 0 | 0 | 6 | 0 | 0 | 0 | 0 | 0 || 6 8 ideologija || 0 | 0 | 0 | 0 | 0 | 0 | 0 | 32 | 0 | 0 | 0 | 0 || 32 9 novinarji in mediji || 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 29 | 0 | 0 | 0 || 29 10 politika/-i || 1 | 0 | 0 | 0 | 0 | 0 | 0 | 5 | 5 | 106 | 0 | 0 || 117 11 posameznik || 0 | 0 | 0 | 0 | 0 | 0 | 0 | 5 | 0 | 16 | 53 | 0 || 74 12 drugo || 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 0 | 7 | 3 | 24 || 35 ------------------------------------------------------------- sum || 7 | 4 | 2 | 0 | 1 | 0 | 6 | 43 | 34 | 129 | 56 | 24 || 306 Labels: ['1 ksenofobija in rasizem', '2 begunci/migranti', '3 islamofobija', '4 antisemitizem', '5 druge religije', '6 homofobija', '7 seksizem', '8 ideologija', '9 novinarji in mediji', '10 politika/-i', '11 posameznik', '12 drugo'] [[ 4 2 0 0 0 0 0 0 0 1 0 0] [ 0 4 0 0 0 0 0 0 0 0 0 0] [ 0 0 2 0 0 0 0 0 0 0 0 0] [ 0 0 0 0 0 0 0 0 0 0 0 0] [ 0 0 0 0 1 0 0 0 0 0 0 0] [ 0 0 0 0 0 0 0 0 0 0 0 0] [ 0 0 0 0 0 0 6 0 0 0 0 0] [ 0 0 0 0 0 0 0 32 0 5 2 0] [ 0 0 0 0 0 0 0 0 29 1 0 0] [ 0 0 0 0 0 0 0 0 4 106 7 4] [ 0 0 0 0 0 0 0 3 0 9 53 3] [ 0 0 0 0 0 0 0 1 0 3 0 24]] Agreement accuracy: 0.853 Nominal Kripp. Alpha: 0.806 Interval Kripp. Alpha: 0.915 Ordinal Kripp. Alpha: 0.846 ------------------------ ----------------------------------- -----ANNOTATOR-SPECIFIC------------ ----------------------------------- Annotator-specific agreements on 'tarča' ----------------------------------------- ------self-agreement---------- IMSyPP anotacije round2 a0 (AS).csv annotator-specific self-agreement on tarča Number of duplicates: 35 Agreement accuracy: 0.857 Nominal Kripp. Alpha: 0.819 Interval Kripp. Alpha: 0.911 Ordinal Kripp. Alpha: 0.851 ------------------------ IMSyPP anotacije round2 a1 (AT).csv annotator-specific self-agreement on tarča Number of duplicates: 42 Agreement accuracy: 0.833 Nominal Kripp. Alpha: 0.769 Interval Kripp. Alpha: 0.866 Ordinal Kripp. Alpha: 0.870 ------------------------ IMSyPP anotacije round2 a2 (AMG).csv annotator-specific self-agreement on tarča Number of duplicates: 14 Agreement accuracy: 0.857 Nominal Kripp. Alpha: 0.782 Interval Kripp. Alpha: 0.983 Ordinal Kripp. Alpha: 0.870 ------------------------ IMSyPP anotacije round2 a3 (HA).csv annotator-specific self-agreement on tarča Number of duplicates: 34 Agreement accuracy: 0.765 Nominal Kripp. Alpha: 0.676 Interval Kripp. Alpha: 0.951 Ordinal Kripp. Alpha: 0.777 ------------------------ IMSyPP anotacije round2 a4 (JZ).csv annotator-specific self-agreement on tarča Number of duplicates: 15 Agreement accuracy: 1.000 Nominal Kripp. Alpha: 1.000 Interval Kripp. Alpha: 1.000 Ordinal Kripp. Alpha: 1.000 ------------------------ IMSyPP anotacije round2 a5 (KB).csv annotator-specific self-agreement on tarča Number of duplicates: 18 Agreement accuracy: 0.778 Nominal Kripp. Alpha: 0.728 Interval Kripp. Alpha: 0.966 Ordinal Kripp. Alpha: 0.904 ------------------------ IMSyPP anotacije round2 a6 (NC).csv annotator-specific self-agreement on tarča Number of duplicates: 19 Agreement accuracy: 0.947 Nominal Kripp. Alpha: 0.933 Interval Kripp. Alpha: 0.390 Ordinal Kripp. Alpha: 0.909 ------------------------ IMSyPP anotacije round2 a7 (NS).csv annotator-specific self-agreement on tarča Number of duplicates: 35 Agreement accuracy: 0.914 Nominal Kripp. Alpha: 0.890 Interval Kripp. Alpha: 0.982 Ordinal Kripp. Alpha: 0.936 ------------------------ IMSyPP anotacije round2 a8 (PP).csv annotator-specific self-agreement on tarča Number of duplicates: 51 Agreement accuracy: 0.843 Nominal Kripp. Alpha: 0.767 Interval Kripp. Alpha: 0.920 Ordinal Kripp. Alpha: 0.737 ------------------------ IMSyPP anotacije round2 a9 (ZF).csv annotator-specific self-agreement on tarča Number of duplicates: 43 Agreement accuracy: 0.837 Nominal Kripp. Alpha: 0.782 Interval Kripp. Alpha: 0.900 Ordinal Kripp. Alpha: 0.676 ------------------------ ----------------------------------------- ------inter-agreement---------- IMSyPP anotacije round2 a0 (AS).csv annotator-specific inter-agreement on tarča Number of duplicates: 417 Agreement accuracy: 0.686 Nominal Kripp. Alpha: 0.582 Interval Kripp. Alpha: 0.623 Ordinal Kripp. Alpha: 0.513 ------------------------ IMSyPP anotacije round2 a1 (AT).csv annotator-specific inter-agreement on tarča Number of duplicates: 478 Agreement accuracy: 0.640 Nominal Kripp. Alpha: 0.500 Interval Kripp. Alpha: 0.330 Ordinal Kripp. Alpha: 0.418 ------------------------ IMSyPP anotacije round2 a2 (AMG).csv annotator-specific inter-agreement on tarča Number of duplicates: 305 Agreement accuracy: 0.649 Nominal Kripp. Alpha: 0.500 Interval Kripp. Alpha: 0.659 Ordinal Kripp. Alpha: 0.501 ------------------------ IMSyPP anotacije round2 a3 (HA).csv annotator-specific inter-agreement on tarča Number of duplicates: 499 Agreement accuracy: 0.591 Nominal Kripp. Alpha: 0.472 Interval Kripp. Alpha: 0.388 Ordinal Kripp. Alpha: 0.372 ------------------------ IMSyPP anotacije round2 a4 (JZ).csv annotator-specific inter-agreement on tarča Number of duplicates: 350 Agreement accuracy: 0.649 Nominal Kripp. Alpha: 0.529 Interval Kripp. Alpha: 0.431 Ordinal Kripp. Alpha: 0.373 ------------------------ IMSyPP anotacije round2 a5 (KB).csv annotator-specific inter-agreement on tarča Number of duplicates: 350 Agreement accuracy: 0.597 Nominal Kripp. Alpha: 0.459 Interval Kripp. Alpha: 0.442 Ordinal Kripp. Alpha: 0.411 ------------------------ IMSyPP anotacije round2 a6 (NC).csv annotator-specific inter-agreement on tarča Number of duplicates: 205 Agreement accuracy: 0.624 Nominal Kripp. Alpha: 0.522 Interval Kripp. Alpha: 0.529 Ordinal Kripp. Alpha: 0.485 ------------------------ IMSyPP anotacije round2 a7 (NS).csv annotator-specific inter-agreement on tarča Number of duplicates: 325 Agreement accuracy: 0.631 Nominal Kripp. Alpha: 0.510 Interval Kripp. Alpha: 0.493 Ordinal Kripp. Alpha: 0.422 ------------------------ IMSyPP anotacije round2 a8 (PP).csv annotator-specific inter-agreement on tarča Number of duplicates: 463 Agreement accuracy: 0.587 Nominal Kripp. Alpha: 0.436 Interval Kripp. Alpha: 0.509 Ordinal Kripp. Alpha: 0.414 ------------------------ IMSyPP anotacije round2 a9 (ZF).csv annotator-specific inter-agreement on tarča Number of duplicates: 478 Agreement accuracy: 0.636 Nominal Kripp. Alpha: 0.504 Interval Kripp. Alpha: 0.573 Ordinal Kripp. Alpha: 0.357 ------------------------