Human floor: reviewer agreement with the enumerator per subtask
| subtask | n_verdicts | agree | disagree | unknown | agree_pct | disagree_pct | unknown_pct | protocol_ok_pct |
|---|---|---|---|---|---|---|---|---|
| blfluency_l1 | 571 | 511 | 22 | 37 | 89.5 | 3.9 | 6.5 | 98.3 |
| blfluency_l4 | 518 | 374 | 45 | 89 | 72.2 | 8.7 | 17.2 | 96.8 |
| computation | 566 | 497 | 32 | 36 | 87.8 | 5.7 | 6.4 | 98.6 |
| idnummag | 570 | 461 | 23 | 82 | 80.9 | 4.0 | 14.4 | 97.9 |
| idwrd_eng | 951 | 536 | 237 | 171 | 56.4 | 24.9 | 18.0 | 97.0 |
| idwrd_urd | 971 | 493 | 294 | 163 | 50.8 | 30.3 | 16.8 | 97.7 |
| letterid_eng | 1142 | 1026 | 61 | 53 | 89.8 | 5.3 | 4.6 | 98.1 |
| letterid_urd | 1141 | 1017 | 86 | 38 | 89.1 | 7.5 | 3.3 | 97.9 |
| listcomp_eng | 570 | 477 | 55 | 35 | 83.7 | 9.6 | 6.1 | 96.4 |
| listcomp_urd | 568 | 459 | 58 | 51 | 80.8 | 10.2 | 9.0 | 97.1 |
| numrep | 569 | 494 | 32 | 42 | 86.8 | 5.6 | 7.4 | 98.6 |
| orf_eng | 1406 | 865 | 249 | 281 | 61.5 | 17.7 | 20.0 | 97.8 |
| orf_urd | 1441 | 782 | 360 | 284 | 54.3 | 25.0 | 19.7 | 96.3 |
| patterns | 569 | 490 | 29 | 49 | 86.1 | 5.1 | 8.6 | 97.8 |
| pw_eng | 921 | 483 | 196 | 226 | 52.4 | 21.3 | 24.5 | 97.6 |
| rdcomp_eng | 457 | 294 | 67 | 94 | 64.3 | 14.7 | 20.6 | 98.5 |
| rdcomp_urd | 491 | 359 | 55 | 71 | 73.1 | 11.2 | 14.5 | 96.9 |
| word_problems | 569 | 495 | 36 | 36 | 87.0 | 6.3 | 6.3 | 97.6 |