Table 5 Rater pairs indicated moderate or substantial agreements with respect to kappa scores and good or excellent ICC scores as well as percentage agreements for individual items
Rater | Number of scenarios rated | Intraclass correlation (ICC) | Kappa | Percentage agreement |
|---|---|---|---|---|
1, 2 | 16 | 0.716 (0.369–0.890) | 0.564 | 83.4 |
1, 3 | 13 | 0.816 (0.515–0.939) | 0.644 | 85.7 |
1, 4 | 18 | 0.870 (0.692–0.949) | 0.667 | 96.7 |