Categorical Inter-rater Agreement Analyzer

Example clinical scenario

Two clinicians independently classify the same participants into the same prespecified diagnostic categories. Each participant contributes one complete pair of ratings.

Method

Exact observed agreement is the diagonal proportion of the square agreement table and uses a Wilson score confidence interval. Cohen’s kappa adjusts observed agreement for agreement expected under statistically independent assignments while preserving each rater’s observed marginal distribution. Ordinal analysis uses prespecified linear or quadratic agreement weights. Kappa confidence intervals use the Fleiss–Cohen–Everitt large-sample variance and a two-sided normal interval without truncation.

Assumptions and limitations

• Exactly two identified raters classify the same independent units using the same categories.
• Every analyzed unit has two complete ratings; incomplete pairs are not silently removed.
• Raters and units should be interpreted according to the sampling design; the app does not generalize automatically to other raters.
• Ordinal weights assume the displayed category positions are equally spaced.
• Large-sample confidence intervals may be unreliable for small, sparse, highly imbalanced, or boundary tables.
• Kappa depends on category prevalence and rater margins and does not by itself establish acceptable agreement.
• Multi-rater, clustered, repeated, partially observed, prevalence-adjusted, Gwet, Krippendorff, ICC, and Bland–Altman analyses are outside this release.

Interpretation and safety

No universal qualitative grading is applied. Interpret the coefficient with exact agreement, the complete table, marginal distributions, uncertainty, study design, and consequences of disagreement. This research tool does not replace study-specific statistical consultation.

Privacy

Manual, pasted, and uploaded ratings are processed in browser memory. The app does not intentionally transmit or retain ratings, file names, worksheet names, or column names.

Validation status

Version 1.1.0 retains RA01-M1 and reproduces the prespecified nominal and ordinal benchmark tables and includes automated boundary, invariance, import, interface, privacy, and responsive-layout checks. Public availability supports independent comparison and manual review; it is not a claim of external, regulatory, or universal statistical validation.

References

1. Cohen J. A coefficient of agreement for nominal scales. Educ Psychol Meas. 1960;20:37–46.
2. Cohen J. Weighted kappa: nominal scale agreement with provision for scaled disagreement or partial credit. Psychol Bull. 1968;70:213–220.
3. Fleiss JL, Cohen J, Everitt BS. Large sample standard errors of kappa and weighted kappa. Psychol Bull. 1969;72:323–327.
4. Wilson EB. Probable inference, the law of succession, and statistical inference. J Am Stat Assoc. 1927;22:209–212.