Weighted Kappa

Overview

Weighted kappa extends Cohen’s kappa to ordered categories [1].

It recognises that not all disagreements are equally serious.

For a four-category severity scale:

• none versus mild is a one-step disagreement;
• none versus severe is a three-step disagreement.

Weighted kappa assigns smaller penalties to near disagreements and larger penalties to distant disagreements.

The weight system must be prespecified and clinically defensible.

Why this matters

Unweighted kappa treats every off-diagonal disagreement identically.

This can be inappropriate for ordinal clinical ratings such as:

• no, mild, moderate or severe disease;
• functional class I to IV;
• imaging grade;
• toxicity grade;
• triage urgency.

Weighted kappa can better reflect ordered disagreement, but a high value may conceal clinically important extreme disagreements if weights are overly forgiving.

When to use weighted kappa

Use weighted kappa when:

• exactly two raters classify the same units;
• categories have a meaningful order;
• disagreement distance is clinically relevant;
• weights are defined before analysis.

Do not use weights merely because they increase the statistic.

If category spacing is unknown, linear or quadratic weights are conventions rather than proven clinical distances.

Research question and variables

The observed table is:

Most disagreement occurs one category from the diagonal.

Data structure

Ordinal paired ratings

simulated data

Variable	Meaning
id	Participant identifier
ordinal_rater_1	Rater 1 category coded 0–3
ordinal_rater_2	Rater 2 category coded 0–3

The numeric codes represent order. They should not automatically be treated as equally spaced measurements.

Statistical assumptions

Weight matrices

Let category indices be $i$ and $j$ among $k$ ordered categories.

Linear disagreement weights

A two-step disagreement is penalised twice as much as a one-step disagreement.

Quadratic disagreement weights

Quadratic weights penalise distant disagreements disproportionately.

Some software reports agreement weights instead of disagreement weights. The analyst must verify the convention.

How the method works

Weighted kappa can be written using weighted disagreement:

$$\kappa_w = 1- \frac{ \sum_{ij}w_{ij}O_{ij} }{ \sum_{ij}w_{ij}E_{ij} }$$

where:

• $O_{ij}$ is the observed cell proportion;
• $E_{ij}$ is the expected cell proportion from marginals;
• $w_{ij}$ is the disagreement weight.

Exact agreement receives weight zero in this formulation.

Worked example

Worked analysis in R

rating_data <- read.csv(
  "data/part-12-rating-data.csv"
)

ordinal_ratings <- rating_data[
  c(
    "ordinal_rater_1",
    "ordinal_rater_2"
  )
]

irr::kappa2(
  ordinal_ratings,
  weight = "unweighted"
)

irr::kappa2(
  ordinal_ratings,
  weight = "equal"
)

irr::kappa2(
  ordinal_ratings,
  weight = "squared"
)

manual_weighted_kappa <- function(
  rater_1,
  rater_2,
  weight = c("linear", "quadratic")
) {
  weight <- match.arg(weight)

  categories <- sort(
    unique(
      c(
        rater_1,
        rater_2
      )
    )
  )

  tab <- table(
    factor(
      rater_1,
      levels = categories
    ),
    factor(
      rater_2,
      levels = categories
    )
  )

  observed <- tab / sum(tab)
  expected <- outer(
    rowSums(observed),
    colSums(observed)
  )

  k <- length(categories)

  distance <- abs(
    outer(
      seq_len(k),
      seq_len(k),
      "-"
    )
  ) / (k - 1)

  disagreement_weight <- if (
    weight == "linear"
  ) {
    distance
  } else {
    distance^2
  }

  1 -
    sum(
      disagreement_weight *
      observed
    ) /
    sum(
      disagreement_weight *
      expected
    )
}

manual_weighted_kappa(
  ordinal_ratings$ordinal_rater_1,
  ordinal_ratings$ordinal_rater_2,
  "linear"
)

Selected analysis output

Unweighted kappa = 0.707
Linear-weighted kappa = 0.830
Bootstrap 95% CI = 0.772 to 0.878

Quadratic-weighted kappa = 0.918
Bootstrap 95% CI = 0.888 to 0.943

(The bootstrap confidence interval calculations are not shown here.)

Visualising ordinal disagreement

The matrix should accompany the coefficient so readers can identify clinically important extreme disagreements. Comprehensive inter-rater reliability guidance discusses the consequences of alternative weighting systems [2].

Linear versus quadratic weights

Linear weights are often easier to explain.

Quadratic weights can produce high coefficients when most disagreements are one step apart.

Quadratic weighted kappa is mathematically related to certain intraclass correlation formulations under particular coding and design assumptions [3].

This does not make ordinal categories truly continuous or equally spaced.

Custom clinical weights

A custom matrix may be justified when:

• mild versus moderate disagreement has small consequences;
• severe versus non-severe disagreement changes urgent management;
• category boundaries have asymmetric consequences.

The matrix should be:

• prespecified;
• clinically justified;
• reported completely;
• examined in sensitivity analysis.

Data-derived weights should be avoided.

Prevalence and marginals

Weighted kappa remains sensitive to category prevalence and rater marginal distributions.

Report:

• the full table;
• category frequencies;
• observed exact agreement;
• one-category agreement;
• extreme disagreements.

A high weighted kappa in a population dominated by one severity category may not transport to a broader spectrum. Mathematical comparisons of weighted and unweighted kappa further demonstrate how table structure affects the coefficient [4].

Confidence intervals

Bootstrap resampling is useful for weighted kappa.

The bootstrap must resample participants and preserve both ratings.

When raters classify multiple correlated units from one participant, resampling should occur at the participant cluster level.

Interpreting the output

Presenting the results

Suggested reporting

Report category order, full table, weight matrix or named convention, observed agreement, weighted kappa with CI and sensitivity analyses using alternative weights.

Do not report “weighted kappa” without identifying the weights.

Common mistakes

Knowledge check

Why use weighted rather than unweighted kappa?

Answer and reasoning

Ordered disagreements can have different severity depending on category distance.

Why is quadratic kappa higher in the example?

Answer and reasoning

Most disagreements are adjacent and receive very small quadratic penalties.

Does quadratic weighting prove equal category spacing?

Answer and reasoning

No. It imposes a mathematical penalty convention.

What should accompany weighted kappa?

Answer and reasoning

The full ordinal table, weight definition, confidence interval and clinically important disagreement counts.

How should bootstrap resampling be performed?

Answer and reasoning

Resample paired units or participant clusters while keeping both raters’ scores linked.

Key take-home messages

References

1. Cohen J. Weighted kappa: nominal scale agreement with provision for scaled disagreement or partial credit. Psychol Bull. 1968;70(4):213–220. https://doi.org/10.1037/h0026256
2. Gwet KL. Handbook of inter-rater reliability. 4th ed. Gaithersburg (MD): Advanced Analytics; 2014.
3. Fleiss JL, Cohen J. The equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliability. Educ Psychol Meas. 1973;33(3):613–619. https://doi.org/10.1177/001316447303300309
4. Warrens MJ. Weighted kappa is higher than Cohen’s kappa for tridiagonal agreement tables. Stat Methodol. 2013;13:1–11. https://doi.org/10.1016/j.stamet.2012.12.004