Home / Tools / Cohen's Kappa Calculator
Cohen's Kappa Calculator
Cohen's kappa measures how well two raters agree on a category, correcting for the agreement you would expect by chance alone. Paste a confusion matrix of counts, or two columns of paired ratings, and read the observed and expected agreement, the kappa with its 95% confidence interval, weighted kappa for ordered categories, and the Landis-Koch band. Everything is checked against R's psych::cohen.kappa(), and stays in your browser.
I want to measure agreement from a KxK confusion matrix of counts
Counts, one matrix row per line. Cell (i, j) is how many subjects rater A put in category i and rater B in category j. Diagonal cells are agreements.
κ = -
How these numbers are computed
w = 1 - |i-j|^e / (K-1)^e, so a near miss counts as partial agreement. Linear weights use e = 1, quadratic use e = 2. Weighting only changes the result when categories are ordered and K is at least 3.kappa +/- z * SE with z from the normal distribution, clamped to the range -1 to 1. This matches psych::cohen.kappa().
✓ No data leaves your browser
✓ Verified against R's psych::cohen.kappa()
✓ Free, no sign-up
How to read Cohen's kappa
Kappa is agreement beyond chance, on a scale where 0 is chance-level and 1 is perfect. Read the value, its interval and the table together, never the headline number alone.
By Landis and Koch, 0.61 to 0.80 is substantial and 0.81 and up is almost perfect, with 0.41 to 0.60 moderate and below 0.40 weak. Many fields want at least 0.60. Treat the labels as a rough guide, and report the number and its interval.
Kappa comes from a sample. The interval shows how far it could move in another sample of the same size. A lower bound above 0 means agreement beats chance at that confidence level; a wide interval, common with small samples, means the estimate is uncertain.
For unordered categories use the unweighted kappa. For ordered categories use weighted kappa so a one-step miss counts as partial agreement. Quadratic-weighted kappa equals the two-way intraclass correlation for an ordinal scale.
What each number means
| Statistic | What it tells you | In R |
|---|---|---|
| Observed agreement | Share of cases the raters classified the same way; the diagonal of the table over the total. | sum(diag(p)) |
| Expected agreement | Agreement expected by chance from each rater's marginal frequencies. | sum(rowSums(p)*colSums(p)) |
| Cohen's kappa | Agreement beyond chance: (observed - expected) / (1 - expected). The headline for nominal categories. | cohen.kappa(m)$kappa |
| Linear-weighted kappa | Ordinal kappa that penalizes disagreements by how many steps apart they are. | cohen.kappa(m, w.exp = 1)$weighted.kappa |
| Quadratic-weighted kappa | Ordinal kappa penalizing larger gaps much more; equals the two-way ICC. | cohen.kappa(m)$weighted.kappa |
| Standard error | Sampling error of kappa, from the Fleiss-Cohen-Everitt large-sample formula. | sqrt(cohen.kappa(m)$var.kappa) |
| 95% CI | Plausible range for the true kappa; a lower bound above 0 beats chance. | cohen.kappa(m)$confid |
| Landis-Koch band | A conventional label (slight, fair, moderate, substantial, almost perfect) for the value. | - |
The calculator reports every row of this table for your data. The unweighted and weighted kappa coincide when there are only two categories.
Frequently asked questions
What is a good Cohen's kappa value?
By the Landis and Koch (1977) benchmark, kappa below 0 is poor, 0 to 0.20 is slight, 0.21 to 0.40 is fair, 0.41 to 0.60 is moderate, 0.61 to 0.80 is substantial and 0.81 to 1.00 is almost perfect. Many fields treat 0.60 or above as acceptable and 0.80 or above as strong. These cutoffs are a convention, not a statistical law, so report the value and its confidence interval rather than leaning on the label alone.
When should I use weighted kappa?
Use weighted kappa when the categories are ordered, for example none, mild, moderate and severe. Weighted kappa counts a near miss as partial agreement instead of a full disagreement, so it fits ordinal ratings better. Linear weights penalize disagreements in proportion to how many steps apart they are; quadratic weights penalize larger gaps much more heavily, and for an ordinal scale quadratic-weighted kappa equals the two-way intraclass correlation. For unordered (nominal) categories, use the unweighted kappa.
Why is my kappa low even though the raters agreed most of the time?
This is the kappa paradox. When one category is very common, raters agree often just by chance, so the expected agreement is high and kappa, which measures agreement beyond chance, stays low even when observed agreement looks excellent. A skewed prevalence or a systematic bias between raters both drag kappa down. Always read kappa alongside the observed agreement and the marginal totals, and consider prevalence- and bias-adjusted kappa when the margins are very unbalanced.
What is the difference between observed and expected agreement?
Observed agreement is the proportion of cases the two raters actually classified the same way, the sum of the diagonal divided by the total. Expected agreement is how often they would agree if each rater assigned categories at random with their own marginal frequencies, computed from the row and column totals. Cohen's kappa removes the chance part: kappa equals (observed - expected) / (1 - expected), so 0 means agreement no better than chance and 1 means perfect agreement.
Does Cohen's kappa work for more than two raters?
Cohen's kappa is defined for exactly two raters. For three or more raters use Fleiss' kappa, which generalizes the chance-corrected idea to many raters, or the intraclass correlation for numeric ratings. This calculator handles the two-rater case, both as a confusion matrix and as two columns of paired ratings, for any number of categories.
Does this match R's psych::cohen.kappa()?
Yes. The observed and expected agreement, the unweighted kappa, the linear- and quadratic-weighted kappa, the standard errors and the confidence intervals are each verified against psych::cohen.kappa() across binary, ordinal and multi-category tables, including perfect agreement, near-chance agreement, skewed prevalence and tiny samples, agreeing to at least seven significant figures. The kappa point estimates are also cross-checked against irr::kappa2().