ICC Calculator
The intraclass correlation answers one question: when several raters score the same subjects, how much of the spread in the scores is real difference between subjects rather than disagreement between raters? Paste your ratings with one row per subject and one column per rater. You get all six ICC forms with confidence intervals, and a guided picker that turns three questions about your study into the single ICC you should report.
I want to measure rater agreement from a wide table, one column per rater
One row per subject, one column per rater. A header row of rater names is optional. Rows with a missing rating are dropped, because ICC needs a complete table.
1. Was every subject rated by the same raters?
2. Must the raters land on the same number, or is the same ranking enough?
3. What produces the score you will actually use?
ICC = -
How these numbers are computed
✓ No data leaves your browser
✓ Verified against R's psych::ICC()
✓ Free, no sign-up
Which ICC do I need?
This is the question the ICC literature is worst at answering, and it is the one that changes your number the most. The same ratings can give an ICC of 0.40 or 0.99 depending on which form you pick, so the choice is not a technicality. Three questions settle it.
If each subject was rated by whoever happened to be available, so the raters differ from subject to subject, there is no rater column to model and you must use ICC(1,1). If the same raters rated everyone, you have a two-way design and can separate rater effects from error, which is almost always the stronger study.
Absolute agreement asks whether the raters produce the same number. Consistency asks only whether they rank subjects the same way, ignoring a constant offset between raters. Pick absolute when the score itself must be right, consistency when only the ordering is used.
Report the form that matches how the score gets made. If one rater will score future subjects, report the single-measure form. Only report the average-measure form if the mean of all raters really is the score you use. The average always looks better, which is exactly why it is easy to misreport.
Two of these can be answered from the data alone; none of them should be. Whether raters are a sample of a wider pool or the only raters that matter is a fact about your study design, not about your numbers, and the arithmetic cannot tell them apart. That is why the picker asks instead of guessing.
The six forms, and what each one assumes
| Form | Model | Answers | Report it when |
|---|---|---|---|
| ICC(1,1) | One-way random | How reliable is one rater, when every subject saw a different set of raters? | Raters were not crossed with subjects. Rater effects cannot be separated from error, so they inflate the error term. |
| ICC(2,1) | Two-way random, absolute | How reliable is one rater, and would a different rater from the same pool give the same number? | The same raters rated everyone, they stand in for a wider pool, and the value itself must be right. |
| ICC(3,1) | Two-way mixed, consistency | How reliable is one rater, once a constant offset between raters is forgiven? | These exact raters are the whole story, or only the ranking matters. A systematic bias is not counted as error. |
| ICC(1,k) | One-way random, average | How reliable is the mean of all k raters, in a non-crossed design? | Same as ICC(1,1), but the reported score is the mean of the raters. |
| ICC(2,k) | Two-way random, absolute, average | How reliable is the mean of all k raters, on an absolute scale? | Same as ICC(2,1), but the reported score is the mean of the raters. |
| ICC(3,k) | Two-way mixed, consistency, average | How reliable is the mean of all k raters, ignoring rater offsets? | Same as ICC(3,1), but the reported score is the mean of the raters. This is the number most similar to Cronbach's alpha. |
The calculator reports every row for your data, so you can see how much the choice moves the answer. Load the Rater bias example to see the extreme case: the same eight subjects give a consistency ICC near 0.99 and a one-way ICC below zero, because the raters rank identically but score on completely different levels.
What each number means
| Statistic | What it tells you | In R |
|---|---|---|
| ICC | Share of the total variance that is real difference between subjects. 0 is no reliability, 1 is perfect. | ICC(x, lmer = FALSE)$results |
| Confidence interval | Plausible range for the true ICC. Koo and Li advise judging reliability on the lower bound. | $results[, c(7, 8)] |
| F, df1, df2, p | Tests the null that the ICC is 0, that is, that subjects do not differ at all. | $results[, c(3, 4, 5, 6)] |
| MSB | Mean square between subjects. Large when subjects genuinely differ, which is what drives the ICC up. | $stats["MS", "subjects"] |
| MSJ | Mean square between raters. Large when raters score on different levels. This is what separates ICC(2,1) from ICC(3,1). | $stats["MS", "Judges"] |
| MSE | Residual mean square: disagreement left after subject and rater effects are removed. | $stats["MS", "Residual"] |
| MSW | Within-subject mean square, pooling rater and residual. ICC(1,1) uses this, which is why rater bias hurts it. | ICC(x)$MSW |
Frequently asked questions
Which ICC should I report?
Three questions decide it. First, was every subject rated by the same raters? If each subject was rated by a different set of raters, you have a one-way design and must report ICC(1,1). Second, if the same raters rated everyone, are those raters a sample standing in for a wider pool, or are they the only raters that will ever matter? A sample means two-way random; the exact raters means two-way mixed. Third, do the raters need to land on the same number, or only rank subjects in the same order? Same number means absolute agreement, ICC(2,1). Same order is enough means consistency, ICC(3,1). Finally, if the score you actually use is the mean of all raters rather than one rater's score, report the average-measure form, ICC(1,k), ICC(2,k) or ICC(3,k).
What is the difference between ICC(2,1) and ICC(3,1)?
ICC(2,1) measures absolute agreement and ICC(3,1) measures consistency. The difference is whether a systematic offset between raters counts as disagreement. If one rater scores everyone three points higher than another but ranks them identically, consistency is near perfect while absolute agreement is not, so ICC(3,1) stays high and ICC(2,1) drops. ICC(3,1) simply removes the rater effect from the error term; ICC(2,1) leaves it in. Use absolute agreement when the number itself has to be right, for example when raters must apply a scoring rubric interchangeably. Use consistency when only the ranking matters, for example when you are selecting the top candidates.
What is a good ICC value?
By the Koo and Li (2016) guideline, an ICC below 0.5 is poor, 0.5 to 0.75 is moderate, 0.75 to 0.9 is good and above 0.9 is excellent. Their key recommendation is to judge on the lower bound of the confidence interval rather than the point estimate, because a small sample can produce a high ICC with an interval so wide that it is compatible with poor reliability. These bands are a convention, not a statistical law, and the older Cicchetti and Fleiss cutoffs differ. Report the value with its confidence interval, the number of subjects and raters, and which ICC form you used.
Why is my ICC negative?
A negative ICC means the variance between subjects is smaller than the variance within subjects, so two ratings of the same subject resemble each other less than two ratings picked at random. It usually means there is no real subject signal in the data, or the sample is too small for the estimate to be stable. The underlying variance component is being estimated as negative, which is impossible in reality. In practice, report a negative ICC as zero reliability while stating the actual value. Note that psych::ICC with its default lmer setting will silently clamp a negative ICC to 0.00, which hides the problem.
What is the difference between single-measure and average-measure ICC?
Single-measure ICC, written ICC(1,1), ICC(2,1) or ICC(3,1), is the reliability of one typical rater's score. Average-measure ICC, written ICC(1,k), ICC(2,k) or ICC(3,k), is the reliability of the mean of all k raters. Averaging cancels out part of the random rater noise, so the average-measure value is always higher, exactly as the Spearman-Brown formula predicts for a longer test. Choose based on how the score is actually produced in practice: if a single rater will score future subjects, report the single-measure form even though the average-measure form looks better. Reporting the average when one rater does the work overstates your reliability.
How many subjects and raters do I need for an ICC?
The arithmetic works with as few as 2 subjects and 2 raters, but the confidence interval will be uselessly wide. Koo and Li recommend at least 30 subjects and 3 raters for a reliability study. The confidence interval is the honest guide: if it spans from poor to excellent, the study is too small to conclude anything, no matter how good the point estimate looks. This calculator always reports the interval next to the estimate for that reason.
Does this match R's psych::ICC()?
Yes, against psych::ICC(x, lmer = FALSE), which is the exact Shrout and Fleiss ANOVA closed form. All six ICC estimates, their F ratios, degrees of freedom, p-values and confidence bounds are verified across clean, biased, ordinal, two-rater, tiny-sample, perfect-agreement and negative-ICC cases, agreeing to at least nine significant figures. The R code this page emits passes lmer = FALSE for that reason. psych's own default, lmer = TRUE, fits the model with lme4 instead: it is an iterative optimizer that agrees only to convergence tolerance, and it clamps negative variance components at zero, so it can report 0.00 where the closed form reports a negative ICC.
Go deeper
ICC, Cohen's kappa and Cronbach's alpha are the three reliability measures people mix up most. Use ICC for numeric ratings from several raters, kappa for two raters assigning categories, and alpha for the internal consistency of items in a scale. ICC(3,k) and alpha are close relatives: on the same data they answer nearly the same question.