Post-Hoc Test Calculator

An ANOVA answers one question: is there any difference among the group means? A significant F tells you at least one group differs, but not which. A post-hoc test answers the follow-up: which specific pairs differ. It is called post-hoc because you decide to look at the pairs after seeing the omnibus result, and that extra looking is exactly what has to be paid for with a correction. If you had one specific pair planned in advance, that is a planned contrast and needs no post-hoc penalty.

Use Tukey HSD when you want all pairwise comparisons after a one-way ANOVA. It is built for exactly that job: the studentized range distribution already accounts for the fact that you are picking the largest gap among k means, so it gives more power than Bonferroni while still holding the family-wise error rate. Bonferroni simply multiplies each p-value by the number of comparisons. It is more conservative, so it finds fewer differences, but it is simple, transparent and valid for any set of comparisons, not just all-pairs. With many groups the gap widens and Tukey is clearly better; with a handful of comparisons the two often agree.

Use Dunn's test when the ANOVA's assumptions do not hold: strong skew, heavy outliers, ordinal outcomes such as Likert ratings, or clearly unequal variances. Dunn is the follow-up to the Kruskal-Wallis test, the nonparametric counterpart of ANOVA. It replaces the values with their ranks across all observations and compares mean ranks, so only the ordering matters and extreme values cannot drag a mean around. It needs the raw values, because ranks cannot be recovered from a mean and a standard deviation.

Family-wise error rate (FWER) is the probability of making even one false positive across the whole family of comparisons. Tukey and Bonferroni control it, so at alpha 0.05 there is a 5% chance of any false claim at all. False discovery rate (FDR), which Benjamini-Hochberg controls, is the expected proportion of the claims you make that are wrong: if BH flags 10 pairs at 0.05, about 0.5 of them are expected to be false alarms. FDR is a weaker guarantee, so it finds more differences. FWER suits confirmatory work where one false claim is costly; FDR suits screening where you plan to follow up anyway.

Groups that share a letter are not significantly different from each other; groups with no letter in common are. So if group A is labelled a, group B is labelled ab and group C is labelled b, then A and C differ from each other, while B differs from neither because it shares a with A and b with C. A group can carry several letters, which simply means it sits in the overlap and the data cannot separate it from either side. It is the most compact way to report all pairwise results at once, which is why journals like it.

You can, and this calculator will show the pairs, but read them as exploratory. The classical advice is to treat a non-significant omnibus F as the end of the analysis, because running post-hoc tests only after a significant F is what protects the error rate of the whole procedure. Tukey HSD is an exception worth knowing: it controls the family-wise error rate on its own, so it is defensible without the omnibus gate. Bonferroni is likewise self-contained. What is not defensible is hunting through pairs, finding one, and reporting it as if it had been planned.

Yes. The Tukey differences, family-wise confidence intervals and adjusted p-values reproduce TukeyHSD(aov()), including unbalanced designs; the Bonferroni route reproduces pairwise.t.test() with its default pooled standard deviation; and Dunn's z statistics and Benjamini-Hochberg adjusted p-values reproduce FSA::dunnTest(), including the tie correction on ratings data. The letters display matches multcompView::multcompLetters(). Verification runs across 14 datasets at three confidence levels, plus a grid of the underlying studentized range distribution.