Confusion Matrix Interpreter
A confusion matrix counts how a classifier's predictions line up against the truth. Drop in your counts (or paste R's caret output) and this tool reads off accuracy with an exact confidence interval, precision, recall, F1, MCC, Cohen's kappa and balanced accuracy, then tells you in plain English which metric to trust when your classes are imbalanced.
Base-rate explorer: watch precision collapse as the positive class gets rarer
Precision (positive predictive value) depends on how common the positive class actually is, not just on how good the test is. Hold sensitivity and specificity fixed, drag the prevalence down, and see the same test produce mostly false alarms. This is the base-rate fallacy, applied to a population of 10,000.
How these numbers are computed
Every metric, and when it matters
A confusion matrix has four counts for a binary problem: true positives, false positives, false negatives and true negatives. Every score below is a different way of squeezing those four numbers into one.
| Metric | Formula | Reach for it when |
|---|---|---|
| Accuracy | (TP+TN) / N | Classes are balanced and the two error types cost about the same. |
| Precision (PPV) | TP / (TP+FP) | False positives are expensive (flagging real email as spam, biopsy on the healthy). Report F0.5. |
| Recall (Sensitivity) | TP / (TP+FN) | False negatives are expensive (missing a cancer, missing fraud). Report F2. |
| Specificity (TNR) | TN / (TN+FP) | You care about correctly clearing the negatives. |
| F1 | 2·P·R / (P+R) | One balanced score for the positive class on imbalanced data. |
| MCC | (TP·TN − FP·FN) / √(…) | The single most reliable score; uses all four cells, not fooled by imbalance. |
| Cohen's kappa | (pₒ − pₑ) / (1 − pₑ) | Agreement corrected for what chance alone would give. |
| Balanced accuracy | (Sens + Spec) / 2 | Accuracy that is not inflated by a dominant class. |
The imbalance trap
On a data set that is 99% negative, a model that predicts "negative" for everything is 99% accurate and completely useless. Accuracy hides this; MCC and balanced accuracy do not. The No Information Rate (NIR) is exactly that trivial baseline, the accuracy of always guessing the largest class, and the tool tests whether your classifier beats it with a one-sided exact binomial p-value, the same one caret prints as P-Value [Acc > NIR].
The accuracy confidence interval here is the Clopper-Pearson exact binomial interval (what binom.test and caret use), not a normal or Wilson approximation, so it stays correct even at tiny sample sizes and accuracies near 0 or 1.
Frequently asked questions
When should I trust accuracy versus F1 or MCC?
Accuracy is only meaningful when classes are roughly balanced. On a 95/5 split a model that always predicts the majority class scores 95% while learning nothing. F1 ignores that inflation for the positive class, and MCC accounts for all four cells and stays near zero for a trivial classifier, so on imbalanced data trust MCC and balanced accuracy over raw accuracy.
What is Matthews correlation coefficient (MCC)?
MCC is a single number computed from all four cells of the matrix. It runs from −1 (perfect disagreement) through 0 (random) to +1 (perfect agreement). Unlike F1 it also credits true negatives, so class imbalance does not fool it, which makes it the most reliable one-number summary for a binary classifier.
How is the accuracy confidence interval computed?
With the Clopper-Pearson exact binomial interval, the same method as R's binom.test and caret::confusionMatrix(). Accuracy is a binomial proportion (correct out of total), and the interval inverts the exact binomial tail probabilities rather than using a normal approximation, so it is valid for any sample size.
What does the No Information Rate mean?
The No Information Rate (NIR) is the accuracy you would get by always predicting the single largest class. caret reports a one-sided binomial p-value for the hypothesis that your accuracy beats the NIR. A tiny p-value means the classifier is doing genuinely better than the majority-class baseline.
Does the emitted R code reproduce these numbers?
Yes. The R block builds your matrix as a table and calls caret::confusionMatrix(..., mode = "everything"), which reproduces every displayed value. The tool's own math is verified against caret across binary, multi-class and edge cases.