Home / Tools / ROC / AUC Calculator

ROC / AUC Calculator

An ROC curve traces the trade-off between the true-positive and false-positive rates as you sweep a classifier's threshold; the AUC summarizes overall discrimination in a single number. Paste outcomes plus predicted scores to get the curve, the AUC with a DeLong confidence interval, three optimal thresholds (Youden, F1, cost-weighted), and a calibration check. Everything runs in your browser.

New to ROC and AUC? Read the 4-min primer

What ROC and AUC mean. A binary classifier produces a score for each case. As you sweep the decision threshold from high to low, you trade specificity for sensitivity. The ROC curve plots the true-positive rate (sensitivity) against the false-positive rate (1 - specificity) at every threshold. AUC is the area under that curve and equals the probability that a randomly drawn positive case scores higher than a randomly drawn negative one. AUC of 0.5 is random; 1.0 is perfect; 0 is perfectly wrong.

How to read the curve and the threshold. The top-left corner is perfect. The diagonal is random. A curve that hugs the top-left has good discrimination. Each point on the curve corresponds to a specific threshold; sliding the threshold trades false alarms for misses. The threshold marker on the curve shows where you currently sit, and the confusion matrix updates so you can see exactly how many TPs, FPs, FNs, and TNs that threshold produces.

Picking the right optimum. Youden's J maximises sensitivity + specificity - 1, which is the same as the point on the curve farthest above the diagonal. F1 maximises 2 P R / (P + R), which weighs precision and recall equally and is preferred when classes are balanced and you want a single score. Cost-weighted minimises FP cost_FP + FN cost_FN, which is the right answer when you can put a number on a missed positive vs. a false alarm. They usually disagree, and that disagreement is information about your problem.

When ROC is the wrong tool. On heavily imbalanced data (positives below 5%), ROC flatters you because changes in FP count barely move the FPR; report a precision-recall curve too. For multi-class problems, ROC is one-vs-rest per class plus a macro average; this tool is binary v1. For survival outcomes use time-dependent ROC. And ROC measures discrimination only; a model can have AUC of 0.9 yet emit miscalibrated probabilities that overstate confidence. The Brier score and reliability bins below catch that.

Try a real-world example to load:

🔬 Well-separated screening

A cleanly separated screening problem. AUC near 0.89.

I want to evaluate a binary classifier and report its AUC with a 95▾% confidence interval.

Outcomes & scores

Paste two columns: outcome (0 or 1), then the predicted score. Whitespace, comma or semicolon separated.

n = 0

Cost ratio FN : FP

used for the cost-weighted threshold (a missed positive costs this many false alarms)

–

Area under the ROC curve (AUC)

–

paste data on the left

FPR (x) vs TPR (y); diagonal = random

paste data to draw the curve

t threshold 0.50

score distribution by outcome

Youden J

–

sens / spec

–

prec / rec

Cost-weighted

–

FP + k FN

Paste outcomes (0/1) and predicted scores to see what the AUC means for your data.

InferencePaste outcomes (0/1) and predicted scores on the left to see the AUC, optimal thresholds and a live confusion matrix.

We swept the threshold across all values to compute sensitivity and specificity at every cutoff, summarized as the area under the ROC curve.

paste data to get a report line

How this is computed

1Sort the cases by score, descending. Walking down, every positive steps the curve up by 1/n_pos and every negative steps it right by 1/n_neg, tracing a staircase from (0,0) to (1,1). Tied scores give diagonal segments.

2The area under that staircase is the AUC, equal to the Mann-Whitney U statistic divided by n_pos × n_neg (ties counted as half). So AUC = P(a random positive scores above a random negative).

3The DeLong variance comes from the structural components of the AUC; the confidence interval is formed on the AUC scale as AUC ± z · SE and clamped to [0, 1] (exactly what pROC returns).

4Youden, F1 and cost-weighted optima are found on pROC's midpoint threshold grid; the confusion matrix, Brier score and reliability bins are evaluated at the threshold you pick.

The same analysis in R

✓ No data leaves your browser ✓ Verified against R's pROC package ✓ Free, no sign-up

Calibration: is the probability trustworthy?

AUC measures ranking, not whether a predicted 0.8 really means an 80% event rate. The Brier score and reliability bins catch a well-ranking but overconfident model.

paste data to compute the Brier score and reliability bins.

ROC anatomy and the DeLong interval

TPR = TP / (TP + FN) FPR = FP / (FP + TN) ROC = { (FPR(t), TPR(t)) : t in scores }

Sweeping the threshold. Sort cases by score, descending. As you walk down, every positive you encounter steps the curve up (TPR increases by 1/n_pos), every negative steps it right (FPR increases by 1/n_neg). The result is a staircase from (0,0) to (1,1). Tied scores give diagonal segments; the area under the staircase is the AUC.

AUC = U / (n_pos * n_neg) U = sum over (i,j) of [score_i > score_j] + 0.5 * [score_i = score_j] i positive, j negative

AUC equals P(score+ > score-). The Mann-Whitney U statistic counts pairs where a positive scores higher than a negative, with ties counted as half. Dividing by the number of such pairs (n_pos times n_neg) gives a probability. So AUC = 0.83 literally means: pick one positive and one negative at random; with probability 0.83 the positive scores higher. This makes AUC scale-free (any monotonic transform of score gives the same AUC).

V_10[i] = (1/n_neg) * sum_j psi(X_i, Y_j) i positive V_01[j] = (1/n_pos) * sum_i psi(X_i, Y_j) j negative psi(x, y) = 1 if x > y, 0.5 if equal, 0 otherwise SE(AUC) = sqrt( var(V_10)/n_pos + var(V_01)/n_neg )

DeLong's structural components. DeLong, DeLong & Clarke-Pearson (1988) showed that AUC is the mean of structural components V_10 (one per positive) and V_01 (one per negative), and that its variance can be estimated nonparametrically from the sample variances of those components. This tool computes V_10 and V_01 directly, then forms the interval on the AUC scale as AUC ± z · SE and clamps it to [0, 1]. That is what pROC::ci.auc(method="delong") returns; a logit-scale interval is a different, materially wider estimate.

Youden: argmax_t TPR(t) + (1 - FPR(t)) - 1 F1: argmax_t 2 P(t) R(t) / (P(t) + R(t)) Cost: argmin_t FP(t) + k * FN(t)

Three optima, three answers. Youden picks the point of maximum vertical distance above the diagonal; it weights sensitivity and specificity equally. F1 weights precision and recall equally and is the right choice when classes are balanced and you care about retrieval. Cost-weighted lets you set the relative cost of a missed positive (k=10 means a missed positive is ten times worse than a false alarm). When the three disagree, the disagreement tells you that costs and prevalence matter for your decision.

Brier = mean( (score - y)^2 ) calibration: bin scores into deciles, plot mean(score) vs mean(y) per bin

Discrimination is not calibration. AUC tells you whether high-risk cases score higher than low-risk ones, but says nothing about whether a predicted probability of 0.8 actually corresponds to an 80% event rate. The Brier score combines both (lower is better, max 0.25 for a binary outcome). Reliability bins make miscalibration visible: if predicted 0.8-0.9 cases have empirical rate 0.5, your model is overconfident in that range.

When this is the wrong tool

If you have…: Use instead
Multi-class outcomes (3+ classes): This v1 is binary. Compute one-vs-rest ROC per class plus a macro-averaged AUC manually, or pick a one-vs-rest tool. Multi-class is on the roadmap.
Survival outcomes (time-to-event): Use time-dependent ROC (survivalROC, timeROC). The ROC curve becomes a function of horizon t. See the survival analysis tutorial.
Heavy class imbalance (positives < 5%): ROC is misleading because the FPR axis barely moves. Report a precision-recall curve and PR-AUC instead; they show the precision degradation that ROC hides.
Only a single confusion matrix: You cannot reconstruct an ROC from one threshold. Use the confusion matrix interpreter for accuracy / precision / recall / MCC at that one threshold.
Probabilities matter, not ranking: AUC is invariant under monotonic transforms; a model can have AUC of 0.95 and be wildly miscalibrated. Always look at the Brier score and the reliability diagram above.

Frequently asked questions

What does AUC actually measure?

AUC (area under the ROC curve) is the probability that a randomly chosen positive example has a higher predicted score than a randomly chosen negative example. AUC = 0.5 is random; 1.0 is perfect; 0.7-0.8 is fair, 0.8-0.9 is good. It is threshold-independent: it summarizes performance across every possible cutoff.

How do I choose a classification threshold from an ROC curve?

Pick the threshold that matches your cost function. Equal-cost errors: use Youden's J (max sensitivity + specificity - 1). Asymmetric costs: shift toward fewer false negatives (cancer screening) or fewer false positives (spam filter). This tool shows the Youden, F1 and cost-weighted optima and lets you set any false-negative to false-positive cost ratio.

When does a high AUC not mean a good classifier?

AUC can mislead on imbalanced data: a model may rank well yet still misclassify the minority class at any usable threshold, so also read the precision-recall curve. AUC also ignores calibration: two models with the same AUC can give very different probabilities. Check the Brier score and reliability bins.

What is the DeLong confidence interval, and why not a bootstrap?

DeLong, DeLong & Clarke-Pearson (1988) derive the AUC's variance analytically from its structural components, so the interval is exact and deterministic - it is pROC's default. This tool reports that interval on the AUC scale, clamped to [0, 1]. A browser cannot reproduce pROC's stratified resampling seed-for-seed, so we skip bootstrap here; run ci.auc(method="bootstrap") in R if you need it.

Does my data get uploaded anywhere?

No. Everything runs in your browser with JavaScript. Your pasted outcomes and scores never leave your device, and every displayed number is computed locally and checked against R's pROC package.

Go deeper

Logistic regression in R Confusion matrix interpreter Logistic regression diagnostics All calculators

Numerical accuracy: AUC is the Mann-Whitney U with mid-rank ties; the DeLong interval is built on the AUC scale and clamped to [0, 1] (exactly what pROC::ci.auc(method="delong") returns, not a logit interval). Optimal thresholds use pROC's midpoint grid. Verified against pROC::auc(), pROC::var(), pROC::ci.auc() and pROC::coords() to 1e-9.