The t-test from scratch: is the difference real?

Lesson 1 of 2 - the one-sample test

The t-test from scratch

Meet Meena. She runs a small business that packs dal (lentils) into 1 kg packets, and her machine is set to drop 1000 grams into each one. Lately a few customers have grumbled that the packets feel light. So Meena does the sensible thing: she pulls 20 packets off the line and weighs them on a kitchen scale. Their average comes to 978 grams - 22 grams short of the target.

Now here is her real problem. She did not weigh all ten thousand packets she makes in a week; she weighed only 20. So which is it?

The machine is genuinely underfilling, and 978 g is the truth, or
The machine is fine at 1000 g on average, and Meena just happened to scoop up 20 slightly-light packets by luck.

That single question - is a difference big enough to be real, or small enough that plain sampling luck could have produced it? - is what a t-test answers, and a p-value is its verdict. In this first lesson you will be able to:

Say in plain words what question a t-test answers, and compute the t-statistic for Meena's packets
Read a p-value correctly, and spot the misreading almost everyone makes
Run a one-sample t-test in R and interpret every number it returns

Prerequisites: you can run R and make a vector, and you know what a mean (the average) and a standard deviation (how spread out the numbers are) are. Everything else we build here, slowly.

Why "22 grams short" is not yet proof

The average wobbles every time you weigh a new batch

Here is the catch that makes Meena's problem interesting. Suppose she puts those 20 packets back and grabs a fresh 20 from the same line. Will the new average be exactly 978 g again? Almost certainly not. Maybe 985 g this time. Grab another 20: maybe 974 g. The average of a handful of packets is a moving target - it jumps around a little every time, purely because a different handful landed on the scale.

So a gap you can see (978 vs 1000) is never proof on its own. Even if the machine were perfectly set to 1000 g, any 20 packets would still come out a little above or below 1000, just from the luck of which packets you grabbed. We keep two things straight:

the true average fill of the machine - a fixed number Meena can never see exactly without weighing every packet. Statisticians call it $\mu$ (the Greek letter "mu").
the average of one batch she weighed - which wobbles from batch to batch. We write it $\bar{x}$ ("x-bar"). For Meena's batch, $\bar{x} = 978$.

The whole game of the t-test is to measure how much $\bar{x}$ wobbles by chance, and then ask: is Meena's 22-gram gap big compared with that wobble, or is it the kind of gap the wobble produces all the time?

Key Insight

A difference you can see is not yet evidence. The real question is always: is this gap large compared with how much the average jumps around by chance? That ratio - the gap measured against the wobble - is what a t-test computes, not the gap by itself.

Putting a number on the wobble

Standard error: how much the batch average jumps around

To compare Meena's gap against the wobble, we first need to measure the wobble itself. That measurement has a name - the standard error of the mean - and a simple formula. The standard error is the typical distance between a single batch's average and the machine's true average:

\[ SE = \frac{s}{\sqrt{n}} \]

Two plain-English ingredients:

$s$ is the sample standard deviation: how much the individual packets differ from one another. If every packet is nearly identical, $s$ is small; if some are 940 g and some 1020 g, $s$ is large. Say Meena's 20 packets have $s = 30$ grams.
$n$ is the sample size: how many packets she weighed. Here $n = 20$.

So Meena's wobble is $SE = 30 / \sqrt{20} \approx 6.7$ grams. In words: even if the machine's true fill never changed, the average of 20 packets would still drift up and down by about 7 grams from one batch to the next, just by chance. Two things fall out of the formula:

More spread between packets (bigger $s$) means a noisier, less trustworthy average.
More packets weighed (bigger $n$) means a steadier average - but only through $\sqrt{n}$. You fight the wobble with the square root of effort, not the effort itself.

Key Insight

The standard error is the ruler we measure the gap in. A 22-gram gap means nothing until you know the wobble: if $SE$ is 7 grams, 22 grams is a big gap (three rulers out); if $SE$ were 50 grams, the same 22 would be well within the normal jiggle.

Check yourself

What happens if Meena weighs more packets?

Meena decides to weigh 80 packets next time instead of 20 - four times as many, from the same machine. What happens to the standard error (the wobble of her average)?

Right. $SE = s/\sqrt{n}$. Multiplying $n$ by 4 multiplies $\sqrt{n}$ by 2, so the wobble is halved. Meena weighs four times as many packets to make her average only twice as steady - that is the square-root law.

Careful: the wobble shrinks with $\sqrt{n}$, not $n$. Four times the packets divides the standard error by $\sqrt{4} = 2$, not by 4.

Signal divided by noise

The t-statistic: the gap measured in wobbles

Now we put Meena's question into a single number. We compare the signal (her 22-gram gap) against the noise (her 7-gram wobble). For one batch tested against a claimed value $\mu_0$ - the value we are checking, here the machine's setting $\mu_0 = 1000$ - the t-statistic is:

\[ t = \frac{\bar{x} - \mu_0}{s / \sqrt{n}} \]

Read it slowly, piece by piece:

The top, $\bar{x} - \mu_0$, is the signal: how far the batch average sits from the value we are testing. For Meena: $978 - 1000 = -22$ grams.
The bottom, $s/\sqrt{n}$, is the noise: the standard error from the last step, $6.7$ grams.

Divide one by the other and you get a single number answering one question: how many wobbles away from 1000 did Meena's average land? For her packets, $t = -22 / 6.7 \approx -3.3$. The minus sign just means she landed below 1000; the size, 3.3, is what matters - her average is about three-and-a-third standard errors below the target.

Is 3.3 a lot? A $t$ near 0 means the average sat right on the target, comfortably inside the normal wobble - nothing to see. A $t$ of 3.3 means it landed far out, in territory the wobble rarely reaches. That starts to look like a real problem, not luck. The next steps make "rarely" exact.

Your turn

Compute Meena's t by hand

Meena's 20 packets average $\bar{x} = 978$ g, with a standard deviation of $s = 30$ g, tested against the target $\mu_0 = 1000$ g. The signal (978 - 1000) is filled in. Complete the noise - the standard error, $s$ divided by the square root of $n$ - to finish her t.

That gives t = -3.28. Meena's average landed about 3.3 standard errors below 1000 - a long way out.The noise is the standard error: s divided by the square root of n. Fill in sqrt(n).

Show answer

xbar <- 978; mu0 <- 1000
s <- 30; n <- 20
t_stat <- (xbar - mu0) / (s / sqrt(n))
t_stat
#> [1] -3.28

What does "rarely" look like?

The null distribution: what chance alone produces

To decide whether $t = -3.3$ is surprising, we need to know what $t$ values look like when nothing is wrong. So let us pretend, just for a moment, that Meena is worried over nothing: the machine really does fill to 1000 g on average, and her 978 came purely from the luck of the draw. Statisticians call this pretend-everything-is-fine assumption the null hypothesis.

If the null hypothesis were true and Meena repeated her 20-packet weigh-in again and again, her $t$ would not be 0 every time - it would scatter around 0. Collect all those "machine is fine" $t$ values and they trace out a known, fixed curve: Student's t-distribution, drawn below. This curve is the yardstick we hold Meena's $t$ against.

It looks almost like the familiar bell curve, with one twist: its tails are a little heavier (fatter at the edges). That is because Meena had to estimate the wobble $s$ from the same 20 packets, and that extra uncertainty makes big $t$ values slightly more common. The exact shape is set by the degrees of freedom, $df = n - 1 = 19$: the fewer packets, the heavier the tails; with lots of packets the curve settles into the ordinary bell.

Note

The curve drawn here is the large-sample (normal) version, which is easiest to read. A true 20-packet t-distribution sits a touch lower in the middle with slightly heavier tails, so the exact p-value is a hair larger - but the idea on the next step, that the tail area is the p-value, is exactly the same.

The verdict number

The p-value is the area in the tails

Now mark Meena's $t$ on that "machine is fine" curve and ask: how often would pure luck push the average this far from 1000? That probability is the p-value - the chance, if the machine were truly fine, of getting a $t$ at least as extreme as the one Meena saw. On the picture it is simply the shaded area in the tails, beyond $+t$ and $-t$ (both sides, because a machine that overfilled by the same amount would be just as surprising). Drag the slider and watch the shaded area, and the p-value, change.

Slide out to Meena's $t \approx 3.3$ and the shaded area is tiny: a "fine" machine almost never produces a 20-packet average that far from 1000. So either something very unlucky happened, or the machine is not actually fine. A small tail area means the data are surprising under "machine is fine," and Meena starts to doubt it. A large tail area would mean her result is the kind of thing chance throws up all the time - no case to answer.

Check yourself

Read the p-value correctly

Meena's test returns $p = 0.004$. Which statement is the correct reading?

Exactly. The p-value is computed by assuming the machine is fine and asking how often chance alone would be this extreme. It is a statement about the data given the assumption, never the assumption given the data.

That flips the logic around. The p-value is the chance of data this extreme assuming the machine is fine - never the chance that "the machine is fine" is true given the data. It does not tell you how likely the hypothesis itself is.

Drawing the line

The decision rule, and the trap in it

To turn the p-value into a yes/no, Meena compares it to a threshold called $\alpha$ ("alpha"), chosen before she looks at the data (0.05 is the common, if arbitrary, convention):

If $p < \alpha$, she rejects the null ("machine is fine") and calls the result statistically significant. Meena's $p = 0.004$ is well under 0.05, so she concludes the machine really is underfilling and gets it serviced.
If $p \ge \alpha$, she fails to reject the null. Notice the careful wording: she does not "accept" or "prove" the machine is fine.

That last distinction is the one almost everyone drops. Failing to reject only means the data were not surprising enough to rule out "machine is fine" - very different from showing the machine is fine. Maybe it is off by a tiny amount that 20 packets simply could not detect. Absence of evidence is not evidence of absence.

Warning

The threshold $\alpha$ is a decision Meena makes, not a law of nature, and she must pick it before seeing the data. A result with $p = 0.049$ is not meaningfully different from $p = 0.051$; treating 0.05 as a magic cliff is how good questions turn into bad statistics.

Run it for real

From Meena's t to her verdict, in R

You already have Meena's t by hand: $t = -3.28$, on $df = n - 1 = 19$ degrees of freedom. Turning that into a p-value is one line - pt() gives the area in the t-distribution's tail:

RInteractive R

t_stat <- -3.28; df <- 19 2 * pt(t_stat, df) # times 2: over- and under-filling both count #> [1] 0.0039

So if the machine were truly fine, a 20-packet average this far off would turn up only about 0.4% of the time. Meena rejects "the machine is fine" and books a service call - the data answered her question.

In practice you do not even compute $t$ yourself: you hand t.test() the raw data and it does every step at once - the t-statistic, the degrees of freedom, the p-value, and a confidence interval. Here it is on a dataset that ships with R so you can run it this second - mtcars, the specs of 32 cars - asking the same shape of question of fuel economy (mpg): is the average different from 20?

RInteractive R

t.test(mtcars$mpg, mu = 20) #> #> One Sample t-test #> #> data: mtcars$mpg #> t = 0.0851, df = 31, p-value = 0.9328 #> alternative hypothesis: true mean is not equal to 20 #> 95 percent confidence interval: #> 17.92 22.26 #> sample estimates: #> mean of x #> 20.0906

Every number is something you now understand:

t = 0.0851 is the signal-over-noise ratio - here the average (20.09) sits almost exactly on 20, so the signal is tiny.
df = 31 is $n - 1 = 32 - 1$, the shape of the "nothing is wrong" curve.
p-value = 0.9328 is the tail area: a $t$ this small is utterly ordinary under the null, so there is no evidence the average differs from 20. (Contrast Meena, whose $t = -3.3$ gave a tiny p - a real effect. This is the other verdict: fail to reject.)
The 95 percent confidence interval [17.92, 22.26] is the flip side of the test: every target value inside it is one we could not reject. It contains 20, which is exactly why we fail to reject 20.

One caveat before you point t.test() at anything: it trusts that the measurements are independent and roughly bell-shaped (or that $n$ is large enough that the average is, by the Central Limit Theorem). For heavily lopsided data with a small $n$ it is the wrong tool - we come back to when that breaks in Lesson 2.

Your turn

Run a one-sample test in R

Test whether the average mpg in mtcars differs from 20 by filling in the value to test against (the mu argument is the $\mu_0$ from our formula).

p = 0.93. With the average at 20.09, there is no evidence the true mean differs from 20.Set mu to the value you are testing against - here, mu = 20.

Show answer

t.test(mtcars$mpg, mu = 20)

Go deeper

References

A few authoritative places to take this further, each worth the click:

Student (1908), The Probable Error of a Mean, Biometrika - the original paper that introduced the t-distribution.
OpenIntro Statistics (free textbook) - a clear, rigorous treatment of inference, the t-distribution and p-values.
R reference: t.test() - the official documentation for the function you just used.
Seeing Theory: Frequentist Inference (Brown University) - a beautiful interactive view of sampling, the null distribution and p-values.

You built the one-sample t-test

You did not just learn to call a function - you built the idea from one real question. A batch average wobbles from batch to batch; the standard error measures that wobble; the t-statistic is the gap measured in wobbles (signal over noise); the null distribution shows what chance alone produces; and the p-value is the tail area that turns it all into a verdict. Then you ran the one-sample test in R and read every line.

But Meena's next question is different: she is not comparing one machine to a fixed target any more - she wants to compare two machines against each other. That needs the two-sample t-test, and it comes with a twist that can flip the answer.

Up next, Lesson 2: Comparing two groups, and reading it honestly - the two-sample and paired tests, why a tiny p-value is not the same as a big effect, and the ways the t-test gets misused.

Navigate