Navigate
Roadmap Tutorials Exercises Tools Get certified
Lesson 5 of 8

Hierarchical Models and Partial Pooling

Lesson 4 left you with a fast engine and a pre-flight dashboard for judging any chain. This lesson points both at the most common situation in applied statistics: many groups, most of them small.

Asha's plant store had a strong second month: 121 orders, each tagged with the plant family it came from. Eight families, and wildly unequal counts: succulents brought 40 orders, carnivorous plants exactly 2. Her question sounds innocent: which family earns the front-page banner next month? The natural answer, rank the families by average order value, is about to crown a family on the strength of two receipts.

By the end of this lesson you will be able to:

  • Say why a league table of raw group averages rewards small samples, and why "trust each group alone" and "lump everyone together" are both extreme prior beliefs in disguise
  • Write the varying-intercept hierarchical model, name its hyperparameters and hyperpriors, and read the between-group spread as a pooling dial that the data itself turns
  • Compute each group's pooling weight from the fitted variances, predict which averages move most, and verify against a known answer key that shrinkage beats the raw averages
  • Recognize the funnel shape hierarchical posteriors produce, connect it to the divergences on Lesson 4's dashboard, and name the reparameterization that fixes it

Prerequisites: Lessons 1 to 4 of this course (the prior-times-likelihood update, the Normal-Normal posterior, credible intervals from draws, and the trust dashboard: R-hat, effective sample size, divergences), plus base R vectors, factors and tapply.

Below is the whole lesson in one picture. Eight groups, some large and steady, some tiny and jumpy, and a dial that blends "trust each group alone" into "one number for everyone". Drag it and watch who moves. (It is drawn as eight clinics measuring patients; swap in eight plant families and the logic is identical.)

The setup

Eight families, and a league table that lies

Here is Asha's second month, built right here so every line on this page runs in interactive R. One extra ingredient makes this lesson special: because we simulate the month, we also get the answer key, the true long-run average order value of each family, which real life never shows you. We write it down (true_avg), keep it face down, and use it at the very end to grade every method honestly.

RInteractive R
set.seed(88) fam <- c("succulent", "fern", "monstera", "orchid", "calathea", "bonsai", "alocasia", "carnivorous") n_fam <- c(40, 30, 25, 12, 5, 4, 3, 2) # orders per family true_avg <- c(51, 48, 56, 61, 52, 66, 55, 57) # the answer key: face down for now family <- factor(rep(fam, times = n_fam), levels = fam) value <- round(rnorm(121, mean = rep(true_avg, times = n_fam), sd = 18)) orders <- data.frame(family, value) table(orders$family) #> #> succulent fern monstera orchid calathea bonsai #> 40 30 25 12 5 4 #> alocasia carnivorous #> 3 2 

  

  





Each order value scatters around its family's true average with the spread you have known since Lesson 2: about $18 from one order to the next. Now do what any spreadsheet would: average each family and sort. The league table, and underneath it, the wobble each entry carries (one standard error, $18 divided by the square root of that family's order count):




  

    
RInteractive R

    
raw <- tapply(orders$value, orders$family, mean)
round(sort(raw, decreasing = TRUE), 1)                  # the league table
#> carnivorous      bonsai      orchid    alocasia    monstera        fern 
#>        75.0        72.8        58.7        57.3        56.2        50.5 
#>   succulent    calathea 
#>        48.0        46.0 

round(18 / sqrt(table(orders$family)), 1)               # the wobble on each average
#> 
#>   succulent        fern    monstera      orchid    calathea      bonsai 
#>         2.8         3.3         3.6         5.2         8.0         9.0 
#>    alocasia carnivorous 
#>        10.4        12.7 

    

      
      
    

    

  

  





Carnivorous plants top the table at $75.00. That number is the average of exactly two orders, and its standard error is $12.7: the raw average of a two-order family routinely misses its own truth by $25 in either direction. Succulents, at the bottom with $48.0, carry a wobble of just $2.8, because 40 orders pin an average down hard. The table is sorted by a mixture of two things: how good each family really is, and how lucky its handful of orders happened to be. The smaller the family, the more the luck dominates, which is why the extremes of any league table of small groups are usually the tiny entries, not the great ones.