Navigate
Courses Tutorials Practice Pricing Tools Upgrade
Lesson 3 of 3

Outliers & Automated EDA

Across Lessons 1 and 2 you met Maya's bakery and one number that kept causing trouble: a single $905 day, a one-off street-festival order. It dragged her average revenue above a typical day (Lesson 1), and a stray point like it can quietly distort a correlation (Lesson 2). That lone value has a name, an outlier, and handling it well is the difference between an honest analysis and a misleading one.

This lesson tackles the two things real data forces on you. First, finding outliers and deciding what to do with them, like Maya's $905 day. Second, doing the whole first look at scale: Maya had 30 numbers you could read by eye, but real datasets have dozens of columns and thousands of rows. The boxplot below is that $905 day, caught: notice it sitting alone, far past the whisker.

By the end of this lesson you will be able to:

  • Spot outliers with the 1.5 x IQR rule and a boxplot, in R
  • Decide what to do with one: investigate, then keep, cap, transform or drop, and document it
  • Scan an entire data frame in a single command with skimr and DataExplorer

Prerequisites: you can run R and load a package with library(), and you have met one-variable EDA in Lesson 1 (histogram, mean vs median, IQR, the boxplot) and correlation in Lesson 2. Every new term is defined as it appears.

The troublemaker

What an outlier is, and why it matters

An outlier is an observation that sits far away from the rest of the data, far enough that it looks like it might belong to a different story. Maya's $905 festival day is the textbook case: 29 ordinary days between $180 and $410, and then one value more than double the next-highest.

Why fuss over a single point? Because one outlier quietly bends the summaries you rely on:

  • It drags the mean. From Lesson 1, Maya's mean was $298 but her median $273; drop the festival day and the mean falls to $277 while the median barely flinches.
  • It inflates the spread. The standard deviation and the range are stretched wide by one far point; the IQR, the middle-half width, is not.
  • It can fake or hide a correlation. As you saw in Lesson 2, a single stray point can swing Pearson's r up or down.
  • It can wreck a model that minimises squared error, because the squared distance to a far-away point is enormous.
Key Insight
An outlier is not automatically a mistake to delete. It is a flag that says look closer. Sometimes it is a typo; sometimes it is the most interesting day of the month. Your job is to find it, understand it, and then choose, on purpose, what to do.