Outliers & Automated EDA

Across Lessons 1 and 2 you met Maya's bakery and one number that kept causing trouble: a single $905 day, a one-off street-festival order. It dragged her average revenue above a typical day (Lesson 1), and a stray point like it can quietly distort a correlation (Lesson 2). That lone value has a name, an outlier, and handling it well is the difference between an honest analysis and a misleading one.

This lesson tackles the two things real data forces on you. First, finding outliers and deciding what to do with them, like Maya's $905 day. Second, doing the whole first look at scale: Maya had 30 numbers you could read by eye, but real datasets have dozens of columns and thousands of rows. The boxplot below is that $905 day, caught: notice it sitting alone, far past the whisker.

By the end of this lesson you will be able to:

• Spot outliers with the 1.5 x IQR rule and a boxplot, in R
• Decide what to do with one: investigate, then keep, cap, transform or drop, and document it
• Scan an entire data frame in a single command with skimr and DataExplorer

Prerequisites: you can run R and load a package with library(), and you have met one-variable EDA in Lesson 1 (histogram, mean vs median, IQR, the boxplot) and correlation in Lesson 2. Every new term is defined as it appears.

What an outlier is, and why it matters

An outlier is an observation that sits far away from the rest of the data, far enough that it looks like it might belong to a different story. Maya's $905 festival day is the textbook case: 29 ordinary days between $180 and $410, and then one value more than double the next-highest.

Why fuss over a single point? Because one outlier quietly bends the summaries you rely on:

• It drags the mean. From Lesson 1, Maya's mean was $298 but her median $273; drop the festival day and the mean falls to $277 while the median barely flinches.
• It inflates the spread. The standard deviation and the range are stretched wide by one far point; the IQR, the middle-half width, is not.
• It can fake or hide a correlation. As you saw in Lesson 2, a single stray point can swing Pearson's r up or down.
• It can wreck a model that minimises squared error, because the squared distance to a far-away point is enormous.

The 1.5 x IQR rule, and the boxplot that draws it

In Lesson 1 you flagged the $905 day with the 1.5 x IQR rule. Here it is in full, because it is the workhorse of outlier detection.

Recall the quartiles: \(Q_1\) is the value a quarter of the days fall below (the 25th percentile), \(Q_3\) the value three-quarters fall below (the 75th percentile), and the interquartile range \(\text{IQR} = Q_3 - Q_1\) is the width of the middle half of the data. The rule draws two fences, a distance of \(1.5 \times \text{IQR}\) beyond each quartile:

\[ \text{lower fence} = Q_1 - 1.5 \times \text{IQR}, \qquad \text{upper fence} = Q_3 + 1.5 \times \text{IQR} \]

Any value beyond a fence is flagged as an outlier. A boxplot is just a picture of this rule: the box spans \(Q_1\) to \(Q_3\), the line inside is the median, the whiskers reach to the most extreme values still inside the fences, and anything past a whisker is drawn as a separate dot. That lone dot above Maya's box is the $905 day.

Each lesson runs in a fresh R session, so let us build Maya's revenue here (run this once), then compute the fences and read off who is flagged:

R will even hand you the flagged points directly, no fence arithmetic required:

Is the biggest value always an outlier?

Maya's busiest ordinary Saturday brought in $410, the highest non-festival day. Does the 1.5 x IQR rule flag $410 as an outlier?

A flagged value is a question, not a verdict

Finding an outlier is the easy part. Deciding what to do with it is where analyses are won or lost, and the worst move is to silently delete it because it is inconvenient.

Start with one question: is this an error, or is it real?

• An error: a data-entry typo ($9,050 fat-fingered for $905), a broken sensor, two rows merged into one, the wrong unit. These are genuinely wrong, and should be fixed or removed.
• A genuine value: a real, rare event. Maya's $905 day actually happened, it was a one-off street-festival order. It is not wrong, just unusual.

The only way to know which you are looking at is to investigate: open the row, check it against another record, ask whoever collected the data. Maya remembers the festival, so her $905 is genuine. Follow this same path every single time:

Handling a genuine outlier

Once you know the $905 day is real, you have four honest options, and dropping it is rarely the best one.

1. Keep it, and report robust summaries. The median and IQR barely notice an outlier, so lead with them instead of the mean and SD. A trimmed mean, which chops a fixed fraction off each end before averaging, is another robust middle ground:

The plain mean claims a typical day is $298; the robust median and trimmed mean agree it is closer to $273 to $278.

2. Cap it (winsorize). Pull any value past the fence back to the fence. You keep the row but limit how far one point can pull a result:

3. Transform. A log transform pulls in a long right tail so one big value no longer dominates, handy before computing a correlation or fitting a model.

4. Drop it, but only with a written reason. Legitimate when the value is a confirmed error, or genuinely outside the question you are answering (Maya might exclude the festival day to forecast an ordinary day). Never drop a value simply because it spoils a result.

Compute a robust typical day

You want one number for a typical day that the festival outlier cannot inflate. Use a trimmed mean: fill in the argument that tells mean() to chop a fraction off each end before averaging.

mean(revenue, trim = 0.1)
#> [1] 277.7083

One command for the whole data frame

Everything so far worked on a single column of 30 numbers you could read by eye. Real data does not cooperate: Maya's full March log has several columns, and a serious dataset can have dozens of columns and thousands of rows. You cannot eyeball that. You need a tool that summarises every column at once.

First, build the full log here (run this once). It is Maya's revenue alongside daily foot_traffic, the weather, and pastries_sold, with one missing pastry count to keep things realistic:

The skimr package summarises the whole frame in one call. skim() groups the columns by type and, for each, reports the missing count, the mean and SD, the quartiles, and a tiny inline histogram:

Read what skim() surfaced for free. The p columns are percentiles: p0 and p100 are the minimum and maximum, and p25, p50, p75 are exactly the quartiles \(Q_1\), median and \(Q_3\) from earlier. So the revenue row shows a maximum (p100) of 905 with a mean far above the median (p50) and a right-skewed ▇▂▁▁▁ sparkline: the festival outlier, flagged automatically. The pastries_sold row shows n_missing of 1 and a complete_rate of 0.967: the missing value, caught without your asking. One call did the entire first pass over the table.

A clean summary table like that beats squinting at raw console output. Toggle the two views below to feel the difference:

DataExplorer: a full first pass in one function

skimr gives you the numbers; DataExplorer goes further and draws the whole report. Its fastest tool is introduce(), a one-row structural snapshot of the data:

In one line it confirms the shape: 30 rows, 4 columns, 1 discrete (the weather text) and 3 continuous, exactly 1 missing value, and 29 complete rows. From there, DataExplorer can plot a histogram of every numeric column, bar charts of every categorical one, a correlation heatmap, and a missing-value map, then bundle them all into a single shareable HTML report:

# the full visual report: one HTML file, a plot for every column.
# run this locally; it renders a document, so it needs a report engine.
library(DataExplorer)
create_report(bakery, output_file = "bakery_eda.html")

The histogram below is the kind of per-column picture that report draws automatically. It is Maya's revenue, with the festival day stranded in the lone far-right bin, the very same outlier surfacing yet again:

What automated EDA is for

skim() and DataExplorer summarised Maya's whole table in seconds. Does that mean you can skip looking at the data yourself and just trust the automated report?

References

A few authoritative places to take this further:

• R for Data Science (2e), Exploratory Data Analysis - the "Unusual values" section covers spotting outliers and the honest choices for handling them.
• skimr documentation (rOpenSci) - the one-line whole-data-frame summary you used here, and how to customise what it reports.
• DataExplorer documentation - introduce(), the plotting functions, and the one-call create_report() automated report.
• NIST/SEMATECH e-Handbook: detection of outliers - the formal methods, from boxplot fences to Grubbs' test, and when each one applies.

Course complete

You can now handle the messy reality of real data. You learned what an outlier is and why it bends the mean, the spread and a correlation; detected one with the 1.5 x IQR fences and a boxplot; decided what to do with Maya's genuine $905 festival day (investigate, then keep with robust stats, cap, transform, or drop with a written reason); and scaled the whole first pass to an entire data frame with skim() and DataExplorer.

That completes Exploratory Data Analysis in R. Across three lessons you went from one variable (shape, center and spread), to two (scatterplots and correlation), to the surprises and the automation that make a real first look trustworthy. EDA is the habit every good analysis starts with, and it is the foundation the visualisation and modelling courses build on next.