EDA Exercises in R: 50 Real-World Practice Problems

Explanation: str() is almost always the first call against an unknown data frame because it answers "how big, what types, what do values look like" in one pass. Wrapping it in capture.output() lets you store the report as a character vector for further inspection or logging. Alternative: tibble::glimpse() produces a tidier, column-oriented view (Exercise 1.4). Common mistake: assigning str(x) directly returns NULL because str prints as a side effect.

Explanation: Looking only at head() misses the symptom of a file that has trailing summary rows, footer text, or values that drift over time. Combining head and tail in one tibble keeps both eyes on the same printout. Common mistake: forgetting that mtcars has row names, so head() returns rows but the row-name column is not a real variable. Use tibble::rownames_to_column() to convert.

Explanation: summary() adapts to the column type: numeric columns get the five-number summary plus mean; factors get counts of each level; logicals get TRUE/FALSE counts. It's a one-stop "is this column sane" check. Common mistake: with character columns it gives only length and class, not value counts. Convert chars to factors first if you need category tallies, or use skimr::skim() (Exercise 7.1) for a richer report.

Explanation: glimpse() rotates str() so each column becomes a row. With wide tables of 30+ columns it is much easier to scan than the default print. The type tag (<dbl>, <ord>, <chr>) tells you whether the column is numeric, ordered factor, character, etc. For a more visual scan with histograms and missingness, see skimr::skim() in Exercise 7.1.

Explanation: class(x)[1] keeps only the primary class for ordered factors (which return c("ordered","factor")). This is a robust one-liner for understanding the column-type mix before you decide which exploration tools fit best. Common mistake: skipping [1] returns a list and table() collapses ordered factors into multiple buckets. For wider audits, purrr::map_chr(diamonds, ~ class(.x)[1]) is the tidyverse equivalent.

Explanation: object.size() gives an estimate of the in-memory size of the R object, useful when deciding whether to subsample before plotting. Coercing everything to character with names yields a tidy printout that you can paste into a Slack message or PR. For a richer profile, pryr::object_size() handles shared references (e.g., copy-on-modify) more accurately than the base function.

Explanation: Tidy-select helpers where(is.numeric), where(is.factor), where(is.character) are the modern idiom. They avoid the trap of sapply(df, is.numeric) returning a logical vector you then need to subscript. Common mistake: diamonds$cut is an ordered factor, which is still is.factor() TRUE; ordered status doesn't matter for the type split unless you handle order semantics in modelling.

Explanation: vapply() is preferred over sapply() here because it enforces the return type per column, so the resulting tibble is type-stable. A data dictionary like this is invaluable when handing off a dataset: reviewers see uniqueness (constant columns? hashy-looking IDs?) and missingness in seconds. For wider profiling, DataExplorer::introduce() returns a similar one-row summary.

Explanation: Histograms are the workhorse univariate plot for continuous variables. The bin count drives the impression: too few bins hide structure, too many create noise. 30 to 50 bins is a reasonable default for a five-figure dataset. Common mistake: using the default geom_histogram() (30 bins) silently leaves a warning visible to readers. Always set bins or binwidth explicitly so reviewers see your choice.

Explanation: Densities sum to area 1, while counts depend on bin width. To overlay them, rescale the histogram with after_stat(density) (formerly ..density..). Common mistake: forgetting the rescale yields a flat density line against the much larger count bars. The density bandwidth is the smoothing parameter; if the curve looks too wiggly, set geom_density(adjust = 1.5) to smooth more.

Explanation: The IQR equals Q3 minus Q1 and is the basis for Tukey's outlier rule (1.5 * IQR beyond the quartiles). Always pass na.rm = TRUE when working with airquality since Ozone has 37 missing values. The default quantile() uses type 7, which is fine for EDA; type 8 is preferred when you need to compare with continuous distributions.

Explanation: A skew ratio of 0.35 means the mean sits about a third of an IQR above the median, a clear sign of right skew. Compared to skewness coefficients (Exercise 6.3), this ratio is dimensionless and easy to read. A common follow-up is to log-transform the column (Exercise 2.7) and recompute; for log-price the ratio collapses to near zero.

Explanation: Pivoting to long format first lets you write a single summarise() call rather than across() with named outputs (which often produces wide ugly column names like mpg_q1). The long-then-summarise pattern is also faster to read for a reviewer who already speaks dplyr. For a one-shot solution, psych::describe(mtcars) returns this and more in one call.

Explanation: percent_rank() maps the smallest value to 0 and the largest to 1, with ties getting their shared rank. It is preferred over rank() / n() for ties handling. Common mistake: using min_rank() returns integer ranks where the biggest values get the smallest numbers if you forget desc(). For cumulative distribution rather than rank, see dplyr::cume_dist() which returns the proportion of values at or below each observation.

Explanation: When a positive variable spans multiple orders of magnitude, a log transform usually makes the distribution closer to symmetric, which helps both visual inspection and downstream linear models. log10() is preferred for read-back ($10^{3.4} \approx 2500$); use log() (natural log) when computing skewness or fitting linear models if you want coefficients that read as elasticities. Common mistake: applying log() to a column with zeros or negatives produces -Inf or NaN; add a small offset or use log1p() for counts.

Explanation: Bimodal distributions are a strong signal that the data is a mixture of two underlying processes, and statistics like mean and standard deviation become misleading. For faithful, the two peaks correspond to two physical eruption regimes. Common mistake: applying a normality test or fitting a single Gaussian masks this structure. The clean follow-up is mixture modelling with mclust::Mclust() or just stratifying the analysis above and below the dip.

Explanation: count(sort = TRUE) is shorthand for group_by() |> summarise(n = n()) |> arrange(desc(n)) and is the cleanest idiom for a frequency table. For row proportions, add mutate(pct = n / sum(n)). Common mistake: using base table() returns an object that prints differently and is harder to pipe through downstream. For tibble output, prefer count().

Explanation: The Titanic object is a 4-D contingency table; as.data.frame() unrolls it into a tidy long form with a Freq column. Once it's tidy, group-and-summarise is the standard pattern. Common mistake: using prop.table(Titanic, margin = 1) works on the array but gives a 4-D proportions array that is hard to read. Pivot to long form first.

Explanation: fct_infreq() orders factor levels by descending frequency, which is the most common ordering for bar charts because it puts the most important bars on the left. Compare with fct_inorder() (order of appearance), fct_rev() (reverse), and fct_reorder() (order by another variable). Common mistake: reordering the column but plotting with coord_flip() flips the visual order, so combine fct_infreq() with fct_rev() for a top-to-bottom horizontal bar chart.

Explanation: fct_lump_n() keeps the n most frequent levels and lumps the rest. Related: fct_lump_prop() lumps levels below a proportion threshold; fct_lump_min() lumps levels with fewer than k observations. Common mistake: choosing n too high defeats the simplification; pick n based on how many bars fit comfortably in your chart (usually 5 to 8). The "Other" level is placed last by default; use fct_relevel() if you want it first.

Explanation: xtabs() accepts a frequency-weighted formula, which is perfect for a pre-aggregated dataset like Titanic. prop.table(margin = 1) divides each row by its total, giving conditional probabilities of survival given sex. Use margin = 2 for conditional given survived status. Common mistake: omitting the margin gives cell proportions (each entry divided by the grand total), which is rarely what you want for a 2x2 contingency analysis.

Explanation: Combining fct_infreq() (descending by count) with fct_rev() (reverse) plus coord_flip() produces a horizontal chart with the longest bar at the top, which is the conventional reading order. Common mistake: forgetting fct_rev() produces a chart with the largest category at the bottom, which most readers find jarring. For multiple-panel comparison, swap coord_flip() for geom_bar() + facet_wrap().

Explanation: Mosaic plots scale better than grouped bar charts when both axes have several levels because each cell area encodes a count directly. Reading rule: width of a column shows the marginal frequency of the column variable; height within a column shows the conditional distribution. For a ggplot2 equivalent, see ggmosaic::geom_mosaic(). Common mistake: treating Titanic as a data frame; here, mosaicplot() accepts the table directly through a formula.

Explanation: is.na() returns a logical matrix; colSums() treats TRUE as 1 to give per-column counts. Sorting puts the worst offenders first, useful when deciding which columns to impute, drop, or model the missingness for. For percentages, divide by nrow(). For wider tables, the equivalent tidyverse call is naniar::miss_var_summary() which adds percentage and pct-of-total directly.

Explanation: gg_miss_var() returns a ggplot object, so you can layer titles and themes on top. The percentage view is more useful than counts when comparing columns of different sample sizes. Related plots in naniar: gg_miss_upset() (combinations of co-missingness), gg_miss_case() (rows with most missing), and vis_miss() (full matrix heatmap). Common mistake: ignoring co-missingness; two columns can each have 5% NA but if they co-miss, listwise deletion still drops 5%, not 10%.

Explanation: complete.cases() returns a logical vector with TRUE for rows that have no missing values. Taking the mean of the logical gives the proportion, which scales to a percentage by multiplying by 100. Common mistake: dropping the 27.45% of incomplete rows before checking which columns drive the loss; if a single column accounts for most missingness, imputing it or dropping the column itself preserves more rows than listwise deletion.

Explanation: The 1.5 IQR rule is a sensible default for symmetric data but flags more points than expected on skewed distributions like ozone. For heavy-tailed data, the 3 IQR rule (used by some boxplot variants) is less aggressive. Common mistake: applying the rule to log-transformed data and then back-transforming; the outliers identified on the log scale may not match the linear-scale outliers. Always document which scale the rule was applied on.

Explanation: Z-score thresholds assume a roughly normal distribution; for heavy-tailed data, the threshold should be higher (often 3 or 3.5). The z-score rule and the IQR rule (Exercise 4.4) frequently flag different points because they use different measures of spread; use both as cross-checks. Common mistake: computing z within a grouped subset but using the global mean/sd. For grouped outliers, use group_by() before the mutate.

Explanation: Winsorization preserves sample size (no rows dropped) but blunts the influence of extremes on means, regressions, and correlations. It is widely used in finance and operations research. Tradeoff: aggressive winsorization (e.g., 1/99 or 5/95) distorts the tail and biases tail-aware estimators like value-at-risk. Common mistake: winsorizing the response variable but not predictors when fitting a regression. For symmetric models, applying it to both or neither is more consistent.

Explanation: A 6.5% drop in mean from trimming the top 1% tells you that one in a hundred high-priced diamonds drives a meaningful chunk of the average. Whether that is "fraud-like" depends on context: for diamonds it is just the heavy-tail nature of the inventory; for credit card transactions, that same pattern would be a red flag. Use this delta-mean analysis to decide whether the tail is informative or noise.

Explanation: Median imputation is the simplest possible imputation and biases variance estimates downward because it shrinks the spread of the imputed column. It is fine for quick EDA and bad for downstream inference. For better imputations, see mice::mice() (multiple imputation) or missForest::missForest() (model-based). Common mistake: imputing using the mean when the column is skewed; median is robust to skew and is the safer default.

Explanation: Encoding a third variable through size is fine for one or two extra dimensions but becomes hard to read past that. For more dimensions, prefer faceting (Exercise 5.2) or GGally::ggpairs() (Exercise 5.5). The loess smoother is non-parametric so it captures curvature better than lm, useful early in EDA before you commit to a functional form. Common mistake: setting se = TRUE on small data produces enormous confidence ribbons.

Explanation: Faceting decomposes a relationship by a categorical variable so you can spot interaction effects visually. Always sample large datasets (5k to 10k points) before plotting; ggplot2 can render 50k points but the output is opaque overplotting. For overplotting, set alpha low and add geom_smooth() to expose the trend. Common mistake: faceting without scales = "free" when panels have very different value ranges, which hides patterns in the smaller panels.

Explanation: cor() defaults to Pearson, which assumes a linear relationship and is sensitive to outliers. For monotonic but non-linear relationships, use method = "spearman" (rank-based). For mixed numeric and ordinal columns, Spearman is also the safer default. Common mistake: using cor() on a data frame with NAs without use = "pairwise.complete.obs" returns NA across the board; set it explicitly when missingness is present.

Explanation: Diverging palettes (e.g., RdBu, PiYG) are the right choice when zero is a meaningful midpoint. The scale_fill_gradient2() centring and explicit limits = c(-1, 1) keep the colour mapping comparable across plots. Alternative ready-made package: corrplot::corrplot() returns a heat-mapped, hierarchically-clustered correlation plot in one call. Common mistake: leaving the diagonal in (it always equals 1) inflates the visual saturation; consider masking it with geom_tile(data = filter(cor_long, var1 != var2)).

Explanation: ggpairs() is the fastest way to scan all bivariate relationships in a dataset with under ~10 numeric columns; beyond that it becomes too dense to read. The default upper triangle shows Pearson correlations, the lower triangle shows scatter, and the diagonal shows densities. Common mistake: passing the whole data frame including non-numeric columns; specify columns = 1:4 (or similar) to avoid noisy panels for factors.

Explanation: Boxplots compress a distribution to five summary stats per group, making cross-group comparisons fast. The "notched" variant (geom_boxplot(notch = TRUE)) draws confidence intervals around the medians; if notches don't overlap, the medians differ at roughly 95% confidence. For small groups, prefer geom_jitter() over a boxplot because a boxplot with 5 points is essentially decorative.

Explanation: Group-and-summarise is the workhorse of EDA. Always include n so a reader can spot under-powered groups (here, only 7 six-cylinder cars). Common mistake: omitting .groups = "drop" leaves the result grouped, which can cause surprises when piping into a subsequent mutate() or filter(). For dplyr 1.1+, you can write summarise(.by = cyl, ...) to express grouping inline without leaving the tibble grouped.

Explanation: Four-aesthetic plots are at the upper bound of what a human can decode in one glance. Use them deliberately and label every legend. For more than four variables, switch to faceting (one variable becomes panels) or ggpairs(). Tip: when encoding both colour and size, keep the size range tight (range = c(0.5, 3)) so the colour signal doesn't get drowned out.

Explanation: Multi-key groupings reveal interactions: 8-cyl cars are similar in mpg regardless of transmission, but manual transmission packs 100 more horsepower on average. Always inspect group sizes alongside means; a group of size 1 or 2 produces a near-meaningless average. For wide-format display, pipe into pivot_wider(names_from = am, values_from = c(mean_mpg, mean_hp)).

Explanation: Highest-absolute-correlation gives you a sensible first feature to include in a linear model and a benchmark against which more sophisticated feature-selection methods (lasso, mutual information, tree importance) must justify themselves. Common mistake: confusing high marginal correlation with high partial correlation; wt and disp are themselves highly correlated, so once wt is in the model the marginal value of disp drops sharply.

Explanation: A Q-Q plot compares the empirical quantiles of a sample to the theoretical quantiles of a target distribution (normal by default). Points lying on the reference line indicate the sample matches the target distribution. Departures: an S-curve indicates heavy tails; a U-curve indicates skew. For other distributions, swap stat_qq() for stat_qq(distribution = qexp) or use qqPlot() from the car package.

Explanation: Shapiro-Wilk is the most powerful normality test for small samples (n < 50) and remains useful up to about n = 5000. A p-value above 0.05 fails to reject normality, but with small n the test has low power, so "passing" Shapiro-Wilk does not prove the data are normal. Always combine with a Q-Q plot (Exercise 6.1). For large samples the test rejects on tiny departures; in that regime, judge normality visually instead.

Explanation: Sample skewness near zero indicates symmetry; positive values indicate a right tail; negative values a left tail. Values beyond +/-1 are commonly called "highly skewed". The collapse from 1.62 to nearly 0 after log transformation confirms the heavy right tail in raw prices and motivates analysing log-price for models that assume symmetric residuals. Related: moments::kurtosis() measures tail heaviness (3 for a normal distribution; >3 means heavier tails).

Explanation: Side-by-side density comparison makes the case for transformation more compelling than reporting a skewness number. The scales = "free" is essential here because the two scales differ by three orders of magnitude. For more than two transforms, pivot longer with the transformed values and facet by the transform name. Common mistake: comparing two density plots with the same x-axis when only one has been transformed; the plot becomes useless.

Explanation: Standardization is a prerequisite for distance-based methods (k-means, kNN, PCA) because columns with larger raw scales would otherwise dominate. After scaling, the mean is exactly 0 and the sd is exactly 1 by construction, but the rounding shows floating-point noise around 1e-16 for the mean. Common mistake: scaling categorical columns with one-hot encodings; scale only the continuous columns or use recipes::step_normalize() to handle the selection automatically.

Explanation: skim() is the single most useful one-liner for first-pass EDA: it groups by column type, reports completeness, summary stats, and tiny in-text histograms for numeric columns. It returns a tibble subclass so you can index it like a normal data frame after the fact. Compare to summary() (Exercise 1.3) which is plainer and doesn't separate by type. For larger reports, see DataExplorer::create_report() (Exercise 7.2).

Explanation: create_report() calls rmarkdown to knit a template HTML page in one shot. It is great for a fast handoff to a stakeholder but bad as a polished deliverable: the plots are stock and the prose is auto-generated. Use it as your first pass, then cherry-pick the most informative plots into a hand-written report. Related: summarytools::dfSummary() and ExPanDaR::ExPanD() offer interactive alternatives.

Explanation: Bundling EDA artefacts into a list (or an R6/S4 object) keeps a reusable pipeline clean: each artefact has a name, the function returns a single thing, and downstream code (report generators, dashboards) can index by name. Common mistake: returning unnamed elements via c(a, b, c) flattens components together. Always use list() for heterogeneous return values.

Explanation: A small, well-typed function like this becomes the building block of a personal EDA toolkit. The vapply() calls enforce the return shape per column so the resulting tibble is type-stable; the num_or_na helper makes the numeric-only logic readable. For richer profiles, layer the function with quantiles, mode for factors, and a text snippet of values. Common mistake: using sapply() instead of vapply(), which silently produces a list when types vary across columns.