Text Mining Exercises in R: 20 Real-World Practice Problems

Explanation: str_split() returns a list (one element per input string) because the input is vectorised. The [[1]] unwraps the single result into a plain character vector. For a single string with simple whitespace splitting, strsplit(x, " ")[[1]] from base R works too. Prefer str_split_1() when you know the input is a single string: it returns the vector directly with no unwrapping.

Explanation: Order matters: lowercase first, then strip punctuation, then split. [[:punct:]] is the POSIX class that covers commas, apostrophes, hyphens, exclamation marks, and most ASCII symbols, but the dollar sign $ is classified as a currency symbol in some locales, so we add it explicitly. Splitting on \\s+ collapses any run of whitespace into a single delimiter, so accidental double spaces will not produce empty tokens.

Explanation: Building list-columns then unnest()-ing is the idiomatic tidyverse pattern for one-row-per-token tables. The lowercase happens before splitting so casing variants collapse. The reason for the doc_id column is forward compatibility: when you scale from one document to many, every downstream join (sentiment lexicons, tf-idf) keys on doc_id, so it pays to introduce it on day one.

Explanation: count(word, sort = TRUE) is dplyr's one-shot frequency table; it groups, tallies, and orders descending in a single call. slice_head(n = 5) is preferred over the older head() because it returns a tibble (preserves class) and chains cleanly. Stop-words like "the" and "to" dominate the top of any raw frequency table, which motivates the stop-word filter you will write in Section 4.

Explanation: Hapax legomena ratios (hapax / total tokens) are a classic lexical-diversity metric used in stylometry and authorship attribution. A high hapax ratio suggests rich vocabulary; a low one signals repetitive writing. The pipeline is identical to a top-N count until the final two steps: filter on n == 1 instead of slice_head(), then sort alphabetically for a stable presentation order.

Explanation: TTR is sensitive to document length: longer documents trend toward lower TTR because common words inevitably repeat. For length-comparable corpora, TTR is fine; for mixed lengths, switch to root-TTR (unique / sqrt(total)) or moving-average TTR over fixed windows. n_distinct() is dplyr's vectorised wrapper around length(unique(x)) and behaves correctly on factors and NAs.

Explanation: str_extract_all() returns a list (one vector per input string) because each tweet can hold multiple hashtags. unlist() flattens, then the standard normalise-dedupe-sort trio finalises the output. Using [[:alnum:]]+ is intentional rather than \\w+: \\w includes underscore in most regex flavours and you typically want to break hashtags on _ in social contexts.

Explanation: Here \\w is the right choice: Twitter/X handles allow underscore, unlike hashtags where users seldom embed underscores. Combining str_extract_all() with unnest() is the cleanest way to convert a variable-length-per-row extraction into a tidy long table where every row is one mention. Aggregating with count() then produces the per-handle tally directly.

Explanation: A production-grade email regex is famously long (the RFC 5322 spec runs hundreds of characters); this pragmatic pattern catches >99% of real addresses with much less risk of catastrophic backtracking. The {2,6} TLD bound is the load-bearing trick: without it, the pattern would greedily match into trailing words. For sue@partner.co.uk, the regex captures the full partner.co.uk because dots are allowed in the domain.

Explanation: The pattern captures $ then a run of digits-and-commas, optionally followed by a decimal portion. The non-capturing group (?:...) keeps the match as a single token (capturing groups would return only the inside on some functions). Note the limitation: the B in $89.5B is lost, so 89.5B becomes 89.5 not 89_500_000_000. For accurate magnitude handling you would post-process with a unit-suffix detector. This pipeline is a good starting point, not a complete solution for accounting-grade extraction.

Explanation: The anti-join idiom !word %in% stop_list is cleaner than anti_join() when the stop list is a plain vector; switch to anti_join(stop_tibble, by = "word") once your stop list comes from a tibble with metadata (lexicon source, language). Production pipelines often layer multiple stop lists: a generic English list (SMART, snowball), a domain-specific list (legal jargon, twitter handles), and a project-specific blocklist of names or boilerplate.

Explanation: Alternation order matters in regex: ing|ed|es|ness|s is greedy left-to-right but regex tries each alternative until one matches, so put the longest first to avoid running becoming runn (s match) instead of run (ing match). This toy stemmer is intentionally crude: it conflates tables with tabl and happiness with happi. Real stemmers (Porter, Snowball, Lancaster) encode dozens of rules; the SnowballC package is the standard wrapper when accuracy matters.

Explanation: The paste(toks[-length(toks)], toks[-1]) trick zips a vector with a shifted copy of itself, yielding pairs without an explicit loop. For trigrams: paste(toks[1:(n-2)], toks[2:(n-1)], toks[3:n]). The tidytext alternative is unnest_tokens(token = "ngrams", n = 2), which is more readable inside a pipeline but adds a dependency. For ad-hoc analysis or unit tests, the base-R shift trick is preferable.

Explanation: rowwise() makes the shift trick safe to run per document (without it, the unlisted vector would mix documents and produce phantom bigrams across the boundary). separate() splits each bigram for the stop-word filter, but the original bg column is kept via remove = FALSE so the final tibble retains the readable "climate change" form. This pattern (count, separate to filter, re-aggregate) generalises to any n-gram filtering task.

Explanation: Trigrams capture phrase-level signals that bigrams miss: we the people is a recognised political phrase whereas we the and the people separately are unremarkable. The triple-shift idiom (toks[1:(n-2)], toks[2:(n-1)], toks[3:n]) generalises straightforwardly: for 4-grams add a fourth shifted vector to the paste(). For real collocation discovery (statistically significant phrases, not just frequent ones), use pointwise mutual information (PMI) or the quanteda::textstat_collocations() helper.

Explanation: The two-step count() |> group_by(doc_id) |> mutate(tf = freq/sum(freq)) pattern is the standard tidy formulation of term frequency: count first to get raw co-occurrences, then normalise by document length. Normalising matters because a longer document mentions every term more often by accident; raw counts would falsely flag long documents as topically heavy on every term. tidytext's bind_tf_idf() computes this internally before applying IDF.

Explanation: Note distinct(doc_id, word) before count(word): that turns a raw frequency count into a document-frequency count (how many docs contain the term, regardless of how many times). The word phone appears in all 3 documents so its IDF is log(3/3) = 0, which means it has zero discriminative value and is filtered out here for clarity. In production, you would keep the zero-IDF rows so downstream joins do not introduce NAs. Different libraries use different smoothings: log(N / (1 + df)), log((1+N) / (1+df)) + 1, etc.

Explanation: slice_max(tf_idf, n = 1, with_ties = FALSE) is the modern dplyr idiom for "argmax per group": cleaner than filter(tf_idf == max(tf_idf)) because it deterministically breaks ties (here, with_ties = FALSE keeps the first). The inner join naturally drops zero-IDF terms (because phone was filtered from idf_tab), which is exactly the behaviour you want for tagging: tags should differentiate documents, so terms common to all docs are useless. This 3-line pipeline is the heart of every "auto-tag this document" service.

Explanation: Lexicon-based sentiment is the simplest sentiment model: assign +1 to known positive words, -1 to known negative words, 0 to everything else, then sum per document. The strengths are interpretability (every score traces back to specific words) and zero training cost; the weaknesses are no negation handling ("not good" still scores +1) and no intensifier weighting. Production systems layer a negation-flip rule (any negation token in the previous 3 words flips polarity) before reaching for ML-based models.

Explanation: Intensity normalisation prevents review length from dominating ranking: a 50-word review with 5 positive hits has intensity 0.10, whereas a 5-word review with 2 positive hits has intensity 0.40, so the second is clearly more enthusiastic per word. The denominator choice matters: dividing by total tokens (including stop-words) understates intensity in verbose reviews; dividing by content-word count is fairer but requires stop-word filtering. Both are valid; document your choice when shipping the metric.