Framing a Problem as Machine Learning

Priya runs the retention team at FreshBox, a meal-kit subscription. Her boss drops by with a familiar worry: "Customers keep canceling. Can we do something about it?"

That sentence is a business problem. It is not yet a machine learning problem, and no model on earth can be pointed at it as written. The most valuable, and most skipped, skill in applied machine learning is the translation: turning a vague question into a precise prediction task a model can actually learn. Get the framing wrong and even a brilliant algorithm answers the wrong question. This whole first lesson is that translation.

By the end you will be able to:

• Explain why a business question is not yet a machine learning problem, and name the four decisions that turn it into one
• Define a precise, computable target and say whether the problem is supervised, and classification or regression
• Keep only the features you will actually have at prediction time, and spot one that would leak the answer
• Choose a metric that matches the decision, and explain why plain accuracy can be worse than useless

Prerequisites: you can run R and read its printed output, and you know that a data frame is rows and typed columns. No prior machine learning is assumed.

A business question is not yet a machine learning problem

"Customers keep canceling, can we do something?" hides a dozen unstated choices. Do something for whom, when? Predict what, exactly? Measured how? A model needs each of these pinned down before it can learn anything useful.

To make the question learnable, Priya has to answer four concrete questions:

1. The unit of analysis. What does a single example look like? One customer? One customer each week? One delivery?
2. The target. What exactly are we predicting, and do we even have the answer for past examples to learn from?
3. The features. What will we actually know at the moment we make each prediction, and nothing we would not?
4. The metric. How will we score a prediction, in a way that reflects what a mistake really costs?

The rest of this lesson answers all four for FreshBox, one at a time.

The whole journey: CRISP-DM in six phases

Before the four decisions, it helps to see where framing sits in the larger arc. The industry-standard map for a data project is CRISP-DM (the Cross-Industry Standard Process for Data Mining): six phases you cycle through, not a straight line.

• Business understanding: what decision are we trying to improve, and what would success look like?
• Data understanding: what data do we actually have, and at what grain (one row per what)?
• Data preparation: build the analysis table, one row per unit, with the target and the features.
• Modeling: fit candidate models to the training rows.
• Evaluation: score them on a metric that reflects the real decision.
• Deployment: put the predictions where the decision is actually made (here, a weekly call list).

Framing lives in the first two phases, and it quietly decides everything downstream. The good news: those two phases boil down to the four decisions from the last step. Let us line them up.

The four framing decisions

Here are the four decisions again, in the order you make them. This is the spine of the lesson: one step each from here on.

Notice the order is not arbitrary. You cannot define the target until you know what one row is; you cannot pick features until you know the target; and the metric only makes sense once you know what you are predicting and what the decision costs. Get the first one wrong and the rest inherit the mistake.

What has to come first?

Priya wants to turn "customers keep canceling" into a machine learning problem. Which set of questions must she answer first, before any model is chosen?

The unit of analysis: what is one row?

The unit of analysis is what a single row of your training table represents, one example the model learns from. It sounds obvious until you try to write it down, and it is the choice everything else hangs on.

For FreshBox, the decision is "each week, phone the customers most likely to cancel soon." The natural unit is one active subscriber, described as of a snapshot date (say, this Monday). A different choice, one row per delivery or per cancellation event, would answer a different question and line up with a different action.

Each lesson runs in a fresh R session, so we build FreshBox's data right here. These are 1000 past subscribers: for each we have some snapshot features, and, because they are historical, we already know what happened next.

One thousand rows, one per customer. days_to_cancel is NA for anyone still subscribed and a number for those who later canceled. That column is how we will build the answer to learn from, next.

What does one row mean?

In the cust table we just built, a single row represents which of these?

The target: what are we predicting, and do we have the answer?

The target (also called the label, or \(y\)) is the thing the model predicts. This is where a problem becomes supervised learning: we have past examples where the answer is already known, and we want a rule that reproduces it on new cases.

Formally, supervised learning fits a function \(\hat f\) that maps a customer's features \(\mathbf{x}\) (their tenure, skipped boxes, and so on) to a prediction of the target \(y\), learned from labeled past examples \(\{(\mathbf{x}_1, y_1), \dots, (\mathbf{x}_n, y_n)\}\). Here \(\mathbf{x}\) is one row of features, \(y\) is whether that customer churned, and \(n = 1000\) is how many labeled customers we have to learn from.

Two forks decide what kind of target you have:

• Do you have labels at all? If yes, it is supervised (learn to predict a known answer). If you only have features and want to find natural groupings, that is unsupervised learning (clustering), a different tool for a different question.
• Is the target a category or a number? A category ("churns" vs "stays") makes it classification. A number ("how many boxes next quarter") makes it regression.

We do have the answer for past customers: days_to_cancel records who canceled and when. Here are a few who did.

Look at customer 32: they canceled, but 44 days after the snapshot. Whether they count as a "churn" depends entirely on the window we choose, which is exactly the next decision.

Which kind of problem is this?

Priya decides to predict, for each subscriber, whether they will cancel within 30 days, recorded as yes or no, and she has that answer for thousands of past customers. What kind of learning problem is this?

Define the target precisely

A precise target is a rule a computer can evaluate, not a vague word. "Churn" becomes: the customer canceled within 30 days of the snapshot. Fill in the blank so the label uses a 30-day window. Anyone with no cancellation (NA) counts as no.

cust$churn <- ifelse(!is.na(cust$days_to_cancel) &
                     cust$days_to_cancel <= 30, "yes", "no")
table(cust$churn)
#>
#>  no yes
#> 920  80

The features: only what you will know at prediction time

A feature is an input the model is allowed to use. The rule that trips up more real projects than any other is simple to state: a feature must be knowable at the exact moment you make the prediction, and never later.

When Priya scores a customer on Monday morning, she knows their tenure, how many boxes they skipped, their support tickets, how long since they logged in, and their plan. She does not yet know days_to_cancel, because that only exists once a customer has actually canceled. Using it to predict churn would be looking into the future: the model would score a perfect 100% in testing and then fail completely in production, where that column is empty for everyone still active. That failure mode is called leakage, and Lesson 3 is devoted to it. For now, the framing move is to keep only the honest columns.

Spot the leak

You are choosing features to score a customer at the Monday snapshot. Which one must you throw out, because you would not actually have it at that moment?

The metric: match the score to the decision

The last decision is how you score a prediction, and it is where good framing quietly wins or loses. The trap is defaulting to accuracy, the fraction of predictions that are correct, because on a rare event it is deeply misleading.

Let \(\pi\) be the base rate, the fraction of customers who actually churn. A model that ignores everything and predicts "will not churn" for all 1000 customers is correct on every non-churner, so its accuracy is exactly \(1 - \pi\), with no skill whatsoever.

Ninety-two percent accuracy, and it would tell Priya to call no one. Accuracy is the wrong headline because it rewards a useless model. The fix is to score the thing the decision actually cares about. Priya's decision is concrete: the team can phone 100 customers a week, so what matters is who lands in that top-100 list. Two metrics measure that directly, where \(k = 100\) is the call budget:

\[ \text{precision} = \frac{\text{churners among the 100 we call}}{100}, \qquad \text{recall} = \frac{\text{churners among the 100 we call}}{\text{all churners}} \]

Precision asks "of the calls we make, how many were worth it?"; recall asks "of the customers about to leave, how many did we reach?". Choosing a call budget is really choosing a threshold on a risk score: call everyone above it. The widget below lets you slide that threshold and watch precision and recall trade off against each other, one number never telling the whole story.

Now score FreshBox's actual decision. The glm call below fits a quick risk score, one number from 0 to 1 per customer; how it works is the subject of later lessons, so treat it as a black box here. We rank every customer by that score, call the 100 highest, and measure precision and recall on that list.

Why is 92% not the win it looks like?

A model that predicts "will not churn" for every customer scores 92% accuracy on FreshBox's data. Why is that the wrong thing to celebrate?

The FreshBox framing spec

Four decisions, all made before a model was chosen. Written down, they turn "customers keep canceling" into a problem a model can solve, and a result Priya can act on.

Every downstream step, which model, which features to engineer, how to tune, is now well defined. That is what framing buys you: not a model yet, but a target worth modeling.

Frame a new problem

A clinic wants to cut missed appointments. A staffer can phone the 50 patients most likely to miss tomorrow's slots and offer to reschedule. Which framing fits that decision?

References

A few authoritative places to take this further:

• Provost and Fawcett, Data Science for Business - the book that hammers "start from the decision," with chapters on framing and expected value.
• CRISP-DM, the cross-industry standard process - the six-phase lifecycle this lesson follows, with the original references.
• Google, Rules of Machine Learning - hard-won field rules on framing a problem, choosing a metric, and avoiding leakage.
• An Introduction to Statistical Learning, ch. 2 (free PDF) - supervised vs unsupervised, classification vs regression, and how to measure accuracy honestly.

Lesson 1 complete

You can now turn a fuzzy business question into a well-posed prediction problem: fix the unit of analysis, define a precise and labeled target, keep only the features you will truly have at prediction time, and score the model on a metric that matches the decision. That framing spec is the foundation every later step is built on.

Next, Lesson 2: The Bias-Variance Tradeoff. You have a well-posed problem; now we meet the single idea that decides whether your model actually works on new data, and why a more flexible model is not always a more accurate one.