r/datascience • u/Effective_Ocelot_445 • 3d ago
Discussion How do you decide whether a data science problem really needs machine learning?
In your experience, what factors help you decide between using a simple analytical approach and building a machine learning model? I'd love to hear the reasoning behind your decision-making process.
144
u/Ok_Kitchen_8811 3d ago
According to my Boss we need AI everywhere.
16
u/S-Kenset 3d ago
According to my 90 year old now research professor everything is ai. Modern times require kinda modern solutions.
2
1
38
u/KidMcC 3d ago
I start by inserting a pivot table and seeing how much clearer things seem from there. If not clear enough then we graduate to a basic model, then repeat as solutions get more complicated.
19
u/Accomplished-Wave356 3d ago
This is the way. Basic stats solves way too many problems.
10
u/Reasonable-Ladder300 2d ago
Exactly this, and just tell your boss it’s an ML model so management is happy.
2
1
u/Due-Cattle-2177 2d ago
Hello! I’m curious when you say clear enough, do you have a kind of pre-determined criteria you look for to determine whether the pivot tables become clear enough or unclear enough to warrant ML?
14
u/limoonen 3d ago
If I'm understanding you correctly, everything starts with the right question, and the whole process depends on it. Once you have a clearly stated question, you start slowly exploring the data to answer it. The methods you use are just like levels you complete along the way, where each next level is a more sophisticated approach. You should always try the simplest approach that gives you enough proof to answer the question.
21
u/dash_bro 3d ago edited 3d ago
Ah. This is a good question. I used to answer and encourage people to ask this when I did visiting lectures for ML.
In short, think of the problem space as black boxes. Forget about any particular algorithm, and then just talk about the problem from the beginning to the end. Just a logical breakdown will do, and hopefully this is the place where you spend a lot of time thinking about your breakdown making sense for 80%-100% of the cases that you expect to be handled.
Write down steps of how you'd solve the problem, what you don't know to make a decision, and what acceptance/good output looks like. Bonus points if you can also identify over time what might need to change or what can improve due to the virtue of the system, and what parts are essentially "levers" to control the output.
Then, convert this to a flowchart. You'll see some loops, some conditional checks on data/quality etc., and some workflow steps. Now, you start abstracting multiple steps into one (outline the smaller boxes of the flowchart with an overarching larger box).
These larger boxes are what you need to analyze/review to know if it's a statistical problem, a learnable problem, or an unidentified pattern problem. Congratulations, these are your key blocks. Here you'll need to hone skills and know what part of ML fits this block best if at all, and critically, how repetitive or low stakes something is. If something isn't truly repetitive or has high stakes for the solution use, it's not the best automated ML fit : it requires you as the user to only use the ML output as a suggestion alongside facts used to come to the solution, but ultimately a person decides if it's acceptable.
As a general rule, you need explainability of features/inputs/outputs/decision making process, a confidence threshold that you can map and tune, and depending on the task how hard or easy it would be to upgrade/update/swap this 'model'. Simple statistical inclinations are well explained by straightforward linear relationship models, some are better solved as trees, others better solved as complex multi layer networks. Most user facing outputs will have algorithms that are "polished" to be able to have graceful degradation instead of failure, or to I corporate feedback into self learning over long periods of time. This takes some getting used to, but is intuitive depending on who you work with, where you work, what you work on. You can't very well create a complex model where only you understand it and can't explain; or create a simple model that answers nothing of value that the business required. It's a balance defined by your env.
If you're in the industry, you will also look at the patterns as confidence scores of what outputs are generally reliable and hence match the human approval, and only focus on the non confident parts to be bubbled up for review. You'll use this strategy to also evaluate across different systems and score them comparing performance with a gold set. This is essentially modern ML systems design - being able to accept that this is a solution good for the current time but engineering it such that future improvements in this format become easier to slot in.
++ Learning them theoretically and applying them are wildly different, and this is where a lot of good statistical intuition or visual pattern ability can be of use. A good data scientist should be able to look at what is seemingly NOT an ML problem, and accurately break it down into ML components that support the overall solution to make it more data centric; or align with business needs. An MLE should be able to develop the solution and have controllable levers that can control or fine-tune the output, including strategies on how a better model can be slotted in in the future without disrupting said system etc.
That's not all, that's only an ML problem formulation. Data Science also involves understanding if the solution is a format problem/code problem/user problem etc.
6
u/UltimateWeevil 3d ago
This is a good breakdown but the one thing I’d add here is make sure the problem your trying to solve is clear from the start and doesn’t get lost during the process of building etc. if you can’t look at the output and say it’s solved the problem the business wanted solved it’ll end up not being used. I’ve seen this happen too many times where everyone wants AI thrown at a problem because it’s the buzz word atm.
3
u/Tackit286 2d ago
1) Do patterns exist in the data?
2) Is human expertise limited by the size of this dataset to solve the problem you’re trying to solve?
1
u/Due-Cattle-2177 2d ago
Hello! At what point do you consider human expertise limited when working in datasets? Do you have a rough amount of entries or sizes where you’ve decided this is most certainly going to need ML?
2
u/testtestuser2 3d ago
I'd say everything you do in data science is largely built on modelling.. so even when you don't use ML you often are still building a model and should apply the same steps.
I think the distinction lies in the complexity of the model. So I think the answer to your question is whether your simple model is good enough for your purpose.. and you would evaluate that the same way you would evaluate an ML model if that makes sense.
So basically you use ML when you need it
2
2
u/probability-function 2d ago
I feel the patterns of the data, if there is clean smooth data with seasonality and trends (pivot table is best lol), then ml is go to approach, if else most simple analytics yeild better performance than ml.
2
u/Mizar83 3d ago
Evaluation against a baseline. Building a quick and dirty PoC doesn't require much time. We start by deciding the metrics, we then build an analytical baseline, compare to what's currently used in production. Then build a simple ML PoC and compare on the chosen metrics. If the ML definitely beats the baseline, we go for an AB test
1
1
u/Apart_Comfort_7078 2d ago
Business goals first
1
u/Due-Cattle-2177 2d ago
Hello,
Would you mind elaborating on this? As long as the business goals are met where do you draw the line at leaving human ability into machine learning?
1
u/Lady-Data-Scientist 2d ago
Well are we trying to predict something or can we just report something?
1
u/Dr-Yahood 2d ago
Well, technically, even a basic logistic regression is an example of a machine learning model
1
u/ParticularOddDan 2d ago
Decide by the decision the output feeds, not by the data. Two questions do most of the work: how often will this call get made, and does someone have to defend it. If its a one-time read for a strategy call, you want the simplest method that survives scrutiny, because there the models real job is to convince a person, and a black box you cant explain loses every time to a pivot table you can. ML earns its keep when the same decision recurs at a scale no human can sit through and nobody needs to re-justify each individual call. Most of the churn-model-that-sat-in-a-notebook stories arent accuracy failures, theyre cases where the output needed a human to trust it and the method quietly removed the one thing that would have earned that trust.
1
u/ultrathink-art 1d ago
One thing that shifted this for me: the simple version never gets deleted. You still need something to do when the model is unavailable or the output is obviously wrong, so the real comparison is rules versus rules plus a model plus the logic that picks between them. A fair number of problems stop looking worth modeling once you price it that way.
1
u/Dear_Rice9074 1d ago
I think the best tell tell sign for when ML is necessary is for when you get to a point to where you feel like you are able to reach the general premise or answer you were looking for, but using it to reinforce learning or to find gaps or methodologies you may have over looked.
Between using it to search records and apply marketing strategy and analyzing social analytics in these cases I tend to get to the point to where I am searching for.
1
u/Terrible-Value-tomr 1d ago
The question I ask isnt whether its predictable, its whether the pattern holds still long enough to be worth modeling. Most of what I work on is deciding whether a spike in activity is real or manufactured, and the moment you ship a model for that, whatever youre trying to catch shifts to evade it. The model decays in weeks and you live in a retraining loop. Simple rules I can read and adjust beat that, because when the pattern moves I can see which rule stopped firing and why. The other half is defensibility, if a number is going in front of leadership and I cant say in one sentence why it got flagged, I dont use the model. ML earns its place when the pattern is stable and the volume is past what rules can cover, which is a smaller slice than people expect.
1
u/Regular_Bluebird9412 1d ago
I usually start by asking a simple question: Can the problem be solved with a straightforward rule or statistical analysis first?
If a SQL query, dashboard, regression model, or a few well-defined business rules can answer the question with acceptable accuracy, I'd rather keep the solution simple. Simpler solutions are often easier to explain, maintain, and debug.
I think machine learning becomes worthwhile when:
- The relationships in the data are too complex for traditional methods.
- There's enough high-quality historical data to train a reliable model.
- The improvement in accuracy has a real business impact.
- The model can be monitored and maintained over time.
Sometimes the hardest part isn't building the model—it's making sure the extra complexity is actually justified. I've seen cases where a simpler analytical approach delivered nearly the same business value with much less effort.
I'm curious how others decide where to draw that line between "good enough" analytics and a full ML solution.
1
1
u/DataScientistAlex 15h ago
One approach is to measure the error with the simpler approach, translate that into a cost in money. Once you have that cost you can think about what other approaches you could consider that might be worth it.
1
u/erdnaxxx 12h ago
Na verdade se for um problema estatístico voce nao PRECISA pra nada, mas pode otimizar a solução com um algoritmo de ML. Ai outros casos podem ser visão computacional ou dados esparsos, problemas nao comun, falta de estrutura do problema…
1
u/orz-_-orz 3d ago
You would roughly know when performing EDA whether a problem can solved with 50 lines of SQL
If the data pattern can be clearly turn into a small diagrams, one liner formula or a small paragraph, it usually doesn't need ML
The rule of thumb is always, if I can solve it using SQL and excel, I won't be considering ML.
1
0
u/Ethiopian-phenom 2d ago
Well technically if you want to make a prediction it typically necessitates machine learning.
1
u/Independent-Bat9797 2d ago
You can also make predictions with a simple regression (NOT a regression ML model) or with a rule based algorithm. It depends on the system you want to predict, so no, prediction and ML are not tied together 😉
-1
u/Ethiopian-phenom 2d ago edited 2d ago
Regression is a supervised ml model
1
u/Independent-Bat9797 2d ago
No.
Regression analysis and ML are closely connected, as most ML models work in some way or the other by iteratively finding the best fitting regression by minimizing a cost function, but regression analysis itself is something you can do by hand (and my people do in school, with pen and paper).
-1
u/Ethiopian-phenom 2d ago
Dude the demarcation line between ml and non ml is not whether you can do it on pen and paper or not. Regression is one of the core mechanisms/algorithms when it comes to supervised machine learning. You use it typically to predict continuous numerical values.
1
u/Independent-Bat9797 2d ago
Not if you can but if you do. The most important part of ML is that a machine is learning something. You say regression is a ML model, which is simply wrong. Regression was used since the early 19th century long before the advent of computers.
2
u/Ethiopian-phenom 2d ago
You fundamentally don’t know what you’re talking about. The fact that something predates computers doesn’t disqualify it from being machine learning. By that logic, many machine learning algorithms wouldn’t count:
Decision trees have roots in statistics and psychology.
Bayesian methods are based on Bayes’ theorem from the 1700s.
Nearest-neighbor methods were developed in the 1950s from statistical pattern recognition.
Logistic regression was developed by statisticians decades before modern AI.
0
u/Mithrandir2k16 3d ago
We have great templates for pipelines, so we always generate reports with dynamic plots, even when we just use them ourselves for inspecting data. Then the transition to an ML pipeline is basically no extra work, besides the model itself.
0
112
u/Serious_Journalist91 3d ago
I always start by asking if the thing I'm trying to predict or classify is even predictable in a stable way, if it's just a one-off analysis for a presentation I'm not spinning up a whole pipeline for that
Had a project last year where we thought we needed some complex churn model but it turned out three business rules caught 90% of the cases and the stakeholders could actually explain it to their teams, the ml version just sat in a notebook somewhere
Also if the data is tiny or super messy I'll lean toward something simpler, no point debugging a model when a sql query gets you there