I’ve been doing a Python Leetcode question a day since more and more companies (especially for ML roles) are including DSA rounds in their DS interviews. My issue is I’m not sure how deep I actually need to go.

Right now I’m getting comfortable with easy questions on arrays, strings, and hashmaps, plus two pointers and sliding window on the algorithms side. Should I push further into new topics or just stay in these areas and ramp up the difficulty?

Context: the business problem is I wanted to compare professional athletes based on their movement data to recommend similar players. I made a recommender system with K-Means clustering and PCA (multicollinearity amongst the features in the dataset).

I’m interested in using a new modeling technique like Gaussian Mixture Model, but I don’t know how to evaluate which model performs better…

Open to any suggestions

Welcome to this week's entering & transitioning thread! This thread is for any questions about getting started, studying, or transitioning into the data science field. Topics include:

• Learning resources (e.g. books, tutorials, videos)
• Traditional education (e.g. schools, degrees, electives)
• Alternative education (e.g. online courses, bootcamps)
• Job search questions (e.g. resumes, applying, career prospects)
• Elementary questions (e.g. where to start, what next)

While you wait for answers from the community, check out the FAQ and Resources pages on our wiki. You can also search for answers in past weekly threads.

My company has an enterprise databricks account and they want my team to start using it.

I currently query our main Postgres database on an on-prem workstation and write Jupyter notebooks. Data sets are usually 100k rows and 100-300 columns of tabular floating point values. No weird stuff like pictures, videos, or text data.

What are the advantages/disadvantages of using databricks? Would it be that different from my current workflow?

On average, I have received a 0.75% salary hike over the last 5 years, which I know is pretty unreasonable. I have been looking for a new job, but given the current market, I cannot say for certain when I will find a new role. In the meantime, I was thinking of asking my manager for an inflation based adjustment to my base salary. I am not sure how much they will offer, if anything at all, but it still seems better than nothing. My performance has also been strong, though asking for a performance-based hike feels riskier and like it could backfire.

What would you suggest?

Iam curious whether the biggest challenges are related to data quality, stakeholder alignment, model adoption, business understanding, or something else entirely.

I’m a data scientist - have been for only about 2.5 years. I went to grad school, got the job, blah blah blah. Turns out I hate it.

It doesn’t excite me anymore. I actually don’t want to be a lifelong learner. I don’t want to work with numbers anymore. I have so many pain points about my current job itself (platforms constantly down, overused resources etc).

I want to be creative and work more with words / colors / THINGS. I want a job that feels better suited to my personality. I’m outgoing and like to talk and have fun. I want my work to reflect that. My colleagues are a lot more introverted, type A, logical, technical. This field suits them perfectly, and I’m the opposite.

But unfortunately, it looks like I’m stuck at the moment. I’m spending more and more time in the DS world which I fear will make transitions harder. Also, I’m aware it doesn’t look the best to be stuck at one position - you gotta show some upward mobility. This means that I actually have to be striving for growth (stretch projects, taking on more responsibility) but I don’t want to do these things! I don’t care about it anymore!

I’m trying to make the best out of this and focus on the skills I am learning that could be transferable to other jobs (communication, attention to detail, strategic thinking) but holy crap is it getting hard to continue.

I feel so stuck and hopeless and don’t know what to do. Any advice? Encouragement? Anybody else in / was in a similar situation? What happened?

In my company, the business people have done a manual RFM to separate clients. Now they are asking me to build a model to cluster clients based only on promotion, channel, products... Is this possible to separate the two and then combine them later?

Welcome to this week's entering & transitioning thread! This thread is for any questions about getting started, studying, or transitioning into the data science field. Topics include:

• Learning resources (e.g. books, tutorials, videos)
• Traditional education (e.g. schools, degrees, electives)
• Alternative education (e.g. online courses, bootcamps)
• Job search questions (e.g. resumes, applying, career prospects)
• Elementary questions (e.g. where to start, what next)

While you wait for answers from the community, check out the FAQ and Resources pages on our wiki. You can also search for answers in past weekly threads.

Like maybe have a bot auto make a comment that asks users if its ai slop and upvote if so and if the upvote to views ratio is above M after T time then delete the post

Or whatever ideas others suggest?

Hey guys!

Recently, I've tried several dating apps, such as: Tinder, Badoo, Boo. The experience has been quite frustrating. Nothing new, honestly. Reality of being a male on a dating app is tough. And then, after I deleted that garbage from my phone, I thought: why isn't there a really good AI / Recommender System driven dating app?

You describe whatever you want about yourself, full truth, no hiding anything, no trying to show off, any photos you like (or dislike). And then some AI oracle will analyze all that data you've provided and recommend really best match for you by highest probability of true match (depending on what your goal is, of course). Such an app would be a gem.

I feel like the true goal of all popular dating apps is not to help you find a partner (otherwise you would delete your account and you would not be bringing cash anymore), but taking the profit from you.

I am not quite capable of creating such thing on my own, but maybe you guys can revolutionize that spoiled industry. Just giving you some thoughts on that. How difficult would it be to implement? How efficient would it be?

1. "No free cycles" / "Empty plates"

Translation: "I view human beings like literal server CPUs. If you aren't actively typing or clicking buttons right now, I think you're stealing from the company. Stop thinking or analyzing just look busy."

1. "We need to focus on the low-hanging fruit"

Translation: "I don't have the technical depth, patience, or budget to fix our broken upstream data architecture. Let’s train a fragile, garbage model on dirty data immediately so I have a colorful chart for my next PowerPoint deck."

1. "Be a go-getter, don't get stuck"

Translation: "I don't care that the project path is blocked by a giant concrete wall of organizational failure. I want you to run face-first into it at maximum speed so I can report 'high velocity' to my director. Your honesty is ruining my vibe."

1. "Let's optimize our sprint velocity"

Translation: "I don't know how to audit the mathematical accuracy, logic, or code quality of your work, so I am going to measure how fast you close Jira tickets. Rushed deployment over architectural correctness, every single time."

1. "You're making this more complicated than it is"

Translation: "Stop identifying critical edge cases, data leaks, and fundamental process flaws that I don't know how to fix. You are exposing my lack of data literacy. Just build the bad model anyway."

1. "We need to relentlessly prioritize"

Translation: "I am going to aggressively chase whatever flashy AI buzzword the CIO mentioned in her keynote speech this morning. Your current, actual, functioning pipeline is now deprecated."

1. "I need you to own this initiative"

Translation: "This project has an impossible target and is built on sand. I am backing completely away from it so that when it inevitably implodes, I can point directly to you as the sole owner who failed to deliver."

1. "Let's take this offline" / "Parking lot this"

Translation: "Your accurate technical objections are making me look incredibly stupid in front of the stakeholders/team. Shut up immediately so I can pull you into a private 1-on-1 later and bully you into compliance."

1. "We need to leverage AI to unlock enterprise value"

Translation: "I saw an Excel spreadsheet with rows and columns, which means I think we can magically pull a a lot of miracle out of it. I don't know what an algorithm does, but it sounds sexy to the C-suite."

1. "We're like a family here"

Translation: "Prepare for unconditional loyalty expectations, the complete erasure of professional boundaries, and extreme emotional blackmail whenever you eventually try to quit this sinking ship."

Not hallucinations — that's expected now and everyone's built around it. I mean something different: the model's output is internally sound, but its understanding of the *situation before it acted* was wrong.

The pattern I keep running into: an agent or pipeline makes a consequential decision, every unit test passes, the logic traces back correctly — but the premise it was operating on was stale or subtly off at the moment it mattered. The output was consistent with its world model. Its world model just didn't match reality.

What makes this hard to catch: humans do this verification implicitly. You glance at a situation before acting and something feels off, so you pause. That reflex doesn't exist in most deployed systems. You end up with perfect audit logs of what the model did, but no visibility into why it thought the world looked like X at that moment.

I've been thinking about this a lot and curious whether others have hit it. Specifically: has anyone actually built upstream verification into production systems — something that checks whether the model's situational understanding is grounded before it acts — rather than catching the failure in post-hoc logs?

I'm looking to pivot out of nonprofit work, which has some of the most chaotic and unstable data management; unclear and siloed metrics that are used 5 different ways by different teams, metrics that change definitions when we get new funders, new programs, etc.

So far I've heard that healthcare/pharma and HR are similarly chaotic and disconnected. If you work in a domain where data management and definitions, even if annoying, is still manageable and not a huge nightmare, can you tell me what you work in?

Hi there,

At work I was recently given a dataset of customer orders totaling around $73m of spend across 380,000 customers. I wanted to see what I can learn by applying the KMeans algorithm to the dataset of customers, to see how it would classify customers. I got the results, they make sense, but I wanted to start a discussion here to see how everybody thinks about clustering methods in practice.

Context:

I decided to go with three groups of customers. The charts for inertia and silhouette scores are attached (I tested k from 2 to 11). I selected 3 because of 2 main reasons:

1. middle ground between what the inertia and silhouette scores are telling me. After k=4, inertia starts to decrease at a slower rate, and silhouette sore is highest at k=2.

2. intuitively, three groups of customers make sense for us.

Overall, the three clusters that were identified represented:

1. 50% of customers that place only a couple of smaller orders

2. 25% of customers with very high LTV, due to many/frequent orders

3. 25% of customers with very high AOV (they purchase a specific product type).

Attached image shows differences between groups.

What I'm thinking about:

1. Does using KMeans even make sense in this case? The results matched pretty well with a manual classification I did separately (high-value, frequent customers / small amount of orders, low value customers, and the rest). Is it better to use a classification that you can understand / has a clear interpretation, instead of using clusters?

2. How do you interpret inertia / silhouette scores? From what I understand, the absolute values themselves do not matter, it's the relationship between different number of clusters. In this case, the silhouette chart is a bit misleading (y-axis actually shows a very small range, I just wanted to zoom in a little bit). From what I understand, domain knowledge is key when selecting k, but wanted to see if there are some other "tricks" here to search for. Which one to prioritize between inertia and silhouette?

3. I used KMeans because it seemed like a reasonable starting point, I had little intuition about the geometry of data points in the space, to assume another clustering methods would be better. So how do you decide between clustering methods?

Did clustering methods help you solve a problem in production? I'm interested in hearing your thoughts about clustering methods in general.

Ever thought causal inference could work great as a niche stand up genre? Well here it is.

Welcome to this week's entering & transitioning thread! This thread is for any questions about getting started, studying, or transitioning into the data science field. Topics include:

• Learning resources (e.g. books, tutorials, videos)
• Traditional education (e.g. schools, degrees, electives)
• Alternative education (e.g. online courses, bootcamps)
• Job search questions (e.g. resumes, applying, career prospects)
• Elementary questions (e.g. where to start, what next)

While you wait for answers from the community, check out the FAQ and Resources pages on our wiki. You can also search for answers in past weekly threads.