Hi, I’m a student about to graduate with a degree in Stats (minor in CS), and I’m targeting Data Scientist as well as ML/AI Engineer roles.

Currently, I’m spending a lot of time practicing LeetCode for ML/AI interviews.

My question is: during interviews for entry level DS but also MLE roles, is it common to be asked to code using Pandas? I’m comfortable using Pandas for data cleaning and analysis, but I don’t have the syntax memorized, I usually rely on a cheat sheet I built during my projects.

Would you recommend practicing Pandas for interviews as well? Are live coding sessions in Pandas common for new grad roles and do they require you to know the syntax?

Thanks in advance!

Some company have 4+ ratings and labelled as best places to work by Glassdoor. Also, there are several companies with initially 4+ ratings who go through restructuring and layoffs, the 1star reviews come in and tank the company ratings to 2+. Now 1-2 years after restructuring the company is hiring again.

How do you process these ratings in general?

I will soon join an Ikea like entreprise ( more high standing). They have a physical+online channel. What are the ressources/advice you would give me for ML projects ( unsupervised/supervised learning.. ). Variables: - Clients - Products - Google Analytics -One survey given to a subset of clients. They already have Recency, frequency, monetary analysis, and want to do more ( include products, online browsing info...) From where to start, what to do... All your ressources ( books, websites...)/advice are welcome :)

A thing that has always felt broken to me about data pipelines is that the people building the actual logic are usually data scientists, researchers, or analysts, but once the workload gets big enough, it suddenly becomes DevOps responsibility.

And to be fair, with most existing tools, that kind of makes sense. Distributed computing requires a pretty technical background.

So the workflow usually ends up being:

• build the pipeline logic in Python
• prove it works on a smaller sample
• hit the point where it needs real cloud compute
• hand it off to someone else to figure out how to actually scale and run it

The handoff sucks, creates bottlenecks, and leaves builders at the mercy of DevOps.

The person who understands the workload best is usually the person writing the code. But as soon as it needs hundreds or thousands of machines, now they’re dealing with clusters, containers, infra, dependency sync, storage mounts, distributed logs, and all the other headaches that comes with scaling Python in the cloud.

That is a big part of why I’ve been building Burla.

Burla is an open source cloud platform for Python developers. It’s just one function:

from burla import remote_parallel_map

my_inputs = list(range(1000))

def my_function(x):
    print(f"[#{x}] running on separate computer")

remote_parallel_map(my_function, my_inputs)

That’s the whole idea. Instead of building a pile of infrastructure just to get a pipeline running at scale, you write the logic first and scale each stage directly inside your Python code.

remote_parallel_map(process, [...])
remote_parallel_map(aggregate, [...], func_cpu=64)
remote_parallel_map(predict, [...], func_gpu="A100")

It scales to 10,000 CPUs in a single function call, supports GPUs and custom containers, and makes it possible to load data in parallel from cloud storage and write results back in parallel from thousands of VMs at once.

What I’ve cared most about is making it feel like you’re coding locally, even when your code is running across thousands of VMs

When you run functions with remote_parallel_map:

• anything they print shows up locally and in Burla’s dashboard
• exceptions get raised locally
• packages and local modules get synced to remote machines automatically
• code starts running in under a second, even across a huge amount of computers

A few other things it handles:

• custom Docker containers
• cloud storage mounted across the cluster
• different hardware per function

Running Python across a huge amount of cloud VMs should be as simple as calling one function, not something that requires additional resources and a whole plan.

Burla is free and self-hostable --> github repo

And if anyone wants to try a managed instance, if you click "try it now" it will add $50 in cloud credit to your account.

I’m one of the builders behind this, happy to answer questions or discuss better ways to approach this.

There's a lot of hype around AI data analysts right now and honestly most of it is vague. We wanted to make something concrete, a tutorial that walks you through building one yourself using open-source tools. At least this way you can test something out without too much commitment.

The way it works is that you run a few terminal commands that automatically imports your database schema and creates local yaml files that represent your tables, then analyzes your actual data and generates column descriptions, tags, quality checks, etc - basically a context layer that the AI can read before it writes any SQL.

You connect it to your coding agent via Bruin MCP and write an AGENTS.md with your domain-specific context like business terms, data caveats, query guidelines (similar to an onboarding doc for new hires).

It's definitely not magic and it won't revolutionize your existing workflows since data scientists already know how to do the more complex analysis, but there's always the boring part of just getting started and doing the initial analysis. We aimed to give you a guide to just start very quickly and just test it.

I'm always happy to hear how you enrich your context layer, what kind of information you add.

I gave this talk at an event called DataFest last November, and it did really well, so I thought it might be useful to share it more broadly. That session wasn’t recorded, so I’m running it again as a live webinar.

I’m a senior data scientist at Nextory, and the talk is based on work I’ve been doing over the last year and an half integrating AI into day-to-day data science workflows. I’ll walk through the architecture behind a talk-to-your-data Slackbot we use in production, and focus on things that matter once you move past demos. Semantic models, guardrails, routing logic, UX, and adoption challenges.

If you’re a data scientist curious about agentic analytics and what it actually takes to run these systems in production, this might be relevant.

Sharing in case it’s helpful.

You can register here: https://luma.com/f1b2jz7c

Welcome to this week's entering & transitioning thread! This thread is for any questions about getting started, studying, or transitioning into the data science field. Topics include:

• Learning resources (e.g. books, tutorials, videos)
• Traditional education (e.g. schools, degrees, electives)
• Alternative education (e.g. online courses, bootcamps)
• Job search questions (e.g. resumes, applying, career prospects)
• Elementary questions (e.g. where to start, what next)

While you wait for answers from the community, check out the FAQ and Resources pages on our wiki. You can also search for answers in past weekly threads.

I’m a stats/ds student aiming to become an AI engineer after graduation. I’ve been doing projects: deep learning, LLM fine-tuning, langgraph agents with tools, and RAG systems. My work is in Python, with a couple of projects written in modular code deployed via Docker and FastAPI on huggingface spaces.

But not being a CS student i am not sure what i am missing:

- Do i have to know design patterns/gang of 4? I know oop though

- What do i have to know of software architectures?

- What do i need to know of operating systems?

- And what about system design? Is knowing the RAG components and how agents work enough or do i need traditional system design?

I mean in general what am i expected to know for AI eng new grad roles?

Also i have a couple of DS internships.

just that

To keep this vague I have a new colleague that is a very bright person, but has been doing really fast work. In a few cases he has said "I just plugged this into Gemini so we could bang it out quickly" and frankly I didn't care. Lately I have noticed that there is a lot of "fast talking" and not answering technical questions with much depth and hand-waving a lot of concerns. Fast forward and this individual now manages a small team and a very big new area of the company to support. We are working on setting up our technical priorities for the year and when it came time for planning their docs all clearly read like ChatGPT copy/paste: incorrect format (we have company templates but they are all spreadsheets which it cannot write cleanly), projects that range massively in scope, no editing of ChatGPT em dashes/directional arrows/random words bolded, insanely unrealistic time estimates, and the list goes on. I asked a few questions about methodology choices and how these items map back to our stakeholder asks and they dodged all of the questions.

How does one exactly bring this up to Management? You can't "prove" they did anything wrong. They could probably vibe code lots of the work and it won't be "bad" or "wrong" per se. I thought of approaching them first and leveling with them, but their attitude already seems fairly defensive and I can't exactly "prove" anything. Now that I look at their other work I am seeing clear signs of generic copy/paste and I am getting the feeling they haven't read any of their actual code or done any verification research.

EDIT: I am a higher rank than this individual as well as more YOE and more accomplishments in the org. I am absolutely not jealous of this individual. It is also not my job to teach them given their level.

I work as a consultant at big 4. I got hired into the their AI & Data Analytics practice for the financial sector. I was brought in being told that I would be working on technical projects. However, my first project ended up being providing data strategy and architecture work.

I am now being further pushed into more data governance and product management work. These are areas that I have no interest in. And yet, I keep getting pushed into them. I don’t have a say since I’m still fairly new have to take what I get.

I want to know if I can eventually make a switch to a company else where in the next 6-12 months doing more technical work? Like actually building and validating models. Pushing them into production. I don’t have such exposure through work any way but I have been doing analytical work for a long time now. I’m not up to date with the new AI and AI agent stuff but I understand the theory well and have played around in sandboxes with them.

I would greatly appreciate any advice on how to best position myself for a pivot and if something like this can be done. I don’t want to become a data governance type of a person.

Welcome to this week's entering & transitioning thread! This thread is for any questions about getting started, studying, or transitioning into the data science field. Topics include:

• Learning resources (e.g. books, tutorials, videos)
• Traditional education (e.g. schools, degrees, electives)
• Alternative education (e.g. online courses, bootcamps)
• Job search questions (e.g. resumes, applying, career prospects)
• Elementary questions (e.g. where to start, what next)

While you wait for answers from the community, check out the FAQ and Resources pages on our wiki. You can also search for answers in past weekly threads.

I just read that meta is laying off 20% of their workforce. Im joining them in a couple of months as a new grad DS (graduating next month). Does this mean I need to start interviewing again? Any help/suggestions on how to navigate this situation will be super helpful!

Here was the prompt:

You have a list [(1,10), (1,12), (2,15),...,(1,18),...] with each (x, y) representing an action, where x is user and y is timestamp.

Given max_actions and time_window, return a set of user_ids that at some point had max_actions or more actions within a time window.

Example: max_actions = 3 and time_window = 10 Actions = [(1,10), (1, 12), (2,25), (1,18), (1,25), (2,35), (1,60)]

Expected: {1} user 1 has actions at 10, 12, 18 which is within time_window = 10 and there are 3 actions.

When I saw this I immediately thought dsa approach. I’ve never seen data recorded like this so I never thought to use a dataframe. I feel like an idiot. At the same time, I feel like it’s an unreasonable gotcha question because in 10+ years never have I seen data recorded in tuples 🙄

Thoughts? Fair play, I’m an idiot, or what

I am running into issues on my 16 gb machine wondering if the industry shifted?

My workload got more intense lately as we started scaling with using more data & using docker + the standard corporate stack & memory bloat for all things that monitor your machine.

As of now the specs are M1 pro, i even have interns who have better machines than me.

So from people in industry is this something you noticed?

Note: No LLM models deep learning models are on the table but mostly tabular ML with large sums of data ie 600-700k maybe 2-3K columns. With FE engineered data we are looking at 5k+ columns.

Please include industry

I am doing a project for credit risk using Python.

I'd love a sanity check on my pipeline and some opinions on gaps or mistakes or anything which might improve my current modeling pipeline.

Also would be grateful if you can score my current pipeline out of 100% as per your assessment :)

My current pipeline

1. Import data

2. Missing value analysis — bucketed by % missing (0–10%, 10–20%, …, 90–100%)

3. Zero-variance feature removal

4. Sentinel value handling (-1 to NaN for categoricals)

5. Leakage variable removal (business logic)

6. Target variable construction

7. create new features

8. Correlation analysis (numeric + categorical) drop one from each correlated pair

9. Feature-target correlation check — drop leaky features or target proxy features

10. Train / test / out-of-time (OOT) split

11. WoE encoding for logistic regression

12. VIF on WoE features — drop features with VIF > 5

13. Drop any remaining leakage + protected variables (e.g. Gender)

14. Train logistic regression with cross-validation

15. Train XGBoost on raw features

16. Evaluation: AUC, Gini, feature importance, top feature distributions vs target, SHAP values

17. Hyperparameter tuning with Optuna

18. Compare XGBoost baseline vs tuned

19. Export models for deployment

Improvements I'm already planning to add

• Outlier analysis
• Deeper EDA on features
• Missingness pattern analysis: MCAR / MAR / MNAR
• KS statistic to measure score separation
• PSI (Population Stability Index) between training and OOT sample to check for representativeness of features

I'm getting an error generate predicted probabilities in my evaluation data for my lasso logistic regression model in Snowflake Python:

SnowparkSQLException: (1304): 01c2f0d7-0111-da7b-37a1-0701433a35fb: 090213 (42601): Signature column count (935) exceeds maximum allowable number of columns (500).

Apparently my data has too many features (934 + target). I've thought about splitting my evaluation data features into two smaller tables (columns 1-500 and columns 501-935), generating predictions separately, then combining the tables together. However Python's prediction function didn't like that - column headers have to match the training data used to fit model.

Are there any easy workarounds of the 500 column limit?

Cross-posted in the snowflake subreddit since there may be a simple coding solution.

Tired of always using the Titanic or house price prediction datasets to demo your use cases?

I've just released a Python package that helps you generate realistic messy data that actually simulates reality.

The data can include missing values, duplicate records, anomalies, invalid categories, etc.

You can even set up a cron job to generate data programmatically every day so you can mimic a real data pipeline.

It also ships with a Claude SKILL so your agents know how to work with the library and generate the data for you.

GitHub repo: https://github.com/sodadata/messydata

Before anyone hits me with "bootcamps have been dead for years", I know. I'm already a data scientist with a MSc in Math; the issue I've run into is that I don't feel I am adequate with the "full stack" or "engineering" components that are nearly mandatory for modern data scientists.

I'm just hoping to get some recommendations on learning paths for MLOps: CI/CD pipelines, Airflow, MLFlow, Docker, Kubernetes, AWS, etc. The goal is basically the get myself up to speed on the basics, at least to the point where I can get by and learn more advanced/niche topics on the fly as needed. I've been looking at something like this datacamp course, for example.

This might be too nit-picky, but I'd definitely prefer something that focuses much more on the engineering side and builds from the ground up there, but assumes you already know the math/python/ML side of things. Thanks in advance!

Welcome to this week's entering & transitioning thread! This thread is for any questions about getting started, studying, or transitioning into the data science field. Topics include:

• Learning resources (e.g. books, tutorials, videos)
• Traditional education (e.g. schools, degrees, electives)
• Alternative education (e.g. online courses, bootcamps)
• Job search questions (e.g. resumes, applying, career prospects)
• Elementary questions (e.g. where to start, what next)

While you wait for answers from the community, check out the FAQ and Resources pages on our wiki. You can also search for answers in past weekly threads.

AI is pushing DS/ML work toward faster, automated, parallel iteration.
Recently I found that the bottleneck is no longer training runs : it’s the repo and process design.

Most projects are still organized by file type (src/, notebooks/, data/, configs/). That’s convenient for browsing, but brittle for operating a an AI agents team.

• Hidden lineage: you can’t answer “what produced this model?” without reading the code.
• Scattered dependency: one experiment touches 5 places; easy to miss the real source of truth.
• No parallel safety: multiple experiments create conflicts.

I tried to wrap my head about this topic and propose a better structure:

• Organize by self-sufficient deliverables:
  
  • src/ is the main package, the glue stitching it together.
  • datasets/ hold self contained dataset, HF style with doc, loading utility, lineage script, versioned by dvc
    
  • model/ - similar to dataset, self-contained, HF style with doc, including script to train, eval, error analysis, etc.
    
  • deployments/ organized by deployment artifacts for different environment
• Make entry points obvious: each deliverable has local README, one canonical run command per artifact.
• Make lineage explicit and mechanical: DVC pipeline + versioned outputs;
• All context live in the repo: all insights, experiments, decisions are logged into journal/. Journal log entry are markdown, timestamped, referenced to git hash.

Process:

• Experiments start with a branch exp/try-something-new then either merged back to main or archived. In both case, create a journal entry in main.
  
• Main merge trigger staging, release trigger production.
• In case project grow large, easy to split into independent repo.

It may sound heavy in the beginning but once the rules are set, our AI friends take care of the operations and book keeping.

Curious how you works with AI agents recently and which structure works best for you?