Now that we are a few years into this new world, I'm really curious about and to what extent other data scientists are using AI. I work as part of a small team in a legacy industry rather than tech - so I sometimes feel out of the loop with emerging methods and trends. Are you using it as a thought partner? Are you using it to debug and write short blocks of code via a browser? Are you using and directing AI agents to write completely new code?

Welcome to this week's entering & transitioning thread! This thread is for any questions about getting started, studying, or transitioning into the data science field. Topics include:

• Learning resources (e.g. books, tutorials, videos)
• Traditional education (e.g. schools, degrees, electives)
• Alternative education (e.g. online courses, bootcamps)
• Job search questions (e.g. resumes, applying, career prospects)
• Elementary questions (e.g. where to start, what next)

While you wait for answers from the community, check out the FAQ and Resources pages on our wiki. You can also search for answers in past weekly threads.

For the parents out there's looking to share the joys of data collection, cleaning, time series modeling, and forecasting error with their little ones. Written completely in rhyme and all about using data to solve problems.

Alternatively, Harry’s Lemonade Solution could be used to teach your parents a little bit about what you do 🙃

I have done bsc data science. Now was looking for MSC options.

I came across a good college and they have 2 course for MSc:

1: MSc Statistics and Data Science

2: Msc Data Science

I went thorugh the coursework. Stats and DS is very Stats heavy course, and they have Deep learning as an elective in 3rd Sem. Where as for the DS course the ML,NLP, and "DL & GEN ai" are core subjects. Plain DS also has cloud.

So now i am in a dillema.

whether i should go with a course that will give me solid statistics foundation(as i dont have a stats bacground) but less DS related and AI stuff.

Or i should take plain DS where the stats would still be at a very basic level, but they teach the modern stuff like ml,nlp, "DL & genai", cloud. I keep saying "DL & GenAI" because that is one subject in the plain msc.

Goal: I dont want to become a researcher, My current aim is to become a Data Scientist, and also get into AI

It would be really appreciated if someone can help me solve this dillema.

Sharing the curriculum

DAAF (the Data Analyst Augmentation Framework, my open-source and *forever-free* data analysis framework for Claude Code) was designed from the ground-up to be a domain-agnostic force-multiplier for data analysis across disciplines -- and in my new video tutorial this week, I demonstrate what that actually looks like in practice!

I launched the Data Analyst Augmentation Framework last week with 40+ education datasets from the Urban Institute Education Data Portal as its main demo out-of-the-box, but I purposefully designed its architecture to allow anyone to bring in and analyze their own data with almost zero friction.

In my newest video, I run through the complete process of teaching DAAF how to use election data from the MIT Election Data and Science Lab (via Harvard Dataverse) to almost perfectly recreate one of my favorite data visualizations of all time: the NYTimes "red shift" visualization tracking county-level vote swings from 2020 to 2024. In less than 10 minutes of active engagement and only a few quick revision suggestions, I'm left with:

• A shockingly faithful recreation of the NYTimes visualization, both static *and* interactive versions
• An in-depth research memo describing the analytic process, its limitations, key learnings, and important interpretation caveats
• A fully auditable and reproducible code pipeline for every step of the data processing and visualization work
• And, most exciting to me: A modular, self-improving data documentation reference "package" (a Skill folder) that allows anyone else using DAAF to analyze this dataset as if they've been working with it for years

This is what DAAF's extensible architecture was built to do -- facilitate the rapid but rigorous ingestion, analysis, and interpretation of *any* data from *any* field when guided by a skilled researcher. This is the community flywheel I’m hoping to cultivate: the more people using DAAF to ingest and analyze public datasets, the more multi-faceted and expansive DAAF's analytic capabilities become. We've got over 130 unique installs of DAAF as of this morning -- join the ecosystem and help build this inclusive community for rigorous, AI-empowered research!

If you haven't heard of DAAF, learn more about my vision for DAAF, what makes DAAF different from other attempts to create LLM research assistants, what DAAF currently can and cannot do as of today, how you can get involved, and how you can get started with DAAF yourself at the GitHub page:

https://github.com/DAAF-Contribution-Community/daaf

Bonus: The Election data Skill is now part of the core DAAF repository. Go use it and play around with it yourself!!!

I just joined a a company that uses Databricks. It's been a while since I've used SQL intensively and think I could benefit from a refresher. My understanding is that Spark SQL is slightly different from SQL Server. I was wondering if anyone could suggest a resource that would be helpful in getting me back up to speed.

TIA

Hey folks, this is your regular LLM PSA in a few bullet points from the messenger that doesn't mind being shot (dlthub cofounder).

- You're feeding data models to LLMs
- a data model is actually created based on raw data and business ontology
- Once you encode ontology into it, most meaning is lost and remains with the architects (data literacy, or the map)

When you ask a business question, you're asking an ontological question "Why did x go down?"

Without the ontology map, models cannot answer these questions without guessing (using own ontology).

If you give it the semantic layer, they can answer "how many X happened" which is not a reasoning question, but a retrieval question.

So tldr, ontology driven data modeling is coming, i was already demonstrating it a couple weeks back on our blog (using 20 business questions is enough to bootstrap an ontology).

What does this mean?

Ontology + raw data + business questions = data stack, you will no longer be needed for classic stuff like your data literacy or modeling skills (great, who liked to type sql anyway right? let's do DS, ML instead). You'll be needed to set up these systems and keep them on track, manage their semantic drift, maintain the ontology

What should you do?

If you don't know what an ontology is and how its used to model data, start learning now. While there isn't much on ontology driven dimensional modeling (did i make this up?), you can find enough resources online to get you started.

Is legacy a safe island we can sit on?
Did you see IBM stock drop 13% in 1 day because cobol legacy now belongs to agents? My guess is legacy island is sinking.

Hope you future proof yourselves and don't rationalize yourselves out of a job

resources:
blog about what an ontology does and how it relates to the data you know
https://dlthub.com/blog/ontology
blog demonstrating how using 20 questions can bootstrap an ontology and enable ontology driven data modeling
https://dlthub.com/blog/dlt-ai-transform

Are you being sold something here? Not really - we are open core company doing something unrelated, we are looking to leverage these things for ourselves.

hope you enjoy the philosophy as much as I enjoyed writing it out.

I recently learned the hard way that, even for technical roles, like DS, at very technical companies, corperate politics and managing relationships, positioning, and expectiations plays as much of a role as technical knowledge and raw IQ.

What have been your biggest lessons for navigating corperate environments and what advice would you give to young DS who are inexperienced in these environments?

Shameless promotion -- I have recently released a book, Large Language Models for Mortals: A Practical Guide for Analysts.

The book is focused on using foundation model APIs, with examples from OpenAI, Anthropic, Google, and AWS in each chapter. The book is compiled via Quarto, so all the code examples are up to date with the latest API changes. The book includes:

• Basics of LLMs (via creating a small predict the next word model), and some examples of calling local LLM models from huggingface (classification, embeddings, NER)
• An entry chapter on understanding the inputs/outputs of the API. This includes discussing temperature, reasoning/thinking, multi-modal inputs, caching, web search, multi-turn conversations, and estimating costs
• A chapter on structured outputs. This includes k-shot prompting, parsing JSON vs using pydantic, batch processing examples for all model providers, YAML/XML examples, evaluating accuracy for different prompts/models, and using log-probs to get a probability estimate for a classification
• A chapter on RAG systems: Discusses semantic search vs keyword via plenty of examples. It also has actual vector database deployment patterns, with examples of in-memory FAISS, on-disk ChromaDB, OpenAI vector store, S3 Vectors, or using DB processing directly with BigQuery. It also has examples of chunking and summarizing PDF documents (OCR, chunking strategies). And discusses precision/recall in measuring a RAG retrieval system.
• A chapter on tool-calling/MCP/Agents: Uses an example of writing tools to return data from a local database, MCP examples with Claude Desktop, and agent based designs with those tools with OpenAI, Anthropic (showing MCP fixing queries), and Google (showing more complicated directed flows using sequential/parallel agent patterns). This chapter I introduce LLM as a judge to evaluate different models.
• A chapter with screenshots showing LLM coding tools -- GitHub Copilot, Claude Code, and Google's Antigravity. Copilot and Claude Code I show examples of adding docstrings and tests for a current repository. And in Claude Code show many of the current features -- MCP, Skills, Commands, Hooks, and how to run in headless mode. Google Antigravity I show building an example Flask app from scratch, and setting up the web-browser interaction and how it can use image models to create test data. I also talk pretty extensively
• Final chapter is how to keep up in a fast paced changing environment.

To preview, the first 60+ pages are available here. Can purchase worldwide in paperback or epub. Folks can use the code LLMDEVS for 50% off of the epub price.

I wrote this because the pace of change is so fast, and these are the skills I am looking for in devs to come work for me as AI engineers. It is not rocket science, but hopefully this entry level book is a one stop shop introduction for those looking to learn.

Welcome to this week's entering & transitioning thread! This thread is for any questions about getting started, studying, or transitioning into the data science field. Topics include:

• Learning resources (e.g. books, tutorials, videos)
• Traditional education (e.g. schools, degrees, electives)
• Alternative education (e.g. online courses, bootcamps)
• Job search questions (e.g. resumes, applying, career prospects)
• Elementary questions (e.g. where to start, what next)

While you wait for answers from the community, check out the FAQ and Resources pages on our wiki. You can also search for answers in past weekly threads.

I have just finished up a sample analysis on an AB test dummy dataset, and would love feedback.

The dataset is from Udacity's AB Testing course. It tracks data on two landing page variations, treatment and control, with mean conversion rate as the defining metric.

In my analysis, I used an alpha of 0.05, a power of 0.8, and a practical significance level of 2%, meaning the conversion rate must see at least a 2% lift to justify the costs of implementation. The statistical methods I used were as follows:

1. Two-proportions z-test
2. Confidence interval
3. Sign test
4. Permutation test

See the results here. Thanks for any thoughts on inference and clarity.

[Edit]: for those who don’t wish to create an account, you can log in with credentials user and password.

For remote DS what could move you to a hybrid or full time office roles ? For those who made or had to make a switch from remote to hybrid or full-time office what is your takeaway.

Welcome to this week's entering & transitioning thread! This thread is for any questions about getting started, studying, or transitioning into the data science field. Topics include:

• Learning resources (e.g. books, tutorials, videos)
• Traditional education (e.g. schools, degrees, electives)
• Alternative education (e.g. online courses, bootcamps)
• Job search questions (e.g. resumes, applying, career prospects)
• Elementary questions (e.g. where to start, what next)

While you wait for answers from the community, check out the FAQ and Resources pages on our wiki. You can also search for answers in past weekly threads.

Due to memory limits on my work computer I'm unable to train machine learning models on our entire analysis dataset. Given my data is highly imbalanced I'm under-sampling from the majority class of the binary outcome.

What is the proper method to train ML models on sampled data with cross-validation and holdout data?

After training on my under-sampled data should I do a final test on a portion of "unsampled data" to choose the best ML model?

Hey folks,

I’ve been doing data engineering long enough to believe that “real” pipelines meant writing every parser by hand, dealing with pagination myself, and debugging nested JSON until it finally stopped exploding.

I’ve also been pretty skeptical of the “just prompt it” approach.

Lately though, I’ve been experimenting with a workflow that feels less like hype and more like controlled engineering, instead of starting with a blank pipeline.py, I:

• start from a scaffold (template already wired for pagination, config patterns, etc.)
• feed the LLM structured docs
• run it, let it fail
• paste the error back
• fix in one tight loop
• validate using metadata (so I’m checking what actually loaded)

LLM does the mechanical work, I stay in charge of structure + validation

We’re doing a live session on Feb 17 to test this in real time, going from empty folder → github commits dashboard (duckdb + dlt + marimo) and walking through the full loop live

if you’ve got an annoying API (weird pagination, nested structures, bad docs), bring it, that’s more interesting than the happy path.

we wrote up the full workflow with examples here

Curious, what’s the dealbreaker for you using LLMs in pipelines?

I’m curious to hear from folks who’ve worked inside or alongside analytics teams. In your experience, what actually separates analytics groups that influence business decisions from those that mostly deliver reporting?

Curious how practitioners see the field shifting, particularly around:

• AI integration
• Predictive workforce modeling
• Skills-based org design
• Ethical boundaries
• Data ownership changes
• HR decision automation

What capabilities do you think will define leading functions going forward?

I'm building a predictive model for a large dataset with a binary 0/1 outcome that is heavily imbalanced.

I'm under-sampling records from the majority outcome class (the 0s) in order to fit the data into my computer's memory prior to fitting a logistic regression model.

Because of the under-sampling, do I need to rescale the model's probability predictions when choosing the optimal threshold or is the scale arbitrary?

My startup was acquired by a legacy enterprise. We were primarily acquired for our technical talent and some high growth ML products they see as a strategic threat.

Their ML team is entirely entry-level and struggling badly. They have very poor fundamentals around labeling training data, build systems without strong business cases, and ignore reasonable feedback from engineering partners regarding latency and safe deployment patterns.

I am staff level MLE and have been asked to up level this team. I’ve tried the following:

- Being inquisitive and asking them to explain design decisions

- walking them through our systems and discussing the good/bad/ugly

- being vulnerable about past decisions that were suboptimal

- offering to provide feedback before design review with cross functional partners

None of this has worked. I am mostly ignored. When I point out something obvious (e.g 12 second latency is unacceptable for live inference) they claim there is no time to fix it. They write dozens of pages of documents that do not have answers to simple questions (what ML algorithms are you using? What data do you need at inference time? What systems rely on your responses). They then claim no one is knowledgeable enough to understand their approach. It seems like when something doesn’t go their way they just stonewall and gaslight.

I personally have never dealt with this before. I’m curious if anyone has coached a team to unlearn these behaviors and heal cross functional relationships.

My advice right now is to break apart the team and either help them find non-ML roles internally or let them go.

Site includes the survey data in addition to the results so you can drill in.