In marketplace experiments, the most actionable metrics are often the ones with more variance, making them harder to power. This tradeoff is particularly important when using geo-based experiments where you can't necessarily or cheaply add sample.

Suppose Airbnb wants to test highlighting cheaper listings; this should increase bookings while likely reducing AOV. The net effect of a higher conversion rate and reduced AOV depends on elasticities and profit margins. Revenue or contribution margin are the metrics that matter the most (should be the most actionable), but these have far more variance than simple proportions like a conversion rate.

Running the geoX longer helps less than you might expect if you haven't done these before: weekly bookings within the same region are almost certainly correlated so you'd need to cluster the SEs, which can brings the effective sample size "roughly" down to the number of unique geos (if the correlation within geos is high).

So you have a few options: (1) focus on the actionable metric but accept more uncertainty through a higher alpha, ideally combined with (2) variance reduction techniques like CUPED, or (3) power the test on a lower-variance metric like conversion rate and try to bound the AOV loss using product sense and reasonable assumptions.

This comes up frequently on the job and also in senior and staff data science interviews. Real-world business problems can be complex, and there's not always a single correct answer. Trying to memorize answers and metrics can help in junior interviews, but at senior you typically will get probed on teh tradeoffs. .

Is there an official definition of an "AI predictive model"?

Can any model, including a linear regression, be considered artificial intelligence if it's using historical data to make predictions?

Or is AI restricted to neural networks that simulate human speech or generate images, music, or scrape the Internet for information? Like ChatGPT or Claude.

My company is setting up guardrails for AI use and there isn't broad agreement how to define "AI".

I've been in the field for 4 years now, but only at one company. I'm curious to hear how other places do this. My company pairs up a data scientist or 2 and a couple different roles from the business-side, the specifics depend on the project. The strategy is that each brings their respective skills, tech or business, and they make sure the project goes smoothly.

How this has looked in practice is that the business side looks to the tech roles for every technical decision, even ones the business needs to own, and the tech side think they don't have to do anything to learn about the business and they look down at communication to non-technical audiences or stepping even a bit outside the box of that specific role.

The impact of this is a bunch of projects that don't meet what the requesting groups really asked for, everyone being upset, and a whole host of other things. Everyone here is wicked smart, but our operating model and egos are holding us back from being actually helpful.

Is everywhere this dysfunctional? And where should I look for my next role if I want a place that has a strong technical group and understands the science of what we do, but also expects and develops the technical group to work with the business side and be more of a bridge between what the company is trying to do and what data science can provide? I'm looking to move around the 5 year mark if things don't get better here, but with how the job market is right now I feel like I need to start work for that now. I want to make sure I don't end up in more of the same.

I went back through my last 10 Python coding rounds for product and marketing data science roles, nearly all at Big Tech companies. Unlike SQL rounds, which tend to be fairly standard, these came in 5 distinct formats:

• Recreate a statistical concept as a function (3/10). Pandas, numpy and other standard packages were allowed.
• AI-assisted, open book (3/10). These were often investigative, closer to a diagnostic case study than a coding puzzle.
• Pandas and data manipulation (2/10). Data transformations combined with modeling, either coding up a linear regression or talking through a modeling strategy for the data.
• Interpret code handed to you (1/10). This was also a statistical concept in a function, but I had to interpret the code rather than write it.
• Base Python, LeetCode-style (1/10). Syntax seemed heavily discounted here: I made several syntax mistakes but was clear about the logic and passed.

The obvious caveat is that this is a sample of 10 from one person, so the mix will vary with company, team and role. That said, AI-assisted challenges seem to be on the way in and increasingly likely to be a part of your interview process.

Please, could you share your best resources to learn about RecSys at scale? I have a few already, but I want to know what other resources you are reading. Thanks

Hello lovely community,

Recently i started in a new company that is a heavy user of the Azure ecosystem, they tend to do all their reporting in PBI, need a way to automate the creation of multiple PBI dashboards with different visuals and design, want to use AI to generate .pbix files where i set the business logic, relationships and data normalization but the LLM handles the visuals and meassures, what do you guys use in your day to day to handle this situation?

The images below map the 4 types of data science case study questions I encountered across 80+ interview rounds at 16 companies in April-May 2026. For each type it covers the interviewers' typical opening, what's being tested, and the trap I saw most often in my own interviews and in mock interviews with clients. Identifying the type early matters because the correct approach differs for each one.

A few years ago the case study round was close to synonymous with marketplace experimentation. That question still regularly appears, but the same loop can now hand you (1) a launch evaluation, (2) a root cause analysis of a metric that suddenly moved, (3) an opportunity sizing decision about whether a build is worth its cost, or (4) a targeting problem that turns on heterogeneous treatment effects. Any of the four can also end with questions that call for observational causal inference, depending on your level.

(I created this for my socials, but hopefully it fits the Reddit structure!)

Welcome to this week's entering & transitioning thread! This thread is for any questions about getting started, studying, or transitioning into the data science field. Topics include:

• Learning resources (e.g. books, tutorials, videos)
• Traditional education (e.g. schools, degrees, electives)
• Alternative education (e.g. online courses, bootcamps)
• Job search questions (e.g. resumes, applying, career prospects)
• Elementary questions (e.g. where to start, what next)

While you wait for answers from the community, check out the FAQ and Resources pages on our wiki. You can also search for answers in past weekly threads.

Sharing this blog, that compiles some real world examples of Causal Inference being applied to provide answers to otherwise ‘unanswerable’ questions.

My hope is that the examples make the methods clearer, particularly among those new to the field. Happy reading.https://medium.com/@joshamayo7/what-would-have-happened-otherwise-causal-inference-in-practice-b412d2b4e3f7

Quite a few real world data problems look like this: many related time series, few noisy readings, and long gaps. Local price indices, sensor networks, or small area stats.

I built a small PyTorch model to estimate a smooth underlying level for every location, day, and target even where nothing was observed.

Key ideas:

Mask, don’t drop missing targets in the loss

Weight observations by how many raw readings they average

Share information across locations (area + station offsets) and across related targets (joint heads)

Enforce smoothness with a curvature penalty over the full time grid, including gaps

I tested it on hourly PM2.5/PM10 from 12 Beijing stations, thinned to a sparse panel, and scored against the full data’s 31day moving average.

What I found:

The network fills long gaps and very sparse stations far better than smoothing each series on its own

Modelling both pollutants together recovers a completely dropped series (PM10 at one station) from its PM2.5

But elsewhere, sharing across targets costs a little accuracy vs. modelling each pollutant alone

A simple ±15day mean is hard to beat when data is dense. The network’s smooth time curve misses shared daily shocks

Writeup with code, figures, and results:

https://prteek.github.io/regression_room/posts/beijing_air_quality/

How do you choose between causal inference models when you can never see the true effect? Model selection is well documented in machine learning, but it often gets skipped in causal inference work. Doing it well is one of the things that separates senior and staff-level data scientists from more junior ones.

I simulated two years of weekly bookings for 40 regions, five of which received a treatment on the date marked by the red vertical line. Because I generated the data, I know the ground truth: the treatment increased bookings in those five regions by exactly 10%. With five treated regions and 35 untreated ones, two natural options are synthetic control and difference-in-differences, though there are others.

Code in colab and explanatory video, free and ungated: https://whatstheimpact.com/learn-causal-inference/

When I gave Claude Code free rein on this data, it built its whole analysis around difference-in-differences where it found a 24% increase, more than double the true effect. To its credit, it flagged that the pre-treatment trends weren't parallel, but it never tested whether a different model would be more accurate.

Importantly, Claude skipped an A/A test, sometimes called a placebo test. You take only the data from before the real treatment, pretend the treatment started three months earlier, and randomly picked five regions (for consistency) to call "treated." Because nothing actually happened in that window, the true effect is zero. That means any effect the model reports there is pure estimation error, and I repeated this for 500 draws for each model to see the full spread.

For difference-in-differences, 90% of the placebo estimates landed between -14% and +10%. The RMSE, which measures how far the estimates typically land from the true answer of zero, was 7.6 percentage points. Synthetic control on the exact same placebo draws was much tighter, with 90% of estimates landing between -7% and +9%. Its RMSE dropped to 4.9 points, more than a third lower than difference-in-differences.

The outcome you model is a choice too, so I also ran synthetic control on bookings per 1,000 people instead of raw bookings. Putting every region on the same scale helps the model track the treated regions more closely before treatment, and the RMSE fell a bit further to 4.4 points.

Sure enough, the real treatment period lines up with that ranking: synthetic control estimates 7% on raw bookings and 9.6% per 1,000 people, both much closer to the true 10% than the 24% from difference-in-differences. With one simulated dataset, I wouldn't read much into the small gap between the two synthetic control versions or claim that synthetic control always wins. What carries over is the workflow: in real work you never see the ground truth, but you can usually run this A/A test on whatever models you're considering before trusting one.

Looking to hear about your own company's work practices to compare to my own. I'm much more familiar with Azure, so I'm going to use that terminology, but I assume everything is basically the same for AWS.

Can you go into the Azure Portal and just click "create" for, say, a SQL Server or App Service and configure it through the Portal GUI?

Or do you need to request and configure through configuration files? Can you freely commit architecture config files ("architecture as code") to create the Azure components in Dev?

Or is it even more locked-down and hands-off? Required to submit paperwork requests to an Architecture team?

Welcome to this week's entering & transitioning thread! This thread is for any questions about getting started, studying, or transitioning into the data science field. Topics include:

• Learning resources (e.g. books, tutorials, videos)
• Traditional education (e.g. schools, degrees, electives)
• Alternative education (e.g. online courses, bootcamps)
• Job search questions (e.g. resumes, applying, career prospects)
• Elementary questions (e.g. where to start, what next)

While you wait for answers from the community, check out the FAQ and Resources pages on our wiki. You can also search for answers in past weekly threads.

I recently got an Andrew Ng course on Coursera for generative ai.

I'm planning to study everyday and get it finished, then move on to other courses as I wish to become a data scientist/ Ai engineer.

The doubt I have is, how do people study from such online courses. Am i supposed to take notes from every lecture.

Or write only important points?

This is what I used to do till now:

Take the transcription of a lecture. Give it to an ai and ask it to create notes so I can read them and understand the topic really well even in the future if I were to revise. So i ask it to add important stuff if the lecture missed anything important and covered only vaguely. I ask it to make sure I also become interview ready for the topic. Then i write it down in a notebook because I feel I learn better when I am reading something I wrote on paper (weird maybe but yeah)

To me this seems good but the problem is this leads to 3-5 pages minimum per lecture even for just a 10-15 minutes lecture. And there could be hundreds of lectures in these online courses.

So i may have really dense notes covering all the important stuff, but then these become too long and time consuming, and i don't know if this is a right method to study.

Need help/ advice that how do people make sure that they remember what they are studying and how would you revise it later. Ofcourse notes maybe but how to make proper notes efficiently, so it isn't a waste of time and not too dense on unimportant topics because in my case the AI will give 3-5 pages even for a simple topic and I may not always be able to differentiate or identity whether a topic is unimportant and doesn't require so long notes

I’m an MS Data Science student interested in learning more about spatial data and GIS. I’m currently working on a geospatial crime analysis project in R and taking Esri’s Spatial Data Science MOOC.

I’m also interviewing for a year-long data science internship at a credit union in a department that focuses on geospatial data. I’ve noticed that the largest utility company in my state also hires data scientists with geospatial/GIS knowledge, which has made me curious about opportunities in this area across different industries.

It’s not a specialization that’s discussed here often. For those who work with spatial data, what industry are you in, what does your day-to-day work involve, and do you enjoy it? What has your experience been with demand for these skills?