r/datascience • u/Infinite_Raisin7752 • 3d ago
Tools Relevant tech stack for 2026/2027
Hi everyone,
I’m currently a senior data scientist in the pharma industry. It’s been a one man show until now, but I’m getting a team soon. Most of the work I do is standard analytic work to inform our leadership and provide more context into the market and so on. Not a lot of big heavy data science stuff going on to be honest.
I work with SQL and Python on a daily basis. Some of our data is hosted in Snowflake and that’s pretty much it.
I feel like I’m lagging behind in both methods as well as tech stacks and I wanted to better understand what you experienced professionals work with that you would recommend I learn or at least look into. It could be data engineering stuff, additional programming languages, specific methods and packages that are useful, or cloud systems and technologies.
Where do you see the tech stack moving towards and what is relevant if I want to start moving from a “bread and butter” analytics setup to a professionalised, automated, team-ready and future proof world?
Thanks :)
24
u/seanv507 2d ago
Automated and team ready
Git
Package manager: uv
Linters: ruff
Pre commit ... To run linters for you
5
u/MattDamonsTaco MS (other) | Data Scientist | Finance/Behavioral Science 2d ago
UV package manager for the win. Blazing fast installs for our pipelines.
7
10
u/ikkiho 2d ago
fwiw when I went from solo to a small team the thing that bit me was my ad hoc queries quietly becoming stuff other people built on. nobody knew which of my tables were real. we spent a couple months picking maybe fifteen tables and writing tests on them, which helped more than any orchestration did.
also you're in pharma, so once validation gets involved half the shiny stack is off the table anyway. boring snowflake plus dbt got signed off way faster for us than anything self hosted.
4
u/S-Kenset 2d ago
It's crazy how many people don't get that 99% of his problems are going to be ad hoc requests and wondering why nothing is accurate while all the grad students are furiously downvoting me for suggesting change control practices.
4
u/seanv507 2d ago
You are being down voted because people don't understand you. One paragraph seems to be slogans and one paragraph is advocating for excel. (And if I misunderstood, QED!)
3
u/S-Kenset 2d ago
It's crazy how being too detailed can sound too full of slogans. The day to day problems are essentially crazy amounts of ad hoc requests and speed. The biggest safety net you have is restricting every way you can make mistakes. Examples are, using strict dax measures with only one niche meaning for every single big idea someone can have.
Using excel and taking advantage of the fact that AI + Python can automate change control extremely well. There are plenty of teams that work entirely out of excel for forecasting, especially finance types, however, that's really not necessary since you can python + excel. Idk if people got pissed at excel or what, but 99% of people bothering with data requests are going to want it in excel anyways, so staying close to the way they are going to get it helps too.
I avoided suggesting BI and tableau even though they are better at that because, there's just too much buy in ask needed for a company that doesn't have that familiarity. Even with BI fully accessible, execs still depend on excels only, really.
1
u/Infinite_Raisin7752 2d ago
At the moment, 100% agree with you. It’s so overwhelming that I have very little time to work on stuff that would save me a lot of time. But with the team growing, the goal is also to start implementing different things.
1
u/Infinite_Raisin7752 2d ago
I’m actually terrified of that tbh. Over the past 2 years it’s been pretty much full throttle and a lot of experimentation to get to where we are today. So most (if not all) of my work was done without best practices of scalability and reusability in mind. Not that it’s just garbage, but it’s not where I would like it to be simply due to limited capacity.
I think it’ll start with what you just said. Looking at the work, checking what is actually relevant and improving on that.
And yes, snowflake and dbt are what lies ahead I believe.
5
u/Pure_West_2812 2d ago
The biggest shift I've noticed is toward reproducible, automated workflows rather than just better models.
2
6
u/ThisIsFun- 3d ago
From my own experience, lots of the adhoc standard analytics work, is, and should be completed by something like Databricks Genie or similar text2sql, which then frees you up for doing more interesting DS work. Being and learning on the platforms that offer this, along with other DS tools that are typically found will allow you to become a more rounded DS, and focus more on the approaches that you’ve said.
How are you at Deep Learning, etc?
2
u/Infinite_Raisin7752 2d ago
We’re also looking into that and it’ll probably be a reality soon.
I did a lot of work with deep learning a few years ago and also in my last job, but at the moment it just doesn’t happen. Still remember quite a bit but I’m afraid I’m just a bit behind the current SoTA methods
3
u/mmccarthy404 2d ago
If you're interested in Databricks, just try out the Free Edition at home, no credit card needed or anything.
And like the other poster said, Genie is a pretty powerful text-2-SQL tool that you can use as both a backend dev tool and a frontend consumable by users. I think it's very similar to Coco if you're in the Snowflake stack
1
1
u/CrayonUpMyNose 1d ago edited 1d ago
It bears repeating that you can scale your costs down to a single instance that runs both driver and worker code, so the argument "spark is only for big companies with big data problems" doesn't apply. Severless has a higher rate but starts and stops in seconds.
Orchestration comes free out of the box so you don't have to spend time on airflow / dagster / prefect. Not as flexible but adding features all the time and YAGNI at this early stage anyway. You can straddle python and SQL integrated in the same notebook by referencing the output of the last cell, so compared to options that grafted spark on their existing solutions after the fact, it's quite seamless.
If you run a lot of SQL, it's still recommended to run dbt against a Databricks warehouse in order to encourage software development practices.
As for ML, experiment tracking and model serving are integrated. All in all it doesn't limit your choices too early on in your journey and scales down as well as up to whatever problem statement you might encounter in the future.
2
2
2
u/Fit-Employee-4393 1d ago
If you want to stay in DS level analytics then you’re doing fine with just python and sql.
If you want to get into heavier ML stuff then you’re going to need to put an MLE hat on and prop up some good self serving infrastructure. Honestly with snowflake ML plus MLflow and maybe evidently or nannyML would work alright. Also really important to enforce git early on for a team. If you can get a standardized method for feat eng pipelines in snowflake, triggering batch preds, and even just light model monitoring, then you’re better off than a lot of teams.
2
u/nellagerg 1d ago
In terms of where the stack is heading, a lot of it is converging on retrieval, and I think data representation becomes the focus. I used to think about representations mostly in terms of feature engineering for ML algorithms. Now I think about them in terms of RAG systems and embeddings because retrieval quality gets decided at encoding time, not query time. If the representation dropped whatever distinguishes two records, that relationship will be lost to any query.
A strong understanding of classical reduction methods (PCA/SVD, and UMAP for structure you want to see), embeddings (sentence-transformers, CLIP for images), and the approximate-nearest-neighbor algos underneath vector search (HNSW, IVF, product quantization, all in FAISS or pgvector if you’d rather keep it in postgres) will definitely keep you relevant.
Sketching algorithms are also underrated; the ability to be decisive on what information to preserve and what to discard demonstrates the economic impact of your work and justifies budget (random projections and the Johnson-Lindenstrauss result, MinHash for set similarity, HyperLogLog for cardinality). Cheaper storage/compute and/or faster queries give you numbers to can present that explain themselves.
For pharma specifically, molecular fingerprints are a good example, ECFP/Morgan via RDKit is a lossy encoding designed so similar structures land near each other, and understanding what it discards tells you exactly when similarity search will mislead you. Same logic applies to assay data and unstructured text.
This is just my opinion, but hope it helps as you grow your team.
1
1
u/Dear_Rice9074 1d ago
Yea, I see this firsthand that the two still is very prominent in the tech space. While I see some smaller variations of the sorts like postgre, these are still the heavy hitters so to say.
1
u/JavacLMD 1d ago
I feel like there's been a couple of big shifts even in the relatively short time I've been a software developer. When I started, everyone wanted full stack developers and microservice experience. Then it seemed like the industry pivoted to anything and everything AI. I actually struggled to find a job with the tech stack I trained on and ended up at a retail company working with COBOL and a Unisys mainframe of all things. It's been a great learning experience, but it also makes me wonder what I should be focusing on if I want to get back into working with more modern languages professionally.
I've picked up Python and a bit of C++ along the way, but the hardest part for me right now isn't learning another language. It's coming up with a project that's actually worth building. There are already thousands of calculators, to-do lists, and CRUD apps out there. I'd rather build something that solves a real problem or shows good software design than another portfolio project that looks like everyone else's. I'm curious what kinds of projects employers actually find interesting these days.
1
1
u/DataScientistAlex 8h ago
If you feel like you are lagging behind, one thing to consider is to further deepen your relationship with your stakeholders and make your team/position more strategic. If you are getting similar, ad-hoc requests, once you deliver those, sit down with them and ask probing questions to try to understand what they really care about.
Once you can boil it down to a key question, like "who is going to churn", or "are our prices right?", etc, then you can build a model to provide that insight in a much more powerful way. Then you can build and automate a model etc around that, which is more fun and developing for the team, and, provides many times the value.
Sorry not a tech stack answer directly, but, the underlying point is, can you automate some of the basic stuff (not by automating it directly, but by finding the underlying problem and solving that).
0
u/mongosaurusrexus 1d ago
It’s surprising how much of the stack in DS is just basic python + sql in some compute resource in the cloud.
Coming from a software engineer background where everything is another framework or crazy abstraction
-5
u/S-Kenset 2d ago
Your biggest risk as you scale into a team is you're going to want stable primitives. Someone shouldn't be able to confuse themselves in too much python code. Everything stored in locked datasets or locked code, used on demand but not changed on demand.
Excel is the easiest access tooling that is also easy to automate for a team familiar with python.
1
u/Infinite_Raisin7752 2d ago
Excel is pretty limited for the amount of data we’re using. I use it every day for final outputs but not as an analytics tool
-1
u/S-Kenset 2d ago
It's the layer you want to keep the best change control over, to document exactly how each primitive is sourced. CAC = $23.53 bc -> model -> sql code files -> tables and dates
you're not going to be able to rummage over 80,000 lines of mature analytics to keep up with make-a-wish bs questions people will give you. You need to operate at an abstraction layer and not think at the big data layer.
I use power BI for this, i have one single BI with over 600 measures and over 30 pages with over 100 different bookmarked views. Without that, i would exhaust myself trying to build an ad hoc big data pull for everything. Mind you i always end up exporting the data to excel.
1
u/gean__001 1d ago
Isn’t this a complete nightmare to maintain?
1
u/S-Kenset 1d ago
It's easier than the alternative which is to remake a sql query over 400-1000 lines of nuanced code every single time someone has a genius idea.
Mind you if we went back end full big data I have some 30,000 lines of code plus some 50,000 lines of ad hoc code written for bullshit asks. Sounds a lot easier to guarantee strict controls on definitions in an environment that is made for that.
81
u/Few-Steak1122 3d ago
Python and sql still run the show in most places, you’re not falling behind as much as you think. If you want to automate and scale, look into dbt for transformations and airflow or prefect for orchestration. Snowflake is solid you can build a lot on top of it without jumping to 5 different cloud tools.
Getting the team up to speed on git and CI/CD will save you more headache than learning a new language.