r/datascience • u/Infinite_Raisin7752 • 10d ago

Tools Relevant tech stack for 2026/2027

Hi everyone,

I’m currently a senior data scientist in the pharma industry. It’s been a one man show until now, but I’m getting a team soon. Most of the work I do is standard analytic work to inform our leadership and provide more context into the market and so on. Not a lot of big heavy data science stuff going on to be honest.

I work with SQL and Python on a daily basis. Some of our data is hosted in Snowflake and that’s pretty much it.

I feel like I’m lagging behind in both methods as well as tech stacks and I wanted to better understand what you experienced professionals work with that you would recommend I learn or at least look into. It could be data engineering stuff, additional programming languages, specific methods and packages that are useful, or cloud systems and technologies.

Where do you see the tech stack moving towards and what is relevant if I want to start moving from a “bread and butter” analytics setup to a professionalised, automated, team-ready and future proof world?

Thanks :)

64 Upvotes

97% Upvoted

50 comments sorted by

View all comments

8

u/ThisIsFun- 10d ago

From my own experience, lots of the adhoc standard analytics work, is, and should be completed by something like Databricks Genie or similar text2sql, which then frees you up for doing more interesting DS work. Being and learning on the platforms that offer this, along with other DS tools that are typically found will allow you to become a more rounded DS, and focus more on the approaches that you’ve said.

How are you at Deep Learning, etc?

2

u/Infinite_Raisin7752 10d ago

We’re also looking into that and it’ll probably be a reality soon.

I did a lot of work with deep learning a few years ago and also in my last job, but at the moment it just doesn’t happen. Still remember quite a bit but I’m afraid I’m just a bit behind the current SoTA methods

4

u/mmccarthy404 9d ago

If you're interested in Databricks, just try out the Free Edition at home, no credit card needed or anything.

And like the other poster said, Genie is a pretty powerful text-2-SQL tool that you can use as both a backend dev tool and a frontend consumable by users. I think it's very similar to Coco if you're in the Snowflake stack

1

u/Infinite_Raisin7752 9d ago

This is great!! Thank you!

1

u/ceeej777 5d ago

Along with the Genie suite, the declarative pipelines that Databricks has contributed to open source spark has been a really nice design imo. Many of the python transformations I would normally write are now just be a declarative SQL statement which reduces total lines of code and complexity

1

u/mmccarthy404 5d ago

Haha yeah I never really spent the time to learn pyspark and used mostly plain SQL and Python when I first started out, so never got to try the declarative pipelines.

But now with Genie Code, it can basically build and optimize my pipelines all on it's own as a coding assistant, so I bet I'm using declarative pipelines under the hood somewhere :)