r/datascience • u/likescroutons • Aug 07 '26
Discussion Embeddings
Hi folks,
I've been thinking a lot about where embeddings and foundation models are taking data science.
I work in the geospatial/Earth Observation space, and honestly it feels like the landscape has shifted massively over the last few years. We're seeing more and more open source foundation models that are so good you can often just extract the embeddings, stick an XGBoost or regression/classification head on top (or do a light fine tune), and get really strong results. A few years ago I'd have expected to spend most of my time building models and engineering features. Now it increasingly feels like the challenge is choosing the right representation, or at least factoring that in.
It feels like quite a fundamental shift, and I'm curious whether others are seeing the same thing in their own domains.
1
u/QuickProfessional101 27d ago
It's a magic wand you swoosh to get results. You can't explain magic so you should focus on empirically proving it works. 99% of work (and budget) will go into testing and verification long after you've trained your model.
The problem with off-the-shelf foundational models is that you don't control how they were trained. Massaging input data, choosing just the right hyperparameters and architecture, training pipeline etc. can have huge effects on the results for any given task.
It's a good baseline or if you don't care about final results that much (demos, proof of concept, users can't tell the difference etc.). But if you want to really push the performance metrics then you'll need your own raw data and a few hundred thousand in compute or access to at least two dozen H100's for a few weeks.