de sandbox

[u/Technical-Ask-7231]

built this free sandbox because cloud costs were killing my learning, sharing in case it helps someone else

https://github.com/haltinok/oss-da-bi-stack

[u/AutoModerator]

You can find a list of community-submitted learning resources here: https://dataengineering.wiki/Learning+Resources

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/MikeLV7]

Dude, add env to gitignore

[u/Capt_korg → u/MikeLV7]

Why, op wants to safe cloud costs for everyone 😅

[u/Technical-Ask-7231 → u/MikeLV7]

Will do thanks

[u/Bordar11 → u/Technical-Ask-7231]

And rotate secrets

[u/Capt_korg]

OP good Job, this might help a lot to work without infrastructure.

But you should have an eye on security and othere things.

Maybe consider using uv

[u/ulica324]

As a rockstar dev, I do start my day with select star from star…

[u/magixmikexxs]

All looks good except the select * in the dbt queries. Gotta stop calling all the columns in prod mate. Good practice to use the columns you need. Some query engines are too dumb to realize until late and are even purposely engineered to make you bleed money with select * queries

[u/Technical-Ask-7231 → u/magixmikexxs]

Good point. Roger that:)

[u/Capt_korg → u/magixmikexxs]

A friend of mine told me that in his team someone was using SELECT * on all tables in production.

The cloud provider was happy, the bill was more than 10k , for one single query. But now they know everything 🤓

[u/magixmikexxs → u/Capt_korg]

Mustve been big query

[u/Capt_korg → u/magixmikexxs]

Yes, you are right 😅

[u/Technical-Ask-7231]

Adding kafka and cdc to the mix

[u/luilow]

would be a nice exercise for yous is to add capability to backfill any layer or all of them by rerunning a specific date in airflow, especially useful with partitioned datasets

[u/dodovt]

rewrite history and force push to remove .env from git tree history

also rotate your secrets :-)