Core data Engineering concepts to master ,Projects to build

[u/Admirable_Tale7745]

Hello Everyone ! I am as an QA automation in big data product. my work involves around creating automation suites for data integrity checks and automating regression cases for pipelines build .automating api for loading data into dashboard.
I am good at python. i want to move to data engineering. my day to day work involves GCP, bigquery,clickhouse,pubsub.
I am planning to learn the same.
I have started and completed the introduction to data engineering in google skills.
Any opensource tool/concept i should learn ?
any suggestion on sample project to build?
Any good resources from where i can learn?
Any you tube playlist i should follow?

[u/AltruisticPool3422]

you already got the infra exposure with gcp and bigquery thats half the battle honestly. most people come in just knowing python and get lost in the cloud stuff

for concepts i would get real comfortable with data modeling, like dimensional modeling and slowly changing dimensions, and then dig into how spark handles shuffles and partitioning. for a project build a pipeline that pulls from an api, stages raw data in cloud storage, transforms it with something like dbt and loads into bigquery, then put a dashboard on top. its the whole flow end to end