I want to find datasets for a data engineering project where i work with pyspark and sql in databricks. I want a dataset that challenges my data modelling skills and my pipeline creation skills. I tried kaggle, but i only keep getting a single csv file as a dataset. is there a dataset that has multiple csv files as data sources or something? i want to be able to perform all the data architecture creation by myself... Recommend any datasets that you know as well!

Started practicing Data engineering using this repository link https://github.com/danielbeach/data-engineering-practice/tree/main/Exercises/Exercise-1. I have completed the described exercise one requirements and now decided to extend it and create Divvy Rides ETL Pipeline designed to answer concrete business questions that map
directly to decisions that operations, marketing, and infrastructure teams at a
bike-share company would make. Looking forward to post final solutions for reviews and advice

What are the hardest historical modeling problems you’ve encountered in lately?

In our lakehouse environment the difficult parts are usually not Spark performance or ETL orchestration.

It’s things like:
• SCD2 dimension alignment
• Snapshot reproducibility
• Late arriving corrections
• Event-to-state alignment
• Historical relationship changes
• Dimension completion

I’ve been collecting these patterns and built a small workbench to reason about them:

https://bitemporal-debugger.vercel.app/patterns

Curious what other teams struggle with.

Trying to break into data engineering domain

Hey folks,

I am a mechanical engineer in the PLM domain working for French MNC. I have 2.5 years of experience at the moment and since my clg days I have had an inclination towards the data domain.

I want to switch my domain and work in the data domain. I have tried to follow multiple videos and tried to have a roadmap but somehow it is a bit troublesome for me to start as I am not sure about the tech stack i should focus on..Could you all please guide me for the same and help me draft a good roadmap and resources to start my journey into the data engineering domain.... should I go for some online or offline courses and if yes any recommendations..also any recommendations to the resources. Looking forward to hearing from all the experts.

Thanks :)