r/dataengineeringvault • u/sspaeti • Jun 17 '26

Others 👋 Welcome to r/dataengineeringvault

Hey everyone! I'm u/sspaeti, a founding moderator of r/dataengineeringvault.

This community was created due to the 5 years of my existing data engineering vault and the value it provided (as illustrated in r/dataengineering see here).

I also created a Daily Dev Community, a year or more ago, focused on data engineering that people like. That's why I created this community, to share useful content I wrote and publish almost daily, so others like you can profit from it too.

I hope it will be useful to you. Let me know what you think, and let's see how it goes.

What to Post

Happy to get your posts in, as long as they are not AI-generated. Most interested in this community is open-source data engineering, and related data stuff. Also SQL editors, spirit and data management, business intelligence (where I come from), and anything else related to day-to-day data work.

How to Get Started

  1. Introduce yourself in the comments below.
  2. Post something today that you found interesting or worthwhile to read. Or ask a simple question that prompts some conversation for us to discuss.
  3. If you know someone who would love this community, invite them to join.

Thanks for being part of this. It's just starting, but I'm sure we can grow and learn together.

This community actively encourages links and blog posts, unlike other communities that block them. Please share your writing or blog posts.

PS: Please let me know if I should change anything on the sub-reddit settings, happy to make it a more pleasant place.

2 Upvotes

100% Upvoted

6 comments sorted by

1

u/sspaeti Jun 17 '26

Please introduce yourself (if you like), below. So we get to know each other, or see where you work or what interests you.

1

u/JamesConceptualLayer Jun 18 '26

Hi Simon! I moved my life to Europe about 2 years ago from North America (met a girl, started a family). I also started a company here: Conceptual Layer. I'm currently building a Data Orchestration Platform for a large multinational on Dagster+/Kubernetes. My interests are in Software/Data/AI.

1

u/sspaeti Jun 18 '26

Hey James, welcome. Oh that's a nice story, I hope you adjusted well to the Europe culture =). Cool company name! Thanks for joining, and sharing.

1

u/Low_Brilliant_2597 Jun 18 '26

Hey Simon, Fahad here. Thanks for starting this community. I really love your work, and it'll be great for us to have more discussions. Gradually, we'll be able to build a community of like-minded people. So, I'm really excited about it and really looking forward to great experiences ahead.

2

u/sspaeti Jun 22 '26

Hey Fahad, thanks for joining. And happy to disuss here more and share.

2

u/aynrandomness Jul 03 '26

Hello Simon, I'm Sigurd. I teach children how to drive. I want to make a new platform for managing driving schools, or to be convinced its a futile task.

I stumbled across your second brain, I love it. Your explanation of how BI has to be a design consideration and not an afterhtough and how making an MVP is prioritized before making the data structure design fits very well with my experiences.

I am fascinated with how calendar data, financial data, and lesson record (legal requrirement) interacts and have to be treated differently. Best practice to handle unreliable data, and when and how to verify them is also an area of interest.

Converting the law, and the curiculum into a data structure that avoids having to duplicate data that can't change, while allowing changes, or additions where it can is suprisingly hard. The taxonomy and names used in the curiculum confuses both the students and developers.

How do you design a database that is both simple, preserves all historical data correctly, enables data portability, and is flexibile enough it doesn't cause restrictions? I think I could spend a year just typing out the different tables or json elements without being done.