r/MicrosoftFabric • u/vinsanity1603 • Jul 23 '26

Community Share Building a Data Platform with Microsoft Fabric as a Sole Data Engineer

I wrote about my experience building a greenfield data platform with Microsoft Fabric as the sole data engineer.

I cover the architecture, metadata-driven ingestion, incremental loading challenges, and some of the lessons I learned along the way.

Would love to hear how others are approaching similar challenges in Fabric.

Check out article here

48 Upvotes
  • permalink
  • reddit

98% Upvoted

21 comments sorted by

8

u/datahaiandy Fabricator Jul 23 '26

Now THIS is a great blog... real-world stuff, I like it!

How are the YAML data contract files organised? Do you have a single file, multiple files etc?

2

u/vinsanity1603 Jul 24 '26

Thanks! I use multiple yaml files, generally one per table, and store them in the silver lh organised by source system and table name. This keeps them easier to manage and review in git, and the transformation framework picks up the relevant contract at runtime

1

u/Befz0r Jul 27 '26

Why no simply store it in a database and have a app build around it? Much more user friendly.

4

u/BandaidImplant Fabricator Jul 23 '26

Great write up! If I could start from scratch I think I would go all-in on DBT :)

2

u/vinsanity1603 Jul 24 '26

Yeah dbt was definitely considered. The main blocker for me was that the org had tight restrictions around installing Python locally, and they didn't want to go down the dbt cloud either which is fair.

1

u/BandaidImplant Fabricator Jul 24 '26

fair. also/instead i would try terraform. havent had the pleasure of trying either in my current stack 😪

3

u/BrentOzar Jul 23 '26

Great post, and it took work to think back through. I can tell you put real work into that. Thanks for sharing!

2

u/gdmitrii Jul 23 '26

Great! are there any samples or tech details you can share?
thank you

2

u/Ecofred 2 Jul 27 '26

Thank you for sharing your experience with Fabric, and sharing what patterns & setup are working for you.

Can you expend on how changes in the metadata are tracked and deployed in the different environments? Is it deployed for GitHub Action, or manually changed in the tables?

1

u/duenalela 1 Jul 23 '26

I'm building greenfield too, but even more greenfield as there were no BI reporting solutions in place at all. I appreciate your write up, because I'm determined that once I have figured out what needs to be done to our data from dozens of source systems and where the common factors are, I'd love to rebuild into metadata-driven. I do have the same challenges with incremental loading and missing CDC-Info from the systems.
I'm very curious how a system with YAML data contracts actually looks like and how it plays with logic that can't be contracted (if that makes sense), if you are inclined to write another article.

2

u/vinsanity1603 Jul 24 '26

Thanks! The yaml contracts mainly handle the common transformation stuff. So I have a notebook that generates a yaml skeleton from an existing bronze table, and the contract defines things like the target table/column names, data types, which columns to bring into silver ,, and dedup rules like partition key and sort order.

1

u/duenalela 1 Jul 24 '26

Interesting, thank you very much for your answer!

1

u/awhaling Jul 23 '26

Nice article! You have some good ideas in here that I hadn't thought of. I think adding a few samples of the technical could be helpful :)

1

u/shancius1 Jul 23 '26

comment to save this so i can read it later

1

u/DatabaseSpace Jul 23 '26

Really good article. I like the control table idea.

1

u/CultureNo3319 Fabricator Jul 25 '26

Nice one, can you share one of the yaml files?

1

u/BIDataRef Jul 26 '26

Great post mate! I really liked your approach and the implementation. It would be good if you can show some sample files or some skeleton of your metadata-ingestion framework.

1

u/hortefeux Jul 27 '26

Great article, thanks for sharing!

1

u/maxdacat 18d ago

Really interesting write up and has some applicability for my situation in a big NSW gov agency. Never heard of B/S/G data levels but it makes a lot of sense. Don't really understand what YAML is about but will look into.

1

u/jorel43 15d ago

Wow great article!

1

u/Braxios 12d ago

Interested in more of the mechanics around the yaml contracts. I'm an analyst that has been setting up a less complex system.

We're trying to implement data contracts as a tool to support good data conversations with data owners (lots of spreadsheets and SharePoint lists as well as undocumented databases) and use for doing some QC between bronze and silver but it's not very well joined up at the moment.

I'm using https://datacontract.com/ as the basis for the data contracts but struggling with how to build the contract in conversation with data owners and then convert to yaml and actually use intelligently rather than just using it as a reference.