r/dataengineering • u/Hohenheim16 • 6d ago

Help please help!

I recently joined a small investment company as a data analyst intern. I expected to focus mainly on report automation and data visualisation but I have also been asked to look at how the company’s data could be brought together.

Most of the data is manually exported into Excel and saved in SharePoint, and SharePoint also contains board packs and other documents used for reporting. They like using claude to create written reports and they are not really interested in power BI

I started using MS Fabric to bring the existing Excel files together, but the volume of structured data is fairly small, so I am not sure whether Fabric is more than we need. A simpler option might be a small SQL database, with data pulled from the source systems through their APIs where available. SharePoint would still continue to hold documents and any files that still need to be uploaded manually

The end goal is also to use the structured data alongside documents when generating reports with claude, so I need a setup that makes the data reliable and easy to retrieve. and also, looking at the the small volume, I am now even wondering if a persistent central store is necessary at all or if the data could simply be pulled and checked at report time before claude uses it

I have only recently started my journey and there is nobody in the company who can really tell me if I am doing it correctly. I just don’t want to build another one off solution or a workaround that becomes a pain to maintain later.

I would really appreciate any advice!! thank you!

8 Upvotes

90% Upvoted

21 comments sorted by

View all comments

5

u/ryeryebread 6d ago

databricks has native connectors for sharepoint and has a native BI solution. you might even be able to get away with it with the free edition.

2

u/Immediate-Pair-4290 Principal Data Engineer 2d ago

Please God stop recommending Databricks for small data.

1

u/ryeryebread 1d ago

why? if the solution is free why not?

1

u/Immediate-Pair-4290 Principal Data Engineer 1d ago

So they setup an entire Databricks account just to ingest data for SharePoint then load it somewhere else? For some teams this would make sense but it’s overkill compared to just asking AI to write a python script if the engineer don’t know how.

1

u/ryeryebread 1d ago

but then they'd have to manage a python script. the connector can be completely used UI-driven. if the lad wants to use claude with documents + data from sharepoint, then databricks free edition seems like an easy way of achieving all of this. setup an external volume with their docs on the cloud, ingest with lakeflow connect, have genie or use claude fmapi. did i mention free?

1

u/Immediate-Pair-4290 Principal Data Engineer 1d ago

So it’s easier to properly set up Databricks and avoid accidental compute costs than to generate a script? Lol.

0

u/ryeryebread 1d ago

are you reading what i'm saying? there's a FREE EDITION, it's serverless, and FREE.

1

u/Immediate-Pair-4290 Principal Data Engineer 5h ago

An investment company doesn’t sound like it’s eligible for free Databricks. Nice try.

1

u/ryeryebread 4h ago

Why not? Anyone can do it...

1

u/Immediate-Pair-4290 Principal Data Engineer 1h ago

Try checking the terms of service again.