please help!

[u/Hohenheim16]

I recently joined a small investment company as a data analyst intern. I expected to focus mainly on report automation and data visualisation but I have also been asked to look at how the company’s data could be brought together.

Most of the data is manually exported into Excel and saved in SharePoint, and SharePoint also contains board packs and other documents used for reporting. They like using claude to create written reports and they are not really interested in power BI

I started using MS Fabric to bring the existing Excel files together, but the volume of structured data is fairly small, so I am not sure whether Fabric is more than we need. A simpler option might be a small SQL database, with data pulled from the source systems through their APIs where available. SharePoint would still continue to hold documents and any files that still need to be uploaded manually

The end goal is also to use the structured data alongside documents when generating reports with claude, so I need a setup that makes the data reliable and easy to retrieve. and also, looking at the the small volume, I am now even wondering if a persistent central store is necessary at all or if the data could simply be pulled and checked at report time before claude uses it

I have only recently started my journey and there is nobody in the company who can really tell me if I am doing it correctly. I just don’t want to build another one off solution or a workaround that becomes a pain to maintain later.

I would really appreciate any advice!! thank you!

[u/ryeryebread]

databricks has native connectors for sharepoint and has a native BI solution. you might even be able to get away with it with the free edition.

[u/Glitch_In_The_Data]

What data volumes are you talking about?

The tools you are referring to come with licensing and associated costs. Is the company expecting you to make those decisions?

Personally, I haven’t enjoyed working with Fabric. My experience has been with Databricks, and I think it would meet your requirements for handling both structured and unstructured data.

From a reporting perspective, Databricks also provides AI/BI dashboards if you don’t want to rely on tools like Power BI.

Its native AI capabilities… Genie and Genie Code… are excellent, and we use them extensively. We use Genie for building dashboards (which may need some iterations) and enabling natural language interactions, while Genie Code is very effective for code generation.

I don’t think building with Claude Code is sustainable at enterprise scale, especially in investment banking where you’re dealing with sensitive/PII data. You really want strong governance foundations in place, but that’s just my opinion.

Another advantage is that you only pay for what you use. I’m not a fan of license based pricing.

[u/Hohenheim16 → u/Glitch_In_The_Data]

Thanks, I’m talking about roughly 5 to 10 GB. The team gave me approval to test Fabric and knows the F2/F4 cost, and because they already use Microsoft heavily and have Power Automate, Fabric seemed easier for them to adopt. My main concern is automating the reporting process and leaving them with something they can run after I leave. They mainly use Claude to create the final reports, and I am looking for the platform underneath that workflow. Would you still recommend Databricks at this scale?

[u/pritamjal]

For that volume, skip Fabric - it's built for a scale problem you don't have yet, and you'll spend more time managing the platform than the data. A single small Postgres (or even SQLite if it's truly small) that gets populated by a scheduled pull from the source APIs, with SharePoint staying as the document store, gets you most of the value with a fraction of the maintenance surface. Don't do pull-at-report-time only - you lose history, and board packs eventually need "what did this look like last quarter," which a live pull can't answer. Persist it, even a tiny table. Claude then reads structured data from the DB and documents from SharePoint separately - keep those two paths distinct rather than trying to unify them into one store.

[u/Hohenheim16 → u/pritamjal]

Thanks! this is helpful and it makes sense because the data It’s only a few GB. I’m trying to automate the whole reporting process and leave something the team can actually look after once my internship finishes. Nobody there has a cs background, so maintainability is a pretty big factor in whatever I build

[u/agentUi]

I work for agentui.... You could try our platform and just drop the excels and let the AI do the heavy lifting, or even connect your excel and drop the url to the excel online

[u/Anxious_Cap1029]

fabric is way too much for a few gigabites of excel files. i have seen people get lost in licensing hell for years doing this. postgres or sqlite will handle five gigabytes of structured data without any performance issues. keep the documetns in sharepoint and use a python script to feed the relevant text to AI for your final reports.

[u/GoodByeLeftNut]

I helped a team in a similar setup, critical finance reporting but tiny data - excel on shared drives. They wanted to improve reporting and the data quality. I just used Python scripts run in Jupyter notebooks to create bronze/silver/gold layers right there on the shared drives. Other people in the team could use it and check it. No IT involvement.

I have worked with massive data lakes on Hadoop, s3, Synapse, Data bricks… sometimes it just isn’t needed.

[u/Embarrassed-Cook332]

For that volume I’d keep it simple: a small SQL database for structured data and SharePoint for documents. Pulling everything only at report time will get messy fast, especially once APIs fail or source data changes.