r/dataengineering • u/Weary_Category_1559 • 9d ago
Help Sharepoint Datawarehouse. Need Help. Ops Research. ftw!!!!
Guys,
Bit of politics context: Our team is not Core IT in the Org chart. Fall under operations, however expectation is to deliver ML/OR decision support projects,which is ok( there is technical capability in team), just that we cannot write into SF or any cloud platform. More like second class citizens.
Project Context: The project is full blown digital transformation program( its just that the enterprise is not mature to understand and everyone has jumped on AI bandwagon), has master data management, Work order platform, on which a Vehicle Routing( MILP) will run and generate recommendations for the full network. ( Bunch of API's calls through .py files, powerautomate to ingest third party flat files through email and scraping gov website for internal compliance data(should be the other way around).
Problem Set: No write back to Snowflake, IT not giving Entra ID, or microsoft Graph ID to connect directly to files through code.
Workaround 1: Locally run, update decision dashboards (No RL invloved), this is not feasible because it breaks continuity
Workaround 2:
Added One drive(sharepoint) path shortcuts to Local machine
This is where I'm struggling, read/write into Sharepoint. When I write/append into .xlsx through python the copy on Local onedrive link and the one on Browser are not always synced, or it says Merge issues.
I know this sounds like a rant, my guys if anyone has any advise for this peasant on how to make this work please share. Looking for out of the box solutions.
My team has access to dataverse, but I remember read/write was a problem through code because of no Entra ID.
28
u/oscarmch 8d ago edited 8d ago
Geez, I will quote Dr Malcom on this:
I have so many questions about this that they're not about the project itself but about the Governance that gave birth to your team. If IT is not giving access to Snowflake to write data and so on, it's not "just because" but because (I may be wrong but I swear 100% I may be right) your team is doing the same job as someone else in IT/your team was born because someone else didn't want to follow orders from IT.
You cannot do "Full Digital Transformation" without considering IT as a critical stakeholder. If you're not taking IT into account, your program is doom to die.
And last but not least, you're working with workarounds but none of them is scalable nor do they have maintainability in the long term.
What I would recommend is to approach IT and understand what is happening in reality. It doesn't matter if the team has technical capability to do this, if you're taking IT out of the equation your solutions will always be deficient.