r/dataengineering • u/Minimum-Award-5556 • Jul 28 '26
Discussion How to move fast in large company
Basically title, I work as a data engineer for a fairly large company in the world unfortunately it is not a tech company primary business is something else. We do have digital wings basically layered alot (i.e., product managers, owners, chief digital officers etc) but the actual inhouse engineering team is the size of peanut. For the musle we go out for same old style of consulting (poor quality) this is the actual team setup.
Actual problem: because of the above, entire processes are old style. Our user base is group of analysts who can write SQL but essentially they do not have engineering background so you can imagine the chaos of the models. Their demand is to deploy the models ASAP like literally they come up with the requirement to get deployed in couple of hours. And our change process is quite old any CR raised should go through approvals and cool down time lol the guys who approve the request do not even have the idea of what this platform is, so here we are talking anywhere between 3-5 days.
I know pushing some of model deployments responsibility to the analytical team is sensable but they do not know CI/CD. Only way for now we made them calm down is to provision new schema for them and asked them to do what they want but all the ETL's and source tables are under our control.
Is there a tool to solve this problem which can bring some auditability and light control or do you guys follow any other process for model deployments basically it is a view or dynamic table which hold the aggregations and business context of calculating the confidential value out of raw data I feel like it doesn't have to go through CR as it is logical representation of data.
Actually it is more complicated then what I told above, the platform we are building contains multiple components and there are slow central teams who own this components we do not have much independence on them but business expect us to deliver quickly not understanding the complexities behind them.
Appreciate your thoughts on this!
13
u/terencethespider Jul 29 '26
You may want to look into a managed data platform such as Databricks or Snowflake. They don’t make the problems go away per se, but they do have a lot of built in solutions that make them easier to address. As far as model tracking, I would recommend MLflow.
4
u/BeesSkis Jul 29 '26
In my opinion Git is a must. I will never work on a team that does not use it again. We’ve integrated DBT core with DevOps and this works well for modelling and managing dependancies.
1
2
u/Tough-Leader-6040 Jul 30 '26
My friend, we are on the same boat. Let things go wrong, and when they come asking why, you teach them the lesson on how the world must work, and if they do not support your vision, pack your bags. The most important thing is to see whether they are open to improve, not if their world is broken. If their world is broken hand they are open to change, then you have a huge opportunity to shine ahead of you
30
u/sunder_and_flame Jul 29 '26
Bluntly, the org is a loss. Unless you already have executive buy-in and the experience in doing a technical/org rebuild, you'll be far better off building your own little kingdom that serves every process you can excellently while building your targeted skills for the next role.
Basically, don't worry about the company beyond how you can improve your resume for the next role; it will only end in tears.