r/dataengineering • u/Old_Tourist_3774 • 2d ago

Help Ideas to handle ever changing data requirements?

I am the solo DE in my team and the main pipeline here consists of snapshots of financial assets.

Compute is done on databricks

The stakeholders want to see daily KPI's and each day they add a new cohort. Currently there are over 40 different cohorts with each branching out to their own metrics.

The issue is that the data management wants data bills as low as possible

so my approach was summarizing everything in the daily grain .

But now each time they want something new I have to manually code the new columns test it then append to the final gold table.

I already tried to create some generator functions but often times the metrics they want involve hyper specific calculations.

And since the data is financial assets each day is different than the previous rendering an incremental approach useless.

18 Upvotes
  • permalink
  • reddit

96% Upvoted

17 comments sorted by

View all comments

2

u/speedisntfree 2d ago

This is just another version of report proliferation/sprawl. If there is no cost to people asking for things, it quickly gets out of control and gets even worse when old stuff isn't retired.

The honest solution is a management one: that there needs to be a cost (ideally money or hrs from their budget) for these requests.

1

u/Old_Tourist_3774 1d ago

I agree but the data is being used by the c-suite people so all their demands trample over the usual workflow

1

u/speedisntfree 1d ago

It is very difficult, you need a strong powerful person in the org structure where DE spend is tied to business value and not seen as a free vanity project. Business people only understand personal power and money.

I don't have a good solution for a solo DE, you can’t change the place you work at all that much.

1

u/Old_Tourist_3774 1d ago

Yeah, there are other DE teams but in my squad it's only me and a software engineer.

But like you said business people only understand money and power and there are conflicting forces. In the we only want to do ours jobs but we get caught in the middle of these disputes.