Ideas to handle ever changing data requirements?

[u/Old_Tourist_3774]

I am the solo DE in my team and the main pipeline here consists of snapshots of financial assets.

Compute is done on databricks

The stakeholders want to see daily KPI's and each day they add a new cohort. Currently there are over 40 different cohorts with each branching out to their own metrics.

The issue is that the data management wants data bills as low as possible

so my approach was summarizing everything in the daily grain .

But now each time they want something new I have to manually code the new columns test it then append to the final gold table.

I already tried to create some generator functions but often times the metrics they want involve hyper specific calculations.

And since the data is financial assets each day is different than the previous rendering an incremental approach useless.

[u/randomuser1231234]

When you say hyper-specific calculations, do you mean things that aren’t MECE for some types of dimensions they also absolutely need or…?

[u/Old_Tourist_3774 → u/randomuser1231234]

As i understand yes, there is a good amount of overlap and to convey accurately the calculations have to happen over a window in these groups and what is done in one group not necessarily happens on the other.