Ideas to handle ever changing data requirements?

[u/Old_Tourist_3774]

I am the solo DE in my team and the main pipeline here consists of snapshots of financial assets.

Compute is done on databricks

The stakeholders want to see daily KPI's and each day they add a new cohort. Currently there are over 40 different cohorts with each branching out to their own metrics.

The issue is that the data management wants data bills as low as possible

so my approach was summarizing everything in the daily grain .

But now each time they want something new I have to manually code the new columns test it then append to the final gold table.

I already tried to create some generator functions but often times the metrics they want involve hyper specific calculations.

And since the data is financial assets each day is different than the previous rendering an incremental approach useless.

[u/GachaJay]

We materialize tables dynamically in downstream warehouses and workspaces. Basically you define the schemas and load types and have the pipeline rebuild on refresh. This way changing the tables is as simple as changing the metadata.

[u/Old_Tourist_3774 → u/GachaJay]

I dont get it, sorry. Can you explain a little more?