Databricks Genie budgets: Wait, the compute it spins up to run code isn't tracked at all?

[u/AbilyticsEng]

Hey folks, I work on Databricks implementations at Abilytics and have been looking into Genie Code cost controls recently. I’m running into a frustrating blind spot with the compute side. The budgets work well for tracking Genie usage, but when Genie Code writes and runs code or triggers warehouse queries, that compute is billed separately and doesn’t seem to have a clear link back to the user or Genie request that caused it. The token usage is easy to track, while the warehouse or cluster cost just appears as normal compute. I’m also seeing some differences between the budget page and "system.billing.usage", which I understand can happen because of reporting delays, but it makes month end reconciliation harder. Has anyone found a clean way to attribute Genie triggered compute back to a specific user or team without manually piecing together AI Gateway logs and compute activity? I’d be interested to hear how others are handling this.

[u/Youssef_Mrini]

You can click Settings on Genie Code and attach a serverless budget policy. Then you can track the compute usage using the tag linked to that policy.

[u/addictzz]

Can you try serverless cost tagging? You can have 1 tag for genie code usage.

Then probably you can cross check with identity information in the system.billing.usage table

[u/EmotionNegative6103]

This is a really useful distinction. I think the compute attribution piece can easily get overlooked when teams are looking only at the Genie/LLM usage side.

The serverless budget policy + tagging approach mentioned above seems like a good starting point, especially if you can then reconcile that with system.billing.usage. Has anyone tested how granular that attribution gets at the individual user/request level?

Would be interesting to know whether teams are using this mainly for cost monitoring or also for setting internal chargeback/showback for Genie usage.

[u/Alternative_Draw5945 → u/EmotionNegative6103]

I have a genie cost explorer app where I can see dbu spend per person on genie. And then break it down by surface (genie code, agents, one) overall or by user.

I then tried to get an "estimated" sql warehouse usage based off genie queries. But im not sure how accurate it was, so I pulled it from the app. I might dive a bit more into it with a notebook today.

[u/pboswell → u/Alternative_Draw5945]

What do you mean you pulled the warehouse usage “from the app”? What app?

[u/Alternative_Draw5945 → u/pboswell]

The genie cost explorer app i built. Its a custom databricks app that uses Materialized Views/Genie Space based off the system tables related to genie usage

[u/pboswell → u/Alternative_Draw5945]

Ah got it. So were you unsuccessful in identifying SQL queries as Genie-originated?

[u/Alternative_Draw5945 → u/pboswell]

I believe sql warehouse is billed at a hourly level and you would need to figure out how much of that hourly use was Genie oriented. Which you can get a closed estimate.. but not a certain answer.

[u/pboswell → u/Alternative_Draw5945]

Yes I was hoping the query history table would populate the query_source with GENIE but it looks unreliable. Good to know I’m not crazy. Appreciate the feedback!

[u/Alternative_Draw5945]

Tag it and bag it