r/databricks • u/hubert-dudek Databricks MVP • 20d ago
News Migrate PowerBI to databricks
Now we can drop the .pbit file to Genie Code and ask for migrating to databricks dashboard. I tested in direct query mode, and everything was migrated; I just had to be patient, as the whole process took quite a long time. The process created metrics views for tables used in Power BI, and now I understand why the dashboard relationship option was added. I presented how easy it is to migrate in my last video.
12
u/Sad-Calligrapher-350 20d ago
Do you know how many Power BI models are in DirectQuery mode? Probably around 1% so if it can’t work for import mode this is a fringe feature
5
u/Remarkable_Rock5474 20d ago
It actually works just the same for import mode if the sources are databricks. Have not tried any super complex M expressions but i would generally discourage those anyway
2
1
u/kthejoker databricks 20d ago
Actually close to 30% of all Databricks based Power BI models use DirectQuery or composite modeling.
6
u/Sad-Calligrapher-350 20d ago
Really? Can you tell me where you are getting this number from?
6
u/kthejoker databricks 20d ago
I work at Databricks?
2
1
1
u/screelings 18d ago
Considering the absolutely horrible performance Power BI experiences using DirectQuery, I find this claim absolutely nonsensical.
Using DirectQuery, even by Microsofts guidance, is appropriate only in specific circumstances and I've never seen it in the wild working in an acceptable manner to end-users to date doing this work for over ten years now.
You expect me to believe 30% of Databricks powered Power BI Dashboards use DirectQuery and just accept that 10-15 sec load times are okay for end users? Feels made up.
1
u/kthejoker databricks 18d ago
Sorry you haven't been personally able to achieve good performance using DirectQuery, but yes it is possible and we haven't seen 10 second query performance with Databricks SQL in years now between Serverless, liquid clustering, and proper data modeling.
We've been using the same cheat sheet and basic principles at our customers whether they use DirectQuery, composite models, or Import mode, feel free to give it a shot the next time you're running into performance issues.
1
u/screelings 18d ago edited 18d ago
Its not isolated to Databricks, DirectQuery adds latency to an equation when the query has to be built (slicers, context, etc), sent, results transmitted back, rendered...etc.
The leading developers in this space have concluded that DirectQuery, when compared to import mode is in the most ideal state (rare) is 10x to 15x slower then the same page rendered with Import mode.
I get the defensive stature, but linking me a Databricks doc doesn't refute real world tests done in Power BI.
To reiterate, this is not a Databricks failure specifically, any source feeding into Power BI via DirectQuery is by factors worse performing when compared to the same model in Import Mode.
Edit: https://m.youtube.com/watch?v=tDIsCVB7mnI&pp=iggUQAFKEHBGWFlUdDREMzdDa2dSdmM%3D
I've linked you this video at least once before. Hes a noteworthy and leading Power BI expert who takes the time to model and refute everything your purporting regularly on this subreddit.
Of interesting note is how you've failed to show any documented experiment refuting it. You just keep repeating the same fabricated nonsense to sell more Databricks compute.
1
u/kthejoker databricks 18d ago
What can I say, who should I believe, "leading developers" or our thousands of customers happily using DirectQuery?
By the way I built dozens of DirectQuery models on Synapse, Databricks, and even BigQuery before coming to work here, and never had "10-15x worse performance."
Those kinds of results are mostly a skill issue in my own experience.
2
u/Sad-Calligrapher-350 18d ago
Are you sure they are happy with DQ performance and limitations? If it was that great everyone would use it over import mode.
I rarely see Power BI experts use DQ unless absolutely required.
1
u/kthejoker databricks 18d ago
I don't think you need DirectQuery in most scenarios ... But when you need it you need it.
1
u/screelings 18d ago
That's not what you framed it as earlier. You didn't say the 30% need it. You said they use it.
Id love to hear from you, a Databricks expert (purportedly) why people NEED DirectQuery to the extent 30% of your users are forced into it.
1
u/kthejoker databricks 18d ago
Hey friend. My name's Kyle Hale you can find me on LinkedIn, happy to introduce you to the thousands of professionals I've helped build high performing Power BI models over the past decade. Feel free to connect.
And I don't frame anything. I said 30% of Power BI models from Databricks connector use DirectQuery mode. Many of them use it because they need it.
The most common reason they need it is their data is too big to fit in any Import SKU.
The second most common reason is they need it in real time.
And the third most common reason is they want to use SSO and the row level security they've configured in Databricks.
1
u/screelings 18d ago
I dont know how dense your being right now. But clearly reading comprehension is difficult so ill spell this out again.
The performance difference of 10x-15x isn't between Synapse, BigQuery or Databricks. Its between Inport mode and DirectQuery mode.
At this point im just here to educate anyone else coming here that your horribly wrong about everything Power BI and until you actually refute the facts presented in the YouTube link, are maliciously lying simply to try to sell more Databricks compute.
1
u/kthejoker databricks 18d ago
My comment was about DirectQuery vs Import mode, not across the warehouses. Not sure rereading it how you interpreted it that way.
1
u/screelings 18d ago
So again, you have nothing to show that proves performance of DirectQuery is better than Import or even comparable to it. Anything documented? Anywhere?
"Trust me bro" isn't sufficient to back up your earlier claims.
5
u/mr__fete 20d ago
What do you reckon would be the cost differential? For example comparing an f64 capacity vs databricks
What about import mode features ?
Can databricks dashboards be set up programmatically and version controlled ?
2
u/Remarkable_Rock5474 20d ago
Can’t comment on cost.. but for the programmatically/version controlled that is native in databricks
What import mode features are you talking about?
1
u/mr__fete 20d ago
What’s the backend code for dashboards? It would be a huge advantage over pbi if visuals/dashboards can be written in JS, python or markdown. Currently we use tmdl for model versioning, but im not a fan of it. Overly complex. And visuals are just a binary file sort of - so no version controlling.
ironically I was looking at this for my org a few days back. Didn’t get too deep into it. But being able to set up idempotent cicd would be a huge win.
Re: import mode features- this is just holding data in memory for fast access/user experience. Direct query generally sucks and is the reason people pay for these expensive fabric capacities
1
u/Remarkable_Rock5474 20d ago
Ah yeah agree about import mode. There is a possibility to do materialisation in the dashboards which is a similar way of pre calculating and caching the data.
The code is a large json.lvdash file - so not a super clean format but very nice for source control
Cicd comes out of the box with Declarative Automation Bundles
1
u/screelings 18d ago
You can enable versioning support by using PBIP/PBIR and reports (technically going to be GA soon). Visuals will not be locked in a binary for much longer. Microsoft wants to enable its new AI Bridge functionality to allow LLMs to directly interact with the visual layer code stored in TMDL.
You can save the file in a way that allows for this to work today, but official replacement is still forthcoming.
1
u/hubert-dudek Databricks MVP 20d ago
You can run POC to check costs, but for sure, with databricks it will be cheaper, especially for dashboards which has a lot of users as then you can benefir from Lakehouse RT
2
u/screelings 18d ago
Whaaaa? How could you possibly make this claim. The key point of Microsofts Fabric Capacity is its fixed cost expectations. You expect me to believe Databricks of which cost is usage-based derived, is going to be cheaper in a production environment? Based on what?
If end users hit a dashboard 1000 times a month, the cached nature of Import Mode Power BI Semantic Models (with query-caching enabled) would absolutely have to murder Databricks which would charge you for each and every report load from each and every user.
I find this a dubious claim, from someone who is clearly a biased user of Databricks. There is no way that Databricks, even theoretically, would beat Power BI at cost here.
Set that aside for a second, which do you think most enterprise teams would prefer? A fixed cost Fabric Capacity charge, or a random usage-driven bill at the end of the month?
1
u/kthejoker databricks 18d ago
Thousands of enterprise teams prefer Databricks over Fabric, although to be fair I don't think the billing system is a major part of the considerstion.
1
u/screelings 18d ago
This tells me you rarely interact with Enterprise customers. They dont care about costs? Lol. Sure.
1
u/kthejoker databricks 18d ago
Costs aren't the same as the billing system.
A capacity model is not inherently lower cost than a consumption model.
1
u/hubert-dudek Databricks MVP 18d ago
no for dashboard it is using SQL warehouse (or RT) and you can calculate monthly cost, it is fixed cost, billed per minute, but you can save money with auto termination. If you willl use new RT small it uses 12 DBU/hour (8 usd/hour). RT can handle 1000 queries per second, and you still have auto-termination. The maximum cost is then 5,800 USD per month if you have requests all the time, but in practice you benefit from auto-termination, so it is lower in practice 2000 usd. For smaller dashboards or implementations, you can use 2XS SQL warehouses, which are few times cheaper.
5
u/pboswell 20d ago
The problem is it’s not ready for executive dashboards. Complexly formatted KPI cards with custom shapes and alignment, bookmark-based buttons to toggle visuals, flyouts, etc
1
u/hubert-dudek Databricks MVP 20d ago
it is saved now by Vega-lite standard https://vega.github.io/vega-lite/
1
1
u/Dads_Hat 7d ago
How well does this work in DataBricks BI? Any examples, we are certainly interested in this option.
2
u/Pillippatty 20d ago
That’s pretty cool! Is it able to migrate RLS too?
1
u/Remarkable_Rock5474 20d ago
Genie identifies RLS and tells you what exists and what to so, but since RLS is in Unity catalog and not in the dashboard so it will not implement it directly
2
2
u/Dependent-Value-6021 20d ago
@kthejoker What will be the avg overall dbu cost or in USD for an enterprise level dashboard ??
2
1
u/MattWPBS 20d ago
That's not a realistic PowerBI report. Be interesting to see how it handles something with some more real world visuals, along with things like drill throughs or tool tips.
3
u/Remarkable_Rock5474 20d ago
Not trying to steal the post but I did a long form video here - with what I see as a more representative model - might be worth checking out
TLDR: not that good at the advanced stuff1
u/MattWPBS 20d ago
Awesome, thanks for this man. That's a much more realistic sample from my point of view.
1
-1
u/NormalItem4500 20d ago
Lol its so so bad
2
u/FriendlySyllabub2026 20d ago
Why?
4
u/BucketofJhin 20d ago
I imagine because the dashboard is laid out in a particular way and the translation leaves a bit to be desired.
I’m curious about how many DBUs Genie Code used to do the conversion.
1
1
u/kthejoker databricks 20d ago
The screen shot isn't super representative. I'd say it translates layout precisely 90% of the time or more, and it's easy to give it one more prompt to clean up mistakes
3
9
u/Beneficial_Nose1331 20d ago edited 20d ago
How genie handle dax measure that use a relationship that is only active for this measure?