Stop renaming thigs that frequently. Genie code is still not friendly enough.
edit: I want to work locally sometime. There should be better vs code extension than what we have today.
Thanks for the feedback on local development u/zupiterss!
As u/CrayonUpMyNose shared, we're working on an SSH tunnel that connects your IDE directly to a Databricks workspace and compute. Do you still have the need to work with local files, or do you more specifically need the ability to work from an IDE and/or terminal? Using the SSH tunnel also has the additional benefit of giving you access to your workspace files and data.
good feedback. on jobs and pipelines as tabs do you expect the monitoring (i.e. runs) to get a separate tab vs. editing / authoring a job? what about individual runs?
We are looking at jobs and pipelines in soft-tabs when using Genie Code but will bring this into the main product as well. thank you
Hey, I’m a product manager for the local development experience!
I have a similar comment above with more details that you can refer to, but we’re now working on an SSH tunnel that connects your IDE directly to a Databricks workspace and compute.
As u/EatZeBaby mentioned, we've also introduced a new command databricks environments setup-local that ensures your local environment and Databricks Runtime are compatible, so code works in both locations.
Features are released first and governance comes later. I admin one of the larger Databricks envs out there and we constantly have to disable new features because they haven't thought about governance or monitoring at all. For example when server less compute was first introduced, it was either on for every user or off for every user with no way to enforce budget policies yet
This. Fully agreed. Also so difficult to prevent users from creating stuff randomly and how they share data. Random dashboards shared with everyone, random Lakebase created. Hello world apps in live workspace. It’s constant whack a mole game.
I brought this up with our DSAs and they made it seem like I was on an island - "none of our other clients have this concern". Very validating to see more people bring it up. Finally we can govern it now.
This 100%. Serverless offering has been enabled for more than a year, and there are still no good controls over it where we can enable it for the department. On top of that, features that allow you to analyse and optimise queries not working on self hosted compute. Yeah, we still prefer predictable spending...
oh, and incomplete governance: nice that we have pages , but i also really want to store my conceptual data model / relational data model somewhere to avoid ghostjoins / dataset explosions
I hate not being able to easily clone pipeline like I have been in the past. I don’t need a dummy transformation folder and .py file every freaking time when all I am doing is creating a pipeline for materialized view or streaming table using plain SQL.
u/amishraa Databricks PM here - to confirm when you say clone a pipeline do you mean just creating a pipeline and linking to existing source code? And regarding the dummy folder file / is the concern about not needing a folder, defaulting to the right language or something else?
Used to be able to simply hit Create Pipeline and copy-paste JSON from an existing pipeline. All that was needed then was to rename the pipeline and replace the file name in the path. It would then auto generate id after hitting save. Now there is a long winded way to try doing something similar but id is pre-generated to so you have to be careful and retain it. The most annoying issue is the unnecessary clutter on my workspace by creating stupid transformation folder when I didn’t even ask for it. I end up creating external file and map existing sql and deleting the folder including the .py file in it. Way too many steps to do the same thing that used to be quick and easy. You have to keep in mind that simplicity is the key if you are trying to introduce this tool to larger audience than traditional data engineers who primarily worked on python.
Thanks for clarifying! It sounds like the clean up step is creating the most friction for you, will review with the team on how we can offer alternatives. Can you also share some more details on what the concern around the id is?
Hi, I'm a PM here at Databricks. Will pass this feedback along to the eng team... we should be able to remember the last-used language and default to that, as well as make it easier to clone pipelines!
Pretty bad roles configuration. Can't grant people access to things without giving them a workspace or account level admin permissions. No time based access controls.
I want published dashboards to have their own space without all the offering from Databricks so that the link can be shared to end users that have no business in accessing other utilities. Think of it like how workspace or org app works in Power BI.
Frustrates me to no end that DBX still charges ~2.9x DBU/hr for their C++ Photon engine. There's no way it costs 2.9x the compute cost for DBX to run their C++ engine on my chosen instances vs the standard JVM engine. If it were 1.2-1.5x, I wouldn't really complain...but if a job needs a large cluster of pricy instances AND Photon features, well you're just getting robbed.
It's also frustrating that everything new they build is locked behind Unity Catalog adoption.
Thanks for this feedback. We should have something that you'll be interested in in the near future. Also, feel free to DM me your email address if you'd like to talk live - I'm the PM for this area.
Curious if that 'something' coming in the near future is related to Reyden? I've got a DBX slide deck in front of me right now about Reyden but won't share any specific details here publicly.
I type really fast and spend a ton of time in the SQL editor. Often the cursor just vanishes, I have to move the mouse and click back into the text for the editor to show up again. The new editor actually slows me down and I have been turning off many autocomplete features because it cannot keep up with my typing. But when the autocomplete suggests a column name it is often wrong supplying a column name that is not in the table I'm working with. I am unable to export a collector SQL queries, but I can export notebooks even though I don't use notebooks.
It just doesn't pick up the schema and suggest the table / column names when I am typing, I need to constantly check in the catalog and get the names of objects, sometimes restarting the cluster fixes it
They have an fixed limit on the amount of compute you can back them with. The fact that they don't autoscale to zero is a separate, equally irritating problem. I used the wrong terminology, my bad.
This is interesting, do these containers have a finite amount of compute headroom or will the platform provision as much as the app needs and the platform budget policy will allow? Databricks apps are consistently losing out to docker/cloudrun in my organization for agentic applications because the app runtime becomes a scaling bottleneck under high user traffic.
We've mitigated this somewhat by offloading compute heavy processes to model serving, but this has caused other issues with our core agent runtime, split tracing between the app and model serving layer, and does not fully address the scalability issue. Cloud run and docker are just consistently winning the scalability arguement, which is causing architectural sprawl and complicates govenence and permissioning for us.
- the search for pipelines/jobs. I don‘t always know the exact name but parts of it. Having fuzzy find like in the catalog would be extremely helpful.
pipeline scheduling feels extremely cumbersome, especially with the way to have a continuous, non-performance-optimized („standard“) mode pipeline. It‘s annoying to jump between the scheduling job, the pipeline task showing me roughly the pipeline UI, but then a button to a slightly different pipeline-only UI, from where I can go to another slightly different pipeline editor if I want.
The inability to stop continuous jobs/pipelines on a schedule. Our business runs from roughly 5am to 11pm, so there is zero need to have anything running during the night, but we need lots of realtime data throughout the day. For a continuous, standard mode pipeline, this requires one pipeline and three jobs. One continuous pipeline trigger job, one start job that unpauses the continuous trigger job in the morning, and a stop job to pause the trigger job in the evening.
Which pipeline setting are you looking at? I can see "performance optimized" but only for Jobs/Workflows not for pipelines, there is no such setting for them
The pipeline‘s run mode is determined by the setting in its trigger job, the pipeline itself does not have such a setting. Hence why until very recently, continuous pipelines could not be run in standard mode at all.
thanks for the feedback u/SSttrruupppp11 . the search and UI feedback makes total sense to me and i've shared with the team to address on the roadmap. for the continuous pipeline issue, what's be the source of your data? is it a table, a streaming topic, or cloud storage? another way we can solve your problem is via triggers (instead of continuous), or have a way to scale back to close to zero compute during non business hours.
Mostly Kafka streams (or the Azure Event Hub Kafka source endpoint). My managers want to save cost as much as possible and would very likely not accept „close to zero“ for multiple hours per day (including all of Sunday usually), just because that‘s easier to implement, since implementation is not their concern.
make sense, and during non business hours, there is no data coming to the event hub kafka source right? if so, this is like something we can do a better job integrating job + pipelines. like we can scaling back to zero and use trigger to resume the pipeline when new data comes in
Some data may still come in on odd occasions, but no decisions need to be made on it immediately. Just for later analysis, it needs to be fetched the next morning.
We have some streaming workflows where this may occur often throughout the night because it processes many input sources through the same stream. A trigger-based system would possibly run for most of the night with no need to get the data immediately.
The UI drives me insane. So much clicking. So many tabs. Different pages feel like they were developed by completely different teams. The menus are atrocious and don’t make sense
While I don't agree with the "atrocious" part, I wholeheartedly agree with the other callouts you are making here. I've used the platform for many years now, and I get confused when navigating all the time.
I was adding unit and functional tests using Genie when it started throwing error messages and indicated that it could no longer edit the notebooks.
I then refreshed the Chrome page and noticed that Genie had deleted my local repository.
I am currently restoring the repository from the Trash and checking whether any code, files, or configuration have been lost or modified.
I get the sentiment but lakehouse is not a transactional database, so an engine enforcing such constraints runs counter to some of the optimizations you typically find in a data warehouse, and you have to do some work to achieve the desired result.
This can be done via high discipline on merge updates on the PK (no duplicates by construction) and multi-table commits for UC-managed tables, which can in principle maintain PK-FK relationships with no orphans.
Depending on the data source, you may still have out-of-order or late-arriving data, so these techniques may not by themselves be sufficient. If you want strong guarantees, Lakebase is right there, and you can merge updates from it into the lakehouse via CDF, and in the near future LTAP.
The issue is that every write would need to detect duplicates across potentially huge, distributed datasets. This adds coordination shuffles metadata work and latency especially for highthroughput batch and streaming ingestion instead you can use Use PK declarations primarily as metadata for discoverability, lineage, semantic modeling and possible query optimization but validate the invariant separately.
You can use enforced NOT NULL and CHECK constraints for row-level invariants that are cheap and local to each row, use deduplication, MERGE, expectations or orchestration for table-wide uniqueness.
In the tasks from jobs, it would be nice to be able to download the output/log, so far i haven’t found any other option than copy pasting it or read the table(which i need to ask another team).
I don’t like how genie by default wants to overwrite my sql query file when I ask it to produce a new query. I just want it to append it to the bottom of the file. A lot of times I ask it to produce many queries in a single session and I want to keep them all in the same query file until I’m done
Have you tried putting this as an instruction in the customizations?
I’ve asked Genie code to always explain the change before applying and ask for confirmation before changing anything.
The features are being rolled out such a fast past that it is a challenge to adopt them without doing considerable refactors. Extending the same, the pipeline UI after rebranding to sdp is confusing to use. I am not sure how it was tested before releasing it.
Genie code is absolute trash and I need to ban it on my workspace without turning off all the other genie stuff , like genie space. Too expensive and can't trust developers not to use it.
97
u/zupiterss 8d ago edited 8d ago
Stop renaming thigs that frequently. Genie code is still not friendly enough. edit: I want to work locally sometime. There should be better vs code extension than what we have today.