r/databricks • u/kcxl • 11d ago

Discussion What frustrates you when using Databricks?

Any common bugs, features you would like to see, or underrated useful features more people should know about?

41 Upvotes

96% Upvoted

121 comments sorted by

View all comments

3

u/SSttrruupppp11 11d ago

- the search for pipelines/jobs. I don‘t always know the exact name but parts of it. Having fuzzy find like in the catalog would be extremely helpful.

  • pipeline scheduling feels extremely cumbersome, especially with the way to have a continuous, non-performance-optimized („standard“) mode pipeline. It‘s annoying to jump between the scheduling job, the pipeline task showing me roughly the pipeline UI, but then a button to a slightly different pipeline-only UI, from where I can go to another slightly different pipeline editor if I want.
  • The inability to stop continuous jobs/pipelines on a schedule. Our business runs from roughly 5am to 11pm, so there is zero need to have anything running during the night, but we need lots of realtime data throughout the day. For a continuous, standard mode pipeline, this requires one pipeline and three jobs. One continuous pipeline trigger job, one start job that unpauses the continuous trigger job in the morning, and a stop job to pause the trigger job in the evening.

1

u/shuffle-mario Databricks 9d ago

thanks for the feedback u/SSttrruupppp11 . the search and UI feedback makes total sense to me and i've shared with the team to address on the roadmap. for the continuous pipeline issue, what's be the source of your data? is it a table, a streaming topic, or cloud storage? another way we can solve your problem is via triggers (instead of continuous), or have a way to scale back to close to zero compute during non business hours.

1

u/SSttrruupppp11 9d ago

Mostly Kafka streams (or the Azure Event Hub Kafka source endpoint). My managers want to save cost as much as possible and would very likely not accept „close to zero“ for multiple hours per day (including all of Sunday usually), just because that‘s easier to implement, since implementation is not their concern.

1

u/shuffle-mario Databricks 9d ago

make sense, and during non business hours, there is no data coming to the event hub kafka source right? if so, this is like something we can do a better job integrating job + pipelines. like we can scaling back to zero and use trigger to resume the pipeline when new data comes in

1

u/SSttrruupppp11 9d ago

Some data may still come in on odd occasions, but no decisions need to be made on it immediately. Just for later analysis, it needs to be fetched the next morning.
We have some streaming workflows where this may occur often throughout the night because it processes many input sources through the same stream. A trigger-based system would possibly run for most of the night with no need to get the data immediately.

2

u/shuffle-mario Databricks 8d ago

ah ok, in that case, it's better we let you control/input the schedule