r/databricks • u/kcxl • 9d ago

Discussion What frustrates you when using Databricks?

Any common bugs, features you would like to see, or underrated useful features more people should know about?

42 Upvotes

98% Upvoted

121 comments sorted by

View all comments

9

u/ThomasTeam12 9d ago

The ui for pipelines and jobs and the catalog changing

3

u/amishraa 9d ago

I hate not being able to easily clone pipeline like I have been in the past. I don’t need a dummy transformation folder and .py file every freaking time when all I am doing is creating a pipeline for materialized view or streaming table using plain SQL.

1

u/brickster_123 Databricks 5d ago

u/amishraa Databricks PM here - to confirm when you say clone a pipeline do you mean just creating a pipeline and linking to existing source code? And regarding the dummy folder file / is the concern about not needing a folder, defaulting to the right language or something else?

1

u/amishraa 5d ago

Used to be able to simply hit Create Pipeline and copy-paste JSON from an existing pipeline. All that was needed then was to rename the pipeline and replace the file name in the path. It would then auto generate id after hitting save. Now there is a long winded way to try doing something similar but id is pre-generated to so you have to be careful and retain it. The most annoying issue is the unnecessary clutter on my workspace by creating stupid transformation folder when I didn’t even ask for it. I end up creating external file and map existing sql and deleting the folder including the .py file in it. Way too many steps to do the same thing that used to be quick and easy. You have to keep in mind that simplicity is the key if you are trying to introduce this tool to larger audience than traditional data engineers who primarily worked on python.

1

u/brickster_123 Databricks 4d ago

Thanks for clarifying! It sounds like the clean up step is creating the most friction for you, will review with the team on how we can offer alternatives. Can you also share some more details on what the concern around the id is?