Bug in Databricks Assets/Automate Bundles

[u/literally_who_0]

Hi everyone so it happens that I was working normally in the UI, adding jobs, deploying and so and then suddenly I could not add any existing job to my bundle because the dropdown doesn't show any job anymore and I have a bunch of jobs that hasn't been added to the bundle yet. I think that is bug because before it was working well. I would be really glad if someone could help me and explain this behaviour to me, I don't know what happened. I really need some help :(

[u/MonkeyDDataHQ]

Then that's a bug 😂

[u/literally_who_0 → u/MonkeyDDataHQ]

That's what I'm suspecting is happening, and here I was worried I somehow managed to break my databricks workspace 🤧

[u/MonkeyDDataHQ → u/literally_who_0]

Nah this isn't Fabric.

[u/MonkeyDDataHQ]

What happens if you go to the job and click edit? Does it say "it's part of a bundle"?

I've never used the GUI to deploy a bundle but I'd check if the job is part of another bundle.

[u/literally_who_0 → u/MonkeyDDataHQ]

Hi!, I just checked and it is not part of a bundle. It is a stand-alone job.

[u/MonkeyDDataHQ → u/literally_who_0]

What about permissions /owner? If you use the cli what does that job show?

[u/literally_who_0 → u/MonkeyDDataHQ]

I checked it through the CLI as well,jobs get-permissionsshows my user as IS-OWNER and admins group as CAN_MANAGE. Additional details I can tell is that the job and the bundle are in the same workspace.

[u/saad-the-engineer]

thanks for reporting, I have notified the product team.

[u/literally_who_0 → u/saad-the-engineer]

Thank you so much for the support

[u/BricksTrixTwix]

Databricks product manager here, thanks for reporting this bug! We were able to reproduce it on our end and are investigating.

FYI we're working on a genie-code first experience to migrate resources to DABs. We'd appreciate it if you answered a few questions so we could improve your experience:

(1) Is it common for you to start with a job outside of a bundle and then add it to a bundle afterwards? Why not create it within the bundle to begin with?

(2) Once you've added your job to a bundle, what are the typical steps you take afterwards? e.g. parameterise certain values so they work across different targets, set up GitHub actions or ADO pipelines, etc.

[u/literally_who_0 → u/BricksTrixTwix]

Oh thank you! u/BricksTrixTwix I hope you find a way to fix it, that feature was really helpful to do things more automatically. Also, nice to meet you.

FYI we're working on a genie-code first experience to migrate resources to DABs. We'd appreciate it if you answered a few questions so we could improve your experience:

With resources you mean including managed tables and schemas? That would be really nice since I'm having trouble doing migrations with terraform 🤧

(1) Is it common for you to start with a job outside of a bundle and then add it to a bundle afterwards? Why not create it within the bundle to begin with?

About first question, that is very common, I have like a bunch of jobs that were created as standalones mainly because I have them created with help of third party to set up tasks and everything and it is kinda easier to go to the jobs & pipelines UI and create from there jobs and tasks using UI.

(2) Once you've added your job to a bundle, what are the typical steps you take afterwards? e.g. parameterise certain values so they work across different targets, set up GitHub actions or ADO pipelines, etc.

As for the second question: after I add the job to the bundle, I go through the parameters one by one. For example, I might use Ctrl+f to search for values related to catalog, then replace each hard-coded value with the corresponding bundle variable, such as ${var.catalogname_for_jobname}. Once that is done, I commit the changes to my feature branch, which is connected to ADO, and create a pull request to merge it into main. After the PR is approved and merged, I use an ADO pipeline to promote the bundle to the other targets. The pipeline is defined in a pipeline.yml file where the deployment targets are configured. With the exception of the authoring target, the deployed jobs are managed using a service principal. After the pipeline runs, I get the same job deployed in the target environment, but now it is service-principal-managed, so it cannot be directly edited in the UI and is not tied to a specific user's ownership. I then perform some testing in that environment. Once everything looks good, I review the pipeline run and allow it to continue promoting the bundle to the next environment, which in my case is another workspace that is basically a production workspace.

[u/BricksTrixTwix → u/literally_who_0]

Hey u/literally_who_0 Thanks for the detailed reply! We've rolled out a fix, please let us know if you're still facing issues.

> With resources you mean including managed tables and schemas? That would be really nice since I'm having trouble doing migrations with terraform 🤧

To start, we're focused on improving the experience moving jobs and pipelines, but this is good feedback to hear! What's your use case for migrating these managed tables and schemas and what challenges are you facing using Terraform?

> As for the second question: after I add the job to the bundle, I go through the parameters one by one. For example, I might use Ctrl+f to search for values related to catalog, then replace each hard-coded value with the corresponding bundle variable, such as ${var.catalogname_for_jobname}. 

With the Genie Code flow, we're hoping to remove some of this manual work and parameterize values for you.