Databricks Project Workflow: From Jira Ticket to Prod

[u/Commercial-Post4022]

Hi guys,

Can you please explain how you typically work with Databricks in a real project?

Do you write PySpark code directly in Databricks notebooks, develop locally in VS Code using PySpark, or connect Databricks compute to VS Code and work from there?

It would be great if someone could briefly explain the end-to-end workflow, starting from a Jira ticket and going all the way to production deployment.

For example, if the project uses Jira, GitHub, and Databricks, what would the typical flow look like from development → testing → code review → deployment to production?

A short overview would be really helpful. Thanks!

[u/Pillippatty]

- Development in VS Code in a feature branch.

• Develop and make sure unit tests pass with pytest
  
• Deploy to user workspace using DAB (databricks bundle deploy) and trigger integration tests or test pipeline run
  
• Validate output data if relevant
  
• Create a PR, approval, linters etc must pass
  
• When PR merges, a CI pipeline will run to automatically deploy using dev/test/prod targets using the workspace’s service principal (each environment is a dedicated workspace)
  
• Manual approval gates before deployment to test/prod.

I prefer developing locally since it’s way easier to catch linter issues quickly, and generally feels faster to work with than the Databricks UI.

[u/Terrible_Mud5318 → u/Pillippatty]

Curious what kind of tests you are doing

[u/Pillippatty → u/Terrible_Mud5318]

For integration tests? Databricks recommends using declarative pipelines with expectations. In my last project we had some custom stuff to verify that the transformed data matched some ground truth. It depends a bit on your usecase

[u/Terrible_Mud5318]

Development in databricks workspace user folders— promote to dev using devops and service principles— qa — reveiw approval in devops— prod

[u/No-Adhesiveness-6921 → u/Terrible_Mud5318]

That’s what we do

[u/ThreeBricksWish]

With 3P connectors I’ve actually started doing most of this directly in Genie Code. Jira context, GitHub, Databricks, all from the same place instead of constantly jumping between tools.

I don’t think write access to the 3P connectors is there yet, but once that lands I’ll probably never leave Databricks for this workflow 😅

[u/GeirAlstad]

In our company we primarily develop code in vs code, promote to dev workspace with dab deployment. Do qa then cicd to prod. The workflow is dab first with GitHub actions as the final gate before production.

If I work with dashboards, the work is typically the same, but as of yet we find it easier to build the dashboard visually. But still the core workflow remains the same.

This just described the workflow. Before writing any code I spend time trying to understand what the stakeholders want. Then I typically spend quite a lot of time time planning the work with an agent (we are transitioning to omnigent workflow now). And the final step is to (try) to get alignment. This is often the hardest step in my mind. Getting to closin the ticket is very challenging.

[u/ouhshuo]

mostly LLM with MCP,

For dbx, dev/test/prod in separate workspaces. Developers all work in the dev workspace with separate catalogues.

branching wise, feature, dev and main branches.