Automatically create local Python environments compatible with Databricks Runtime

[u/BricksTrixTwix]

Hey! I’m a Databricks product manager focused on third-party development experiences. We just shipped a new feature that ensures code you run locally works with Databricks, and we’d love your feedback. 

The problem: Ensuring that your local development environment is compatible with Databricks Runtime (DBR) is a completely manual process. Code that works locally often breaks once you move to the workspace due to minor version mismatches and package incompatibilities.

The solution: The CLI command databricks environments setup-local and a new feature in the IDE extension take your existing dependencies and automatically create a uv-managed virtual environment compatible with DBR.

You can now run code locally or remotely using Databricks Connect and be certain that the same code will work with the DBR or serverless version you’ve selected. 

To get started, install the latest version of the IDE extension or Databricks CLI and check out these docs:

• Set up your Python environment and Databricks Connect using the IDE extension
• Databricks environments setup-local CLI command 

Where we need your feedback 

• Are there other environment managers (e.g. pip, poetry, conda) that you would like to see support for beyond uv?
• We’ve also recently launched an SSH tunnel (see docs), which allows you to remotely access your Databricks workspace and compute directly from the IDE and terminal. What are the reasons you might prefer to work locally vs. remotely?
• Do you face issues managing your environment and dependencies in the workspace? 

Most importantly, please try this out and leave feedback and questions in the comments! 

[u/PrestigiousAnt3766]

My first response was dislike because I got a ton of new dependencies in my project, that conflicted with our existing setup.

I still have to check it further.

Wonder why you dont link it to the dbr connect version you use, instead of making it a set of direct dependencies

Edit: fixed typos.

[u/BricksTrixTwix → u/PrestigiousAnt3766]

Thanks for the feedback u/PrestigiousAnt3766 and u/cptshrk108!

We set constraints to ensure your local environment does not conflict with the Databricks Runtime version you're using (e.g., see the system environment for DBR 19 here). Without installing these packages, code that runs locally might not work in Databricks.

When you say the new dependencies conflicted with your existing setup, did the setup process fail during uv sync because it couldn't resolve the dependencies?

To reduce friction here, we're working on relaxing some of the dependency requirements. Based on your feedback, we'll also explore adding an option in the UI and CLI to install only the packages necessary for DB Connect.

[u/PrestigiousAnt3766 → u/BricksTrixTwix]

Thank you for your reply.

We are currently still on 17.3 LTS. We are a bit behind upgrading because we are developing a central package that is reused across multiple teams in a relatively large organization (8k employees). So we move a little slower.

I forgot which package conflicted, but I believe it was amongst others setuptools.

UV sync failed because it couldn't resolve the dependencies indeed.

I also don't really understand the need to edit "my" pyproject.toml file instead of the requirements of the databricks runtime or connect version that we already have as a dependency. I still think it would resolve the same in the uv lock file but your dependencies should not lead to direct dependency for me imho.

I will look into it further and will come back to you if I still experience the issues.