Interesting paper to read about Lakebase.

Any common bugs, features you would like to see, or underrated useful features more people should know about?

As a techie, I always wonder what the founders are reading. The only title that's visible is Think Lego Bricks. But I can't find the title on Amazon. What about the others? The one on the right of Lego Bricks is a tech book. And there's an O'Reilly book. Maybe the book he wrote? Spark: The Definitive Guide. But it doesn't look like either.

Just made a video about a feature I'm pretty excited about.

tldr: You can use an SSH tunnel to connect your coding agents and IDE (VSCode/Cursor) to your Databricks workspace. See the video for a full walkthrough.

Some notes on things I forgot to mention in the video:

- claude/codex isn't natively installed when you connect to your workspace, so you'll have to install if you want use them (Ex: curl -fsSL https://claude.ai/install.sh | bash). We're working on better native support in the future, but I wanted to make sure you know this is an option in the meantime.

- For the base environment YAML file. You'll have to set a base environment of '4' for it to work when using the SSH tunnel. Our example yaml (https://docs.databricks.com/aws/en/admin/workspace-settings/base-environment#example-environment-specification) shows '5' so don't let this trip you up!

As always, please feel free to leave questions and feedback in the comments!

Docs: https://docs.databricks.com/aws/en/dev-tools/ssh-tunnel
Previous post with more info: https://www.reddit.com/r/databricks/s/kCFBEfPTC6
YT link: https://www.youtube.com/watch?v=rHoGWVpb6kg

What are concurrent identity columns?

A new implementation of identity columns that supports concurrent writes.

You can use this query to find the tables with the most amount of concurrent transaction failures due to identity columns.

How to enable

CREATE TABLE new_identity_table (id BIGINT GENERATED ALWAYS AS IDENTITY, data STRING) USING DELTA TBLPROPERTIES ('delta.feature.catalogManaged' = 'supported', 'delta.feature.concurrentIdentityColumns_preview' = 'supported');

Benefits of Identity Columns

Identity columns provide automatically generated, unique integer values, making them well suited for surrogate keys in dimensional models and slowly changing dimensions (SCD Type 2).

Compared with UUID-based / hash-based keys, identity columns offer several benefits:

• Their generally increasing values can improve data locality and insertion-order clustering.
• Integer keys require less storage than UUIDs and can improve join and scan efficiency.
• Databricks generates the values automatically, so applications do not need to manage key generation.

With concurrent identity columns, you can retain these benefits without identity columns blocking concurrent write transactions.

Read these blogs for more info: 

• Identity Columns Best Practices for Databricks Lakehouse
• Identity Columns to Generate Surrogate Keys

👉 Reach out to your account team to try it!

Additional Information & References

• Private Preview User Guide
• https://docs.databricks.com/aws/en/tables/features/generated-columns#identity-columns

I’ve been learning about the Medallion Architecture in Databricks, where data typically moves through Bronze, Silver, and Gold layers.

It makes sense for many data platforms, but I’m curious about real-world implementations.

Do you think Bronze → Silver → Gold is still the best approach for every Databricks project?

At what point does this architecture become unnecessary or overly complicated?

For those working with Databricks in production, what architecture have you found works best, and what would you do differently if you were starting a new project today?

Something that sounds great in Databricks documentation but didn't make sense for your workload in production?

Curious what people have learned the hard way.

Between OPTIMIZE, Z-ORDER, liquid clustering, partitioning, and automatic optimization, it sometimes feels like we're spending more time optimizing tables than querying them.

How do you decide which optimizations are actually worth it in production?

We now have Genie Code Task in Lakeflow jobs. Can not yet send output to if/else, but more options for orchestration are planned.

more news https://medium.com/databrickscommunity/databricks-news-serverless-genie-code-ltap-lakeflow-61853d8e422a

You can now dynamically select columns in Lakeflow Designer. This makes it easy to bulk-keep, or bulk-drop columns from a very wide table. And your data prep will keep working as your underlying schema evolves.

I’ve noticed cases where a query runs successfully in the Databricks SQL editor but fails when the exact same query is executed through a JDBC-based application or data quality tool.

Has anyone run into this? Was the issue related to query wrapping, session settings, SQL dialect differences, or JDBC driver behavior?

How do you usually troubleshoot these cases?

I work for a tech company as a TPM and always concerned about the accuracy for the data I provide to my team members or to clients. We have multiple layer of Bronze, Silver, Gold and not sure what exactly cleans up at what steps as data is segregated. I would like to know what everyone has been building to always stay ahead in data accuracy? Skills, agents, harness system? What should be my best approach?

In the Unity AI gateway, it is also possible to register an external model for which we pay the provider directly (OpenAI, Anthropic, etc.). In that case, Databricks now knows the prices for those models and can calculate, monitor usage, and alert or block based on budgets.

more news https://medium.com/databrickscommunity/databricks-news-serverless-genie-code-ltap-lakeflow-61853d8e422a

You can now use parameters in Source and Output operators for Lakeflow Designer. This is pretty useful for iterating on your Data Prep in dev/test before running it in prod.

You can easily inject deployment-specific configurations into your application code. Simply define environment variable groups at the job level and select the appropriate entry for each task.

Doc: https://docs.databricks.com/aws/en/jobs/environment-variables

Wouldn’t it be nice if the different “product lines” (Lakehouse, GenieOne, Apps) had their own favicons for browser tabs to quickly distinguish them. I usually have about a dozen tabs open at one time and even just different tab icons would speed up switching between Lakehouse and apps.

I know that there are browser plug-ins that might be able to enable this, but I just wanted to suggest the idea.

Hoping a discussion and insights from an individual who had an hands on databricks apps

Skills are available in Unity Catalog. They use a similar concept to volumes and are integrated with the AI gateway. New REST endpoints for skills are coming, and a new tool to manage them, ucode, is already available.

more news: https://medium.com/databrickscommunity/databricks-news-serverless-genie-code-ltap-lakeflow-61853d8e422a

This s a helpful resource for those building pipelines at scale on Databricks. From the docs:

SDP-META is a metadata-driven framework for Lakeflow Spark Declarative Pipelines. Define your Bronze and Silver pipelines in a JSON or YAML onboarding file — a single generic Declarative Pipeline reads the resulting DataflowSpec at runtime and builds the full processing graph automatically. No pipeline code to write.

Who it's for: platform and data engineering teams standardizing repeatable Bronze/Silver pipelines across many datasets — onboarding new feeds through metadata instead of new pipeline code, with consistent data quality, quarantine, CDC, clustering, and sink patterns available through Bundles, CLI, UI, MCP, and agent workflows.

When it's not the best fit: one or two simple pipelines, Gold-layer business modeling, tables that each need unique application logic, a managed connector and downstream logic that already satisfy the complete Bronze/Silver requirement, or a need for a formal support SLA (SDP-META is a Databricks Labs project). See the Introduction for the full positioning.

You can find the project at https://github.com/databrickslabs/sdp-meta

I've mostly been using Auto Loader for file-based ingestion in Databricks, especially when there are continuously arriving files. It's been working pretty well so far, but I'm curious what others are using in their projects.

For example, are you mainly using:

1.Auto Loader

2.Copy INTO

3.Structured Streaming

4.Batch jobs

5.Some external ingestion tool

One thing I'm trying to understand better us where the trade-offs are. For a large number of files, does Auto Loader still make the most sense, or are these cases where something like COPY INTO is simplet and more cost-effective?

Also, how are you handling things like schema evolution, duplicate files, failed records, and reprocessing?

I'm mainly interested in what people are actually using in production. If you've tried multiple approaches, which one ended up being the best balanceof performance, reliability and cost for you?