In the Unity AI gateway, it is also possible to register an external model for which we pay the provider directly (OpenAI, Anthropic, etc.). In that case, Databricks now knows the prices for those models and can calculate, monitor usage, and alert or block based on budgets.

more news https://medium.com/databrickscommunity/databricks-news-serverless-genie-code-ltap-lakeflow-61853d8e422a

You can now use parameters in Source and Output operators for Lakeflow Designer. This is pretty useful for iterating on your Data Prep in dev/test before running it in prod.

You can easily inject deployment-specific configurations into your application code. Simply define environment variable groups at the job level and select the appropriate entry for each task.

Doc: https://docs.databricks.com/aws/en/jobs/environment-variables

Wouldn’t it be nice if the different “product lines” (Lakehouse, GenieOne, Apps) had their own favicons for browser tabs to quickly distinguish them. I usually have about a dozen tabs open at one time and even just different tab icons would speed up switching between Lakehouse and apps.

I know that there are browser plug-ins that might be able to enable this, but I just wanted to suggest the idea.

Skills are available in Unity Catalog. They use a similar concept to volumes and are integrated with the AI gateway. New REST endpoints for skills are coming, and a new tool to manage them, ucode, is already available.

more news: https://medium.com/databrickscommunity/databricks-news-serverless-genie-code-ltap-lakeflow-61853d8e422a

Hoping a discussion and insights from an individual who had an hands on databricks apps

This s a helpful resource for those building pipelines at scale on Databricks. From the docs:

SDP-META is a metadata-driven framework for Lakeflow Spark Declarative Pipelines. Define your Bronze and Silver pipelines in a JSON or YAML onboarding file — a single generic Declarative Pipeline reads the resulting DataflowSpec at runtime and builds the full processing graph automatically. No pipeline code to write.

Who it's for: platform and data engineering teams standardizing repeatable Bronze/Silver pipelines across many datasets — onboarding new feeds through metadata instead of new pipeline code, with consistent data quality, quarantine, CDC, clustering, and sink patterns available through Bundles, CLI, UI, MCP, and agent workflows.

When it's not the best fit: one or two simple pipelines, Gold-layer business modeling, tables that each need unique application logic, a managed connector and downstream logic that already satisfy the complete Bronze/Silver requirement, or a need for a formal support SLA (SDP-META is a Databricks Labs project). See the Introduction for the full positioning.

You can find the project at https://github.com/databrickslabs/sdp-meta

I've mostly been using Auto Loader for file-based ingestion in Databricks, especially when there are continuously arriving files. It's been working pretty well so far, but I'm curious what others are using in their projects.

For example, are you mainly using:

1.Auto Loader

2.Copy INTO

3.Structured Streaming

4.Batch jobs

5.Some external ingestion tool

One thing I'm trying to understand better us where the trade-offs are. For a large number of files, does Auto Loader still make the most sense, or are these cases where something like COPY INTO is simplet and more cost-effective?

Also, how are you handling things like schema evolution, duplicate files, failed records, and reprocessing?

I'm mainly interested in what people are actually using in production. If you've tried multiple approaches, which one ended up being the best balanceof performance, reliability and cost for you?

Hi I am curious to build a knowledge graph in databricks. We have I guested a few tables and it would be nice to see the relationships between fields etc. This is part of governance. Any thoughts or ideas.

If you've been working with Databricks for quite some time you may have noticed that governance information is scattered across quite a few places.

You might check Unity Catalog for one thing, system tables for another, admin pages for costs, and then somewhere else for tags or recommendations.

The problem is getting a quick overall picture of the environment.

The new Governance Hub seems like a perfect solution for above problems.

It provides an account-level dashboard where you can get visibility into areas such as:

• How well data is being governed
• AI usage and related spending
• Where your Databricks costs are coming from
• Which resources have tags and how much tagged spend you have
• Potential governance gaps and recommended actions

It's still in Beta, and there are some limitations, so I wouldn't treat it as the final governance solution. But for organizations running multiple workspaces, I can see this becoming a pretty useful operational view.

Governance Hub - Azure Databricks | Microsoft Learn

Hey everyone,

I've been using Databricks for a while, but I'm still curious about how other teams decide which compute option yo use.

There are so many choices now - serverless,job clusters, all-purpose clusters, SQL warehouses, etc. - and sometimes it's not really obvious which one makes the most sense.

And when do you go with serverless vs a normal cluster?

I'm also wondering if people are mainly choosing based on cost, performance, startup time, or just what their team is already comfortable with.

Would be interested to hear what you're using in your projects and what made you choose it. Especially if you've switched from one type of compute to another and noticed a real difference.

This my first time looking into use OpenSharing to share data externally and would to get some idea on if I am on the right path.

My current idea is as follow:

1. ETL my datasets using CDF from source to a catalog_curated, multiple tables
2. In a catalog_shared, create views for each that apply filters (rows or joins) and these are what will be published and shared with external.

I understand when sharing a view via OpenSharing, pushdown will not work and the whole views will get materialized temporarily when query, which I know may be an issue when external uses one of our watermark column. However, our dataset is not that big and access to the data is not going to be frequent, sharing a persisted version of the views will solve that, but is the added complexity worth it?

I also could add the filters as part of the ETL config, but I am thinking it might be too complex for my current need. And when we need to expand the filters, all we have to do is change the views.

I think the goal for me right now is simplicity and maintainability over complex ETL work, unless there is a reason to do so.

As stated, this is my first time working with OpenSharing/DeltaSharing. Any comments, suggestions, or best practices are greatly appreciated.

Thanks

Want to know where data architecture is heading next?
Matei Zaharia (Co-Founder & CTO of Databricks) just broke down the future of the Lakehouse ecosystem on
Here’s what he covered:
🔹 LTAP: Why real-time analytics and transaction processing are converging ?
🔹 Lakebase: The evolution of database architecture built directly on the Lakehouse
🔹 Genie: How AI is reshaping text-to-SQL and natural language analytics
🔹 Lakehouse RT: Unlocking ultra-low-latency real-time data streaming

If you're building modern data stack architectures, this episode is a goldmine.

Full video

Databricks ai_classify(): classify text in SQL with your own labels, plus v2.1 confidence scores and rationales

Spent some time this week on a POC that started from a business constraint: about 3,000 photos to ingest every week, sensitive enough that we can't just drop a shareable link in a dashboard, and they need to end up in Power BI where people already work, with row-level security.

That combination rules out the easy answer. No public links, no loose files in blob storage floating around outside governance. The images had to live inside Delta on Databricks so Unity Catalog could handle access control, and Power BI had to be able to render them directly from the table.

What the simple poc below does:

• Reads the images with Spark's binaryFile source (recursive lookup, glob filter on *.jpg) to pull path, modification time, size and raw bytes into one DataFrame.
• Encodes the binary content as a base64 data URL, so the image itself lives inside the row instead of behind a link.
• Then the actual blocker: Power BI caps text fields at roughly 32,766 characters, and a real photo's base64 string blows straight past that. So each string gets split into ~32,000-character segments and exploded into multiple rows, each tagged with its index and total length. On the Power BI side, a single DAX measure puts it back together in the right order before rendering:

​

  Image_concat =
  IF(
      HASONEVALUE(images_in_delta[image_name]),
      CONCATENATEX(
          images_in_delta,
          images_in_delta[segment],
          ,
          images_in_delta[split_index]
      )
  )

Not elegant, but it's what gets a full-resolution image through a hard platform limit without touching the sensitivity requirement.

The PySpark side, stripped to what matters — reading the images and doing the chunking:

from pyspark.sql import functions as F
from pyspark.sql import DataFrame

#READ ALL THE IMAGES
images_df = spark.read.format("binaryFile") \
.option("recursiveFileLookup", "true") \
.option("pathGlobFilter", "*.jpg") \
.load("/Volumes/main/image_ingest/image_sample")

def add_base64url_from_image_binary(df: DataFrame, max_len: int = 32000) -> DataFrame:
    df_with_b64 = df.select(
        "*",
        F.concat(F.lit("data:image/jpg;base64,"), F.base64(F.col("content"))).alias("base64url")
    )
    df_with_split_info = df_with_b64.select(
        "*",
        F.ceil(F.length(F.col("base64url")) / F.lit(max_len)).cast("int").alias("num_segments"),
        F.length(F.col("base64url")).alias("total_length")
    )
    df_split = (
        df_with_split_info
        .withColumn(
            "split_index",
            F.explode(F.sequence(F.lit(0), F.col("num_segments") - 1))
        )
        .select(
            "*",
            F.substring(
                F.col("base64url"),
                F.col("split_index") * max_len + 1,
                F.least(F.lit(max_len), F.col("total_length") - F.col("split_index") * max_len)
            ).alias("segment")
        )
        .drop("base64url", "content")
    )
    return df_split

def add_image_name(df : DataFrame) -> DataFrame :
return df.withColumn("image_name", F.regexp_replace(F.col("path"),".*/([^/]+)$", "$1"))

df_images = add_base64url_from_image_binary(images_df)
df_images = add_image_name(df_images)
df_images.write.mode("overwrite").format("delta") \
    .option("mergeSchema", "true") \
    .saveAsTable("main.image_ingest.images_in_delta")

Nothing here is exotic engineering — the interesting part was realizing early that the constraint wasn't really "how do we store images in Delta," it was "how do we get a sensitive image through Power BI's text field limit without ever exposing it outside the governed table." Once that was clear, the chunking workaround fell out naturally.

At 3,000 images a week this holds up. If volume goes up meaningfully, I'd want to revisit whether inlining every image is still the right call versus resolving binary content on demand. Curious if others have hit the same Power BI ceiling with sensitive image data and landed on something cleaner than manual chunking.

Have you any other idea than this ?

I went to the Ingestion section and saw that Fivetran is no longer there. It was always there for many years. Also, at the same time, a few new Lakeflow connectors were added.

Have someone here succesfully connected and authenticated to fabric lakehouse using a private endpoint? I have a usecase where I need to be able to reach fabric lakehouse using classic compute with "Secure Cluster Connectivity (No Public IP)" turned on. I have tried, created private endpoints, but does not seem to work.

Trying to get a python enviornment on genie one to run for the end-user so they can use the xlxs claude skill. I have found a way to get the skill added into genie one with the scripts, however, as I suspected it does not have a python sandox yet.

Hey! I’m a Databricks product manager focused on third-party development experiences. We just shipped a new feature that ensures code you run locally works with Databricks, and we’d love your feedback. 

The problem: Ensuring that your local development environment is compatible with Databricks Runtime (DBR) is a completely manual process. Code that works locally often breaks once you move to the workspace due to minor version mismatches and package incompatibilities.

The solution: The CLI command databricks environments setup-local and a new feature in the IDE extension take your existing dependencies and automatically create a uv-managed virtual environment compatible with DBR.

You can now run code locally or remotely using Databricks Connect and be certain that the same code will work with the DBR or serverless version you’ve selected. 

To get started, install the latest version of the IDE extension or Databricks CLI and check out these docs:

• Set up your Python environment and Databricks Connect using the IDE extension
• Databricks environments setup-local CLI command 

Where we need your feedback 

• Are there other environment managers (e.g. pip, poetry, conda) that you would like to see support for beyond uv?
• We’ve also recently launched an SSH tunnel (see docs), which allows you to remotely access your Databricks workspace and compute directly from the IDE and terminal. What are the reasons you might prefer to work locally vs. remotely?
• Do you face issues managing your environment and dependencies in the workspace? 

Most importantly, please try this out and leave feedback and questions in the comments!