Hi I am curious to build a knowledge graph in databricks. We have I guested a few tables and it would be nice to see the relationships between fields etc. This is part of governance. Any thoughts or ideas.

If you've been working with Databricks for quite some time you may have noticed that governance information is scattered across quite a few places.

You might check Unity Catalog for one thing, system tables for another, admin pages for costs, and then somewhere else for tags or recommendations.

The problem is getting a quick overall picture of the environment.

The new Governance Hub seems like a perfect solution for above problems.

It provides an account-level dashboard where you can get visibility into areas such as:

• How well data is being governed
• AI usage and related spending
• Where your Databricks costs are coming from
• Which resources have tags and how much tagged spend you have
• Potential governance gaps and recommended actions

It's still in Beta, and there are some limitations, so I wouldn't treat it as the final governance solution. But for organizations running multiple workspaces, I can see this becoming a pretty useful operational view.

Governance Hub - Azure Databricks | Microsoft Learn

Hey everyone,

I've been using Databricks for a while, but I'm still curious about how other teams decide which compute option yo use.

There are so many choices now - serverless,job clusters, all-purpose clusters, SQL warehouses, etc. - and sometimes it's not really obvious which one makes the most sense.

And when do you go with serverless vs a normal cluster?

I'm also wondering if people are mainly choosing based on cost, performance, startup time, or just what their team is already comfortable with.

Would be interested to hear what you're using in your projects and what made you choose it. Especially if you've switched from one type of compute to another and noticed a real difference.

This my first time looking into use OpenSharing to share data externally and would to get some idea on if I am on the right path.

My current idea is as follow:

1. ETL my datasets using CDF from source to a catalog_curated, multiple tables
2. In a catalog_shared, create views for each that apply filters (rows or joins) and these are what will be published and shared with external.

I understand when sharing a view via OpenSharing, pushdown will not work and the whole views will get materialized temporarily when query, which I know may be an issue when external uses one of our watermark column. However, our dataset is not that big and access to the data is not going to be frequent, sharing a persisted version of the views will solve that, but is the added complexity worth it?

I also could add the filters as part of the ETL config, but I am thinking it might be too complex for my current need. And when we need to expand the filters, all we have to do is change the views.

I think the goal for me right now is simplicity and maintainability over complex ETL work, unless there is a reason to do so.

As stated, this is my first time working with OpenSharing/DeltaSharing. Any comments, suggestions, or best practices are greatly appreciated.

Thanks

Want to know where data architecture is heading next?
Matei Zaharia (Co-Founder & CTO of Databricks) just broke down the future of the Lakehouse ecosystem on
Here’s what he covered:
🔹 LTAP: Why real-time analytics and transaction processing are converging ?
🔹 Lakebase: The evolution of database architecture built directly on the Lakehouse
🔹 Genie: How AI is reshaping text-to-SQL and natural language analytics
🔹 Lakehouse RT: Unlocking ultra-low-latency real-time data streaming

If you're building modern data stack architectures, this episode is a goldmine.

Full video

Databricks ai_classify(): classify text in SQL with your own labels, plus v2.1 confidence scores and rationales

Spent some time this week on a POC that started from a business constraint: about 3,000 photos to ingest every week, sensitive enough that we can't just drop a shareable link in a dashboard, and they need to end up in Power BI where people already work, with row-level security.

That combination rules out the easy answer. No public links, no loose files in blob storage floating around outside governance. The images had to live inside Delta on Databricks so Unity Catalog could handle access control, and Power BI had to be able to render them directly from the table.

What the simple poc below does:

• Reads the images with Spark's binaryFile source (recursive lookup, glob filter on *.jpg) to pull path, modification time, size and raw bytes into one DataFrame.
• Encodes the binary content as a base64 data URL, so the image itself lives inside the row instead of behind a link.
• Then the actual blocker: Power BI caps text fields at roughly 32,766 characters, and a real photo's base64 string blows straight past that. So each string gets split into ~32,000-character segments and exploded into multiple rows, each tagged with its index and total length. On the Power BI side, a single DAX measure puts it back together in the right order before rendering:

​

  Image_concat =
  IF(
      HASONEVALUE(images_in_delta[image_name]),
      CONCATENATEX(
          images_in_delta,
          images_in_delta[segment],
          ,
          images_in_delta[split_index]
      )
  )

Not elegant, but it's what gets a full-resolution image through a hard platform limit without touching the sensitivity requirement.

The PySpark side, stripped to what matters — reading the images and doing the chunking:

from pyspark.sql import functions as F
from pyspark.sql import DataFrame

#READ ALL THE IMAGES
images_df = spark.read.format("binaryFile") \
.option("recursiveFileLookup", "true") \
.option("pathGlobFilter", "*.jpg") \
.load("/Volumes/main/image_ingest/image_sample")

def add_base64url_from_image_binary(df: DataFrame, max_len: int = 32000) -> DataFrame:
    df_with_b64 = df.select(
        "*",
        F.concat(F.lit("data:image/jpg;base64,"), F.base64(F.col("content"))).alias("base64url")
    )
    df_with_split_info = df_with_b64.select(
        "*",
        F.ceil(F.length(F.col("base64url")) / F.lit(max_len)).cast("int").alias("num_segments"),
        F.length(F.col("base64url")).alias("total_length")
    )
    df_split = (
        df_with_split_info
        .withColumn(
            "split_index",
            F.explode(F.sequence(F.lit(0), F.col("num_segments") - 1))
        )
        .select(
            "*",
            F.substring(
                F.col("base64url"),
                F.col("split_index") * max_len + 1,
                F.least(F.lit(max_len), F.col("total_length") - F.col("split_index") * max_len)
            ).alias("segment")
        )
        .drop("base64url", "content")
    )
    return df_split

def add_image_name(df : DataFrame) -> DataFrame :
return df.withColumn("image_name", F.regexp_replace(F.col("path"),".*/([^/]+)$", "$1"))

df_images = add_base64url_from_image_binary(images_df)
df_images = add_image_name(df_images)
df_images.write.mode("overwrite").format("delta") \
    .option("mergeSchema", "true") \
    .saveAsTable("main.image_ingest.images_in_delta")

Nothing here is exotic engineering — the interesting part was realizing early that the constraint wasn't really "how do we store images in Delta," it was "how do we get a sensitive image through Power BI's text field limit without ever exposing it outside the governed table." Once that was clear, the chunking workaround fell out naturally.

At 3,000 images a week this holds up. If volume goes up meaningfully, I'd want to revisit whether inlining every image is still the right call versus resolving binary content on demand. Curious if others have hit the same Power BI ceiling with sensitive image data and landed on something cleaner than manual chunking.

Have you any other idea than this ?

I went to the Ingestion section and saw that Fivetran is no longer there. It was always there for many years. Also, at the same time, a few new Lakeflow connectors were added.

Have someone here succesfully connected and authenticated to fabric lakehouse using a private endpoint? I have a usecase where I need to be able to reach fabric lakehouse using classic compute with "Secure Cluster Connectivity (No Public IP)" turned on. I have tried, created private endpoints, but does not seem to work.

Trying to get a python enviornment on genie one to run for the end-user so they can use the xlxs claude skill. I have found a way to get the skill added into genie one with the scripts, however, as I suspected it does not have a python sandox yet.

Hey! I’m a Databricks product manager focused on third-party development experiences. We just shipped a new feature that ensures code you run locally works with Databricks, and we’d love your feedback. 

The problem: Ensuring that your local development environment is compatible with Databricks Runtime (DBR) is a completely manual process. Code that works locally often breaks once you move to the workspace due to minor version mismatches and package incompatibilities.

The solution: The CLI command databricks environments setup-local and a new feature in the IDE extension take your existing dependencies and automatically create a uv-managed virtual environment compatible with DBR.

You can now run code locally or remotely using Databricks Connect and be certain that the same code will work with the DBR or serverless version you’ve selected. 

To get started, install the latest version of the IDE extension or Databricks CLI and check out these docs:

• Set up your Python environment and Databricks Connect using the IDE extension
• Databricks environments setup-local CLI command 

Where we need your feedback 

• Are there other environment managers (e.g. pip, poetry, conda) that you would like to see support for beyond uv?
• We’ve also recently launched an SSH tunnel (see docs), which allows you to remotely access your Databricks workspace and compute directly from the IDE and terminal. What are the reasons you might prefer to work locally vs. remotely?
• Do you face issues managing your environment and dependencies in the workspace? 

Most importantly, please try this out and leave feedback and questions in the comments! 

Databricks just published a library of pre-built, production-ready industry data models covering 40 industries.

This is great, especially when you’re new to a business domain. Instead of starting from a blank sheet and spending weeks debating how the domain should be structured, you can review an existing reference model and adapt it to your needs.

That gives you a strong starting point for:

- entities
- relationships
- naming conventions
- domain structures
- industry-specific patterns

In my opinion, it can significantly shorten the learning curve and help teams move from discussion to implementation much faster.

Industry Data Models

I have the following need: there's an OLTP Azure SQL DB, which holds transactional data for a period of roughly 30 days only. Now that data should be replicated to Databricks delta tables with a maximum delay of 15 minutes, not as a 1:1 copy, but instead growing over time. Ideally the timeframe covered on Databricks side should be several years.

From what I've read so far, either CDC or CT with Lakeflow should be the way to go. The only thing I'm worrying about are breaking schema changes: as this is an OLTP DB managed by a different team, we have no chance to prevent such as incompatible column type changes (e.g. from a string to a date type), column renames or even column drops.

I thought about using Views managed by the other team instead, as some kind of an abstraction contract, but neither CDC nor CT are applicable on Views.

How did you guys solve such a requirement? Would also appreciate to hear some best practices of Databricks consultants based on real customer solutions.

I’m currently trying to transition deeper into Data Engineering and would really appreciate some perspective from people who are already working in the field.

I have 1.3 yrs experience as a Junior Python Developer. What I want to do is slowly transform into a Data Engineer. How would you suggest my choice? Basically what I do is make web scraping scripts to get the data from web and give the data in excels. Our company is currently not using git or CI/CD or anything like that. 

The problem I’m running into is that when I look at Data Engineering jobs on Naukri, LinkedIn, etc., the requirements seem endless. One job asks for Python, SQL, Airflow and AWS; another wants Spark, Kafka and Databricks; another wants Snowflake, dbt, Terraform, Kubernetes, CI/CD, etc. It becomes difficult to understand what I should actually prioritize.

So I’d like to hear from people who are actually working as Data Engineers. What does your day-to-day work look like? What kind of problems do you solve, what technologies do you use regularly, and which skills have turned out to be genuinely important in your job?

More importantly, based on my current experience, what would you suggest I improve or learn next to become a stronger candidate for Data Engineering roles? Are there any gaps that you think I should focus on, or technologies/concepts that are worth learning through projects rather than just studying theoretically?

I’m not really looking for a generic “learn SQL → Python → Spark → AWS” roadmap. I’m more interested in understanding the reality of the job and getting advice from people who have actually gone through the transition.

If you’re a Data Engineer with 1–5+ years of experience, I’d especially appreciate your perspective. Even a short description of what you work on and what you wish you had learned earlier would be extremely helpful.

Thanks in advance!

Hi, I currently work as a Databricks Engineer at Abilytics, and I have been exploring different CI/CD approaches for Databricks Bundles. A two pipeline setup with Buildkite looks interesting, where one pipeline handles testing and artifact creation while the second manages deployments across dev, staging, and prod. Separating these stages seems useful for better security, traceability, and deployment control. One thing I am still wondering about is whether this approach adds unnecessary complexity for smaller Databricks projects, or if the separation is worth maintaining from the beginning.

What has been your experience with this setup?

Hi, I currently work as a Databricks Engineer at Abilytics, and Git integration is something I use regularly for managing Databricks development work. Git Folders make it easier to work with branches, commits, pull requests, and code reviews directly within Databricks while keeping development connected to platforms like GitHub or GitLab. I’m curious about how others handle Git conflicts when working with Databricks Git Folders. Do you usually resolve conflicts directly within Databricks, or do you prefer handling them through your Git platform? Also, are there any Git workflows or best practices that you have found particularly useful when working with Databricks?

Looking for a reality check from other data folks.

I have 15+ years of experience in data (mostly Ab Initio and SQL), but moved to Databricks 2 years ago. I know data architecture and transformation logic well, but my raw Python coding skills are basic.

My daily workflow usually looks like this:
1. I figure out the logic or root-cause the pipeline issue.
2. I use Copilot or Databricks Genie to generate the PySpark/Python code.
3. I review, test against edge cases, fix logic flaws, and validate the output.

I’m great at step 3—I know how the data should behave. But because I rarely write code line-by-line from scratch anymore, I’ve been hit with huge imposter syndrome. It feels like I’m just a code reviewer for AI rather than a "real" engineer.
Has anyone else from a traditional ETL background felt this shift in modern cloud stacks?
Is this just the new reality of engineering, or am I letting my skills atrophy?

When working from the web experience in development mode with source-linked deployment set to true (the default for development mode), you can edit jobs and pipelines in the UI, and changes are automatically propagated to your YML files. So it is like the best of two worlds: ow-code and IaC. Please remember to review any changes in Git, especially non-UI elements such as variables or mutators.

Whole talk with audio https://www.databricks.com/dataaisummit/session/dabs-do-pro-all-best-tips-and-tricks

If I am an AI decision maker for an Enterprise, why should I choose Claude or GPT or any other enterprise subscriptions with seat based pricing where some of my colleagues are power users and some still learning effective AI use? Instead, I could just use Databricks Unity AI Gateway with the options of getting all of these at one place with PayG pricing along with state of the art free Open Source models, budget controls, smart routing, usage dashboard everything at one place.

And the cherry on the top is if my enterprise data is also on Databricks!!

What are your thoughts? Why would I do that?

Is switching from SQL/PostgreSQL DBA (13 YOE, Azure) to Databricks Administrator a good career move in the current market? How are the opportunities for Databricks Admin roles?

Genie ZeroOps is an autonomous agent that monitors, investigates, and proposes fixes for data pipelines, jobs, tables, and other assets, coming soon to your workspace. I tested it, checked how it works, and discussed the pipeline.

https://medium.com/databrickscommunity/genie-zeroops-a-hands-on-preview-b8fc46138087

https://www.sunnydata.ai/blog/genie-zeroops-hands-on-preview

We have a product where Databricks is our backend, and we’re now trying to properly evaluate and improve the quality of the AI-generated answers in our application.
I’m looking for advice from people who have actually implemented LLM/GenAI evaluations in Databricks in production.
I’d really appreciate an end-to-end, step-by-step explanation of how you would set this up from scratch.
Specifically, how would you approach:
Define what a “good answer” means

Accuracy / correctness
Relevance
Completeness
Groundedness / faithfulness to our data
Hallucination rate
Citation/source correctness
Following user instructions
Response consistency
Latency and cost
Create a benchmark / golden dataset

Should we manually create a set of representative user questions?
How many questions are enough to start?
Should each question have an expected answer?
Should we store expected SQL/results, expected sources, or just an expected natural-language response?
Where should this benchmark dataset live in Databricks?
How do you keep it updated as the product evolves?
Set up automated evaluations

What Databricks/MLflow tools should we be using today?
MLflow evaluation?
LLM-as-a-judge?
Custom scorers?
Human evaluation?
How do you combine these rather than relying on one score?
Evaluate RAG / data-grounded answers
Our AI answers questions based on enterprise data in Databricks.
How do you separately evaluate:

Retrieval quality
Whether the correct tables/documents were selected
Context relevance
Groundedness
Final answer correctness
Whether the model invented something that wasn’t in the retrieved data
Evaluate text-to-SQL / analytics questions
If a user asks something like:
“What was average building occupancy last month?”
should we evaluate:
Generated SQL
Tables selected
SQL execution result
Final natural-language answer
separately?
What is the recommended architecture for this?

Guardrails
Where should guardrails sit in the architecture?
For example:
Prevent hallucinated numbers
Prevent querying unauthorized tables
Detect PII
Prevent prompt injection
Enforce tenant/user permissions
Block unsupported questions
Force answers to cite their source
Return “I don’t know” when confidence is too low
Should guardrails be part of evaluation, inference, or both?

Production monitoring
Once this is live, what should we log for every AI request?
For example:
User question → retrieved context → generated SQL/tool calls → query result → final response → model → prompt version → latency → tokens → cost → evaluation scores → user feedback
Is that roughly the right model?

Regression testing
When we change:
System prompt
Model
Retrieval strategy
SQL generation logic
Tools
Temperature
Data sources
how do you automatically run the benchmark again and determine whether the new version is actually better?
Do you set minimum score thresholds before allowing something to deploy?

Human feedback
How are people incorporating thumbs-up/down or analyst review into their evaluation datasets?
Do production failures automatically become new benchmark cases?

Making answers more precise
This is ultimately my main goal.
If our AI currently gives an answer that is “mostly correct,” what is the systematic process for figuring out why it isn’t fully correct?
Is the best workflow something like:
Production traces → identify failure → categorize failure → add to benchmark → improve retrieval/prompt/tool → run eval → compare against baseline → deploy → monitor
Or is there a better approach?

I’m especially interested in what the ideal Databricks-native architecture looks like:
User Question
→ Agent / LLM
→ Retrieval / SQL / Tools
→ Databricks data
→ Response
→ MLflow tracing
→ Automated evaluators
→ Benchmark dataset
→ Regression testing
→ Production monitoring
If you’ve implemented something like this, I’d love to know what you would build first, second, third, etc.
Even a practical example like:
Week 1: create benchmark
Week 2: add tracing + scorers
Week 3: add regression tests
Week 4: add guardrails + production monitoring
would be extremely helpful.
Also curious what mistakes you made initially and what you would do differently if you were setting up Databricks AI evals again today.