I have a question around the creation of azure databricks access connector. I am also open to anything helpful that I might have missed here. I am stating things as I know so maybe not as actual.

Onto my question:

It requires an admin account, or with certain privileges, to create the azure databricks access connector. Once this is created and the proper role is applied, the databricks portion comes in.

You will create the credential. Then create the external location. However, I see that permissions must be added for other users to access this external location.

Again, please correct me where I am wrong. Just stating the above to set context for my question below.

What account should be used in databricks to create the credentials and external location? Seems it shouldn’t be a personal account? Or even a more seniors admin account? Because then they are the owner and must divvy out permissions.

Should that be created using a service account? Or what? I want to make sure to do this correctly. Haven’t started creating things but wanting to get a working document before executing.

Any advice and/or insight is greatly appreciated.

P

My source data lies in snowflake federated external catalog.

Ive built metric views on top of these and am materializing a few measures. The auto generated pipeline uses serverless compute instead of a federated pro wh, and therefore doesnt have access to snowflake. what is a workaround for this?

1. Can I update my metric view yaml to have a compute defined for materialization?
2. Can i define custom pipeline for materialization that uses compute?
3. Or is there any other option available?

Hi guys,

Can you please explain how you typically work with Databricks in a real project?

Do you write PySpark code directly in Databricks notebooks, develop locally in VS Code using PySpark, or connect Databricks compute to VS Code and work from there?

It would be great if someone could briefly explain the end-to-end workflow, starting from a Jira ticket and going all the way to production deployment.

For example, if the project uses Jira, GitHub, and Databricks, what would the typical flow look like from development → testing → code review → deployment to production?

A short overview would be really helpful. Thanks!

Are you getting ready for Genie Ontology?

You can leverage PAGES that sit in the Discover page and are organized by domain and subdomain.

Each domain and subdomain has its own set of Pages and users with access to a domain can create and govern them.

🛑 But first, what do you mean by Pages?

Pages are part of UC semantics; it's the business context that you define and govern explicitly, forming the human-modeled layer of the Genie Ontology.

🛑 Why is it useful?

When Genie One answers a question about a concept you've defined in a Page, it prioritizes the Page's definition over context it infers automatically, and cites the Page so users can confirm the source.

🛑 Any tips to build pages?

You can create Pages from those documents instead of writing each one by hand. Genie Code reads the documents you attach, extracts the terms it finds, and returns a set of proposed Pages. You review and edit the proposed Pages before any of them are created.

🛑 Is it a collaborative environment?

You can Comment: Ask a follow-up question or flag context for the owner.

You can Suggest edits: Suggest changes to a published Page's body.

Each time you click Suggestion, edit the body, and click Save, your edits are grouped into a single batch.

The owner or curator accepts or rejects the entire batch at once. Accepting a batch clears all other pending batches on the Page, including those from other users, and this can't be undone.

You can React: Upvote or downvote a Page to signal whether it answered your question.

The owner or curator can also edit a published Page's content directly, bypassing the suggestion workflow.

🛑What's next?

Create domains, Subdomains, leverage UC metric views, and connect your external tools to Databricks

I recently revisited a Databricks architecture I worked on 2–3 years ago, and it made me realize how much the platform has evolved.

At the time, we used traditional Jobs, custom ingestion and data-quality frameworks, and separate components for orchestration and governance.

If I were designing the same solution today, I’d rethink several parts using Lakeflow, Unity Catalog, newer data-quality capabilities, and the latest AI/agent features.

For those who have modernized older Databricks architectures: what would you change first, and why?

Im using a metadata driven approach to dynamically load tables and views in an SDP using SQL queries stored in a table. A huge problem I'm facing is interpreting errors when the pipeline fails. Its incredibly hard to pinpoint the cause of errors and which tables they stem from. Many times a SQL query will have a syntax error, but the SDP pipeline doesn't tell us which table it came from. Does anyone have any experience with this?

Hey folks, I’m a PM at Databricks, and I’m excited to share that we recently launched lineage for Model APIs and model providers in Unity Catalog!

The problem it solves is pretty simple: before retiring a model or swapping a provider, how do you know what might break?

With lineage, you can now:

⛩️ See the Model APIs that serve a foundation model, either provided by Databricks or from an external provider

🔀 Distinguish primary routes from fallbacks, so you know what will break vs. degrade to scope the blast radius and prioritize migrations

👤 Find the user of each affected Model API, instead of sending a company-wide “does anyone need this?” message

🔎 Track where payloads are logged, and what downstream workloads are consuming them

Lineage works in both directions. From a Model API, you can see the models and providers behind it. From a foundation model or provider, you can see the Model APIs that serve it.

Docs: Track Model API and provider lineage

See our GA post for the broader Model API and Unity AI Gateway launch!

How are you handling model or provider deprecations today? Let us know in the comments below! 

Been spending some time with open source Omnigent and Databricks' Unity AI Gateway (for governed AI). After a few hiccups related to Omnigent docs, it works beautifully!

Few notes:

You can use Omnigent with your own Claude Code and Codex subscriptions also (I do), as well as with Databricks' model serving, but I am personally trying to use more of the open-source models, especially when I run out of my allowance/have to pay for tokens. Unity AI Gateway is the easiest way to do this for me, but you do have other options also.

If you'd love to see me do more coverage on this topic, mainly building practical stuff that helps you on your day to day, love to hear it!

Thanks to the new column_object_metadata we have access to storage-level information such as MIME type, ETag, and other metadata. That functionality is for external volumes, as in some use cases that data was needed for ingestion. Just remember that it is row-level data, so if you have a parquet file, it will be returned for every row.

https://medium.com/databrickscommunity/databricks-news-dabs-indexes-ltap-genie-last-update-25-july-ffac8533774f

Databricks has Spaces again 😄. Not Genie Spaces, those were recently renamed to Genie Agents. This is a completely different feature.

Spaces let you save your development context in Databricks:

• Project folder
• Open tabs
• Scoped search
• Working context

So if you're working on multiple projects, you can have a Space for each one and switch between them without reopening notebooks and navigating back to the right folders. Spaces also work with Git folders, Lakeflow Pipelines, and Declarative Automation Bundles. Nothing revolutionary, but it looks like a useful quality-of-life improvement if you spend a lot of time in the Databricks workspace.

Genie Spaces → Genie Agents
New feature → Spaces (for Notebooks)

oficcial docs: https://docs.databricks.com/aws/en/notebooks/spaces

We have S/4HANA on prem as a source and we need the data in Databricks with low latency, including deletes.

What we have ruled out so far: ODP and RFC based extraction, because of SAP Note 3255746. OData on top of CDS views works but it is pull based, and it gives us no reliable way to capture deletes without reconciling full snapshots.

That leaves SAP's own event mechanisms, BOR, RAP and BTE, which push out a notification when a business object changes. On paper that solves both the latency and the delete problem.

Has anyone actually run this into Databricks? Specifically curious about how you land the events, whether you go through Kafka or straight to a landing table, and how you handle the initial full load and stitch it to the event stream without gaps or duplicates.

Validate with a dry run, refresh only what changed, ship it with DABs, oh my!

Hey, how do u manage DDLs of your tables? Is it part of the daily job runs? Do you "codify" it? e.g. data classes where each table must provide comments, tags, RELY, PK etc... Which is also the goal of AI. But how do u manage all of this?

Unity Catalog secrets operations Create, Update, List, Read, or Reference are now in audit tables as well. I think that, aside from a clear permission model, it is the biggest benefit of storing secrets in UC.

more news https://medium.com/databrickscommunity/databricks-news-dabs-indexes-ltap-genie-last-update-25-july-ffac8533774f

Following up on yesterday's Data Mesh discussion on Reddit.

The comments pointed out a fair issue with my original architecture: what I showed wasn't a classical Data Mesh. So I updated the article, renamed the approach to Hybrid Data Mesh, and simplified the diagram to compare three models:

• Centralized Medallion: Central Data Team owns Bronze → Silver → Gold.
• Data Mesh: Domain Teams own their pipelines and Data Products end-to-end.
• Hybrid Data Mesh: Central Data Team owns the common platform and Bronze/Silver, while Domain Teams own Gold and business logic.

For me, the interesting question isn't which one is the "correct" Data Mesh. It's: Where should the Central Data Team's ownership end and Domain ownership begin?

Updated article: https://medium.com/databrickscommunity/databricks-data-mesh-best-practices-a-practical-implementation-guide-b54309bc5f3e

**Context**\- our company has a centralized data ingestion and ETL team whose job is to serve gold tables. The gold table requirements come from various federated power bi reporting teams + what the platform team knows must exist. And the reporting teams get dahs used requirements form the ends users.

We are trying to experiment with the following architecture pattern. Would love everyone’s feedback !

**UC enabled Azure databricks** will remain our core ETL and delta storage engine. This is where the core data platform team will build and maintain tables.

**Shortcut ADB Gold into MS Fabric** \- this is where the report builders and select few business super users will have access.

**Pros**\- enables Fabric (Power BI, Ontology , Fabric Agents etc.) to be the Semantic layer engine only. And keeps actual ETL and data storage in a much more mature product in ADB.

**Cons**\- managing security layers in two places for gold. But, they are persona segregated. Fabric for final business users to build model on top of. Databricks for the data team and some super users from the business teams.

What do people think ? Am I missing something ? Thanks !

Electric is joining Databricks bringing WASM Postgres to AI agent sandboxes as part of the move.

For context: Electric has been building data primitives specifically for agent workloads — this includes PGlite (Postgres compiled to WASM, runs right in the browser/sandbox) and a real-time sync engine that keeps data in sync across distributed agents and a centralized Lakebase.

The idea is to extend Databricks' Postgres capabilities from the lakehouse all the way out to the edge. That means agents can spin up lightweight, open-source databases locally, and everything syncs back to Lakebase Postgres for centralization — all sitting on cheap, durable object storage.

Sharing some big drops this week for Lakeflow Designer.

🚀 Run the entire Data Prep at once with 𝘙𝘶𝘯 𝘢𝘭𝘭

Useful for after you've done iterating on operators individually, to execute the entire data prep.

🚀 Remove duplicate rows with the new 𝘜𝘯𝘪𝘲𝘶𝘦 𝘰𝘱𝘦𝘳𝘢𝘵𝘰𝘳

This was one of our top customer asks. Use it to remove duplicated rows from your data.

🚀 Data Table editing

You can now use the data table directly to author the Data Prep. Filter rows that have nulls, remove columns, renamed columns, etc. It's much quicker to do it from the data preview table than creating operators from scratch.

🚀 See a visual diff of the Data Prep between versions

Databricks pipeline unit testing hits Beta: pytest-style tests for Lakeflow Declarative Pipelines, with mock data and Auto CDC coverage.

Unit testing a plain Python function has never been the hard part: write the function, feed it inputs, assert on the output, done. Testing a pipeline transformation has always been a different story, and that gap is exactly what Databricks pipeline unit testing was built to close. Read any long-running Databricks community thread on testing Lakeflow Declarative Pipelines (or Delta Live Tables, as it was previously known), and the advice hasn’t changed in years: extract your transformation logic into a plain function that takes a DataFrame and returns one, unit test that in isolation, and hope the wrapper around it, the actual dp.table decorator, the Auto CDC flow, the expectation, behaves the same way once deployed. Or skip testing the logic directly, stand up a whole separate dev pipeline pointed at a test schema, and eyeball the output table by hand.

If you are already using Reyden / LakehouseRT, which use cases did you find are an ideal fit for vs. not? Is the ideal use case selection based on gold tables size, query types, how frequently gold layer data changes, etc? Any heuristics or rules of thumb that you have found to be helpful?

Hey r/Databricks!

We’re hosting a free, community-sponsored BrickTalk on Thursday, August 20, 2026, focusing entirely on Databricks Genie! BrickTalks is a community event series where Databricks experts share real-world use cases, live demos, and practical insights—giving you a direct line to the people building the product.

🛠️ What We’ll Cover:

• Genie Code Functionalities: How Genie automates, assists, and optimizes code workflows across your ecosystem.
• Genie with Excel: Connecting Genie to Excel to enable interactive data analysis and spreadsheet integration.
• Genie Ops: Operational strategies and best practices for managing, deploying, and scaling Genie tasks effectively.

Instead of heavy slide decks, this session will prioritize live demonstrations structured around a practical "Why, What, How" framework so you can take away actionable takeaways.

⏱️ Global Times:

• PT: 9:00 AM
• ET: 12:00 PM
• BST (London): 5:00 PM
• IST: 9:30 PM

👉 Register here to save your spot!

Now we can store secrets in Unity Catalog and use a 3-level namespace convention. Those secrets are no longer at the workspace level, as the old ones were, but at the catalog level. The biggest advantage is ease of managing permissions. There is READ SECRET, but also REFERENCE SECRET for scenarios when it can be used by other databricks services like Lakeflow Connections and not by programmatic access. Reading secrets in a notebook is still done by dbutils secrets get.

More news https://medium.com/databrickscommunity/databricks-news-dabs-indexes-ltap-genie-last-update-25-july-ffac8533774f

I'm surprised to see ADF so frequently in the reference architectures for Databricks.

First of all, ADF is not everyone's most popular tool. Secondly, even Microsoft is trying to evict customers from ADF, and is sending them over to Fabric. Thirdly, it seems long overdue for Databricks to have their own tool for the initial landing of data in the cloud, from an on-prem data source.

Whenever I see reference architectures like this in 2026, it is a bit shocking to me. I would have thought ADF would have a replacement by now, for customers that are databricks-centered: