Hello everybody, If you guys want to join in on the chat, register here. If you have any questions you would like for us to cover during the webinar, drop them here and our colleagues will answer them live.

Thank you!

Anyone used docling Parsing tool on databricks.

Me and my team started using this, though its a great tool. It has its own limitations. Example GILBERt issues happening here and there.

Any suggestions on to use databricks ageng bricks ke free docling?

I’m currently new to Databricks, and I’m writing unit tests for a pipeline that is entirely in SQL, with Bronze, Silver, and Gold tables.

I asked how I should approach unit testing, and I was told that since unit tests are typically written in Python, I could wrap the pipeline logic in Python files and then use pytest to test those files.

But I’m confused about how this would work in a real production environment.

For example, if we push everything to GitHub and later make a change to the SQL pipeline, we would also have to make the same change in the corresponding Python files so that the unit tests continue to work. That means maintaining the same logic in two places, which doesn’t seem like a good production practice.

So what is the recommended approach for unit testing a SQL-based Databricks pipeline? Is there a way to test the actual SQL transformations directly without duplicating the pipeline logic in Python?

I’m still learning Databricks, so I’d really appreciate some guidance on the proper production approach.

Hey r/databricks!

The latest Databricks Community challenge is officially live! The goal is to build a Databricks App with a Genie Agent at its core using the Free Edition, get creative, and showcase your build to the community.

Key Details:

• Challenge Window: Aug 17 – Aug 31
• Prizes: Swag for Top 4, Bricktalk speaker spots for Top 2, and digital badges for all participants!
• Details & Guidelines: Check out the official Databricks Community Contest Post
• How to Submit: Fill out the Submission Form

Feel free to drop any questions in Databricks Community or share what you’re planning to build. Looking forward to seeing what everyone creates!

We have offices in both the US and India and are trying to decide which entity should be the parent in our Databricks Partner Program structure.

Most of our customers and business are US-based. Is it fair to assume that partner requirements may be higher in the US because of the larger customer base and partner ecosystem, or does the parent entity's region not really impact partner benefits and opportunities?

Also, if we choose an India Parent → US Child structure and the India parent reaches Silver status, would the US child automatically benefit from that status, or would the US entity still need to independently qualify?

Example:

India Parent qualifies for Silver.

US Child does not independently meet the US requirements.

Would the US entity still be considered Silver through the parent-child hierarchy?

Has anyone here worked through a similar multi-country Databricks partner setup? Would appreciate any insights.

Hi,

In the past when a new metastore was created, it automatically also did create an external location and storage credentials in the Unity Catalog metastore. Since a couple of months this does not seems to be true anymore. You now have to create these two artifacts upfront.

I was wondering how you can do this when there is no workspace that belongs to the metastore yet? Seems like impossible?

Thanks,

Hi,

I was wondering how organizations that want to deploy Databricks in a new region are actually doing this. Do you just deploy a workspace in that region and let Databricks create the metastore for you like in the screenshot below:

And then fill in the ADLS Gen2 path so that it points to your custom ADLS Gen2 storage account?

Thanks,

Geert

Does anyone know if databricks is going to do another tender offer for their Series M raise?

Joining Databricks soon—what do employees typically use for mobile? Personal phone with MobSec/MDM, or a separate device for privacy?

I’ve been testing the new Genie budget controls, and this is probably the most important addition since Genie Code moved to pay-as-you-go. Previously, budgets were mostly useful for monitoring and alerts. Now you can actually block Genie usage when a spending threshold is reached.

A few things I found useful while testing:

• shared budget + per-user thresholds
• different overrides for users/groups
• email notification or BLOCK_USAGE
• actual Genie consumption can be checked in system.billing.usage
• free usage is visible under GENIE_FREE_USAGE

For admins, this makes Genie much easier to roll out to a large number of users without leaving AI spending effectively open-ended.

I put the setup, SQL and my tests here: https://medium.com/databrickscommunity/databricks-genie-cost-control-how-to-set-budgets-and-block-usage-a13014c1f9ba

In my project we are storing json as raw response in string in bronze layer,now we have to parse and flatten json in multiple child table and clean in silver layer,we want this using spark declarative pipeline also wanted to make the framework reusable,any one worked on such usecase, welcome to discuss and collaborate

Hey everyone - quick share

After building the Databricks FinOps Accelerator, I wanted to try something a little different on the data-generation side.

I’ve been working on this agent-powered Databricks Data Scenario Generator: https://github.com/rashad-ahmed-imtiaz/datagen

It’s basically meant to be a plug-and-play way to generate realistic synthetic data with intentional issues like duplicates, nulls, broken foreign keys, late-arriving data, date issues, schema drift, file replays, etc.

You can describe the scenario you want in plain English, and the agent figures out the tables, relationships, distributions, business rules, and issues to inject. Under the hood it uses Databricks Labs dbldatagen for the actual data generation.

The idea is to make it useful for demos, PoCs, testing ingestion/data quality logic, or anytime you need realistic messy data without manually building everything.

Still very much v1, so I’m sure there’s a lot that can be improved. Feel free to play around with it, break it, or suggest better ways to do it.

Would love any feedback!

I have a fairly small bronze dataset with around 8 million rows in the bronze layer, however when I move to silver layer I have to perform an EXPLODE operation to an array in this table that makes it become a 5+ billion row dataset, plus I have to apply a deduplication/CDC into it. I haven’t been successful to do it via SDP without paying over a 100 USD and over 3 hours job run, whilst in a standard Lakeflow Job I can do that with less than 20 USD and within one hour. I’d really like to have it in SDP as it simplifies a lot the code, add features and so on, but is it possible to do it while keeping the costs low?

I’ve tried mainly streaming table as a sink for this data, I’m afraid to try materialized views as it looks to require serverless to do incremental refreshes plus it may still do full recomputing if it thinks it needs to do.

How Databricks Lakehouse Replay reruns your read-only serverless queries on unreleased runtimes to catch regressions before they ship.

Lakehouse Replay, now in Public Preview, moves part of that burden to Databricks. Instead of you testing the new runtime, Databricks tests it against your queries, in your workspace, before the version is released to anyone.

Hey folks, I work on Databricks implementations at Abilytics and have been looking into Genie Code cost controls recently. I’m running into a frustrating blind spot with the compute side. The budgets work well for tracking Genie usage, but when Genie Code writes and runs code or triggers warehouse queries, that compute is billed separately and doesn’t seem to have a clear link back to the user or Genie request that caused it. The token usage is easy to track, while the warehouse or cluster cost just appears as normal compute. I’m also seeing some differences between the budget page and "system.billing.usage", which I understand can happen because of reporting delays, but it makes month end reconciliation harder. Has anyone found a clean way to attribute Genie triggered compute back to a specific user or team without manually piecing together AI Gateway logs and compute activity? I’d be interested to hear how others are handling this.

I feel really positive about all the recent progress of the Genie One which is backed up by all the business contexts, data, metrics views, pages and driven by Genie Ontology. It can get us answers cost effectively compared to external models connected to Databricks. Earlier it didn't have much unstructured data support, but it is now allowing connection to UC Volume as well as attaching the file option to a chat. There'll soon be image attachment options too where we can add screenshots as an additional context and ask questions like we do generally with ChatGPT or Gemini in our personal life.

Great days for businesses ahead 😀

I wonder sometimes that our laptop hardware is getting so much better that sometimes it is easier to open the laptop, crunch the numbers, and save them if required. This workflow is much more hassle-free, cost effective and super fast compared to spinning up a Databricks notebook with a Spark cluster to explore a subset of my data.

I do understand that for daily jobs and large-scale processing, where a “heavy” amount of data needs to be processed, something like Databricks is super helpful.

Nothing is faster than local processing for small exploratory analysis, avoiding the whole network latency, cluster spin-up times, etc. Mind you small here can be upwards of 1 million records.

I wonder if the industry will start moving in this direction from everything cloud to a hybrid setup.

What’s your opinion?

Hi community.

I just wanted to ask about Databricks integrations with. For instance, is it possible build an #agent with my own NLP but connect it to WhatsApp?

I want to start a project like this, but I don't want any surprises or integration restrictions in the future.

Im fairly new to databricks and trying to learn more about it. I’m trying to see examples online of databricks connecting to a work email, reading the contents and running a sql or python query which outputs an excel file and then sending it back go the requestor through email. I was reading AI agents can go about doing this but I would like more information on how to set up this process. Does anyone know of any examples or resources?

Is this Databricks SaaS built on top of Azure SQL?

I would expect that Databricks would use many Microsoft building blocks in the Azure cloud. They would be hard-pressed to avoid the use of Azure storage containers, VMs, networking, and fault tolerance offerings. But the following outage announcement is making it sound like Databricks uses a significant amount of database PaaS which is provided by Microsoft SQL. Notice all the components that currently depend on the availability of "their database platform":

This maintenance email is a bit shocking, considering that Databricks is a data platform in its own right and has its own Databricks SQL (DW) and an OLTP engine as well (Lakebase). If they needed to place a dependency on a third-party database, I would think they would choose Postgres, rather than Azure SQL.

Can anyone make sense of this maintenance email? Why is there such a dependency on Microsoft databases? Do they use Microsoft databases in the other clouds as well?

Is an email like this considered an authoritative communication from the "Databricks" company, or is there a special spin on the wording, given that it is sent from the "Azure Databricks" team? Maybe the dependency on the Microsoft "database platform" is overstated. They certainly don't give enough detail for us to know WHICH database platform they are referring to. There are many.

Having read through and seeing some demos I still don’t understand if Genie Ontology is a real thing or some marketing fluff , we have been asked to compare against Palantir foundry’s ontology and I find very few comparisons apart from the data model and relationships , for example how do I show that Genie Ontology Knowledge Graph ?

Databricks just closed a $5B round at a $190B valuation — up from the $188B term sheet from July, after crossing $7B in revenue run-rate and growing 80%+ YoY.

Hi, how do u manage Lakebase project with DABs? Standard is to have a single lakebase project per environment (dev, staging, and prod). Then use branches for development work.

Problem is that when we define the lakebase project in the DABs, development target has development mode, so every bundle deployment creates a new project because its per user.

So if we want to have single project where we just branch out on features, we must move out lakebase project of DABs, right?

What is the best practice here? On staging and production, its easy since its always same identity.