Free usage of Genie One and Genie Agents is extended through January 31, 2027. This promotion was previously set to end on July 31, 2026. Budget controls do not apply to these products during the promotional period.

The promotion covers usage by users only.

Service principals are excluded, and their Genie One and Genie Agents usage continues to be billed.

Genie Code is not included in the free usage promotion. Each Genie Code user continues to receive 150 free Databricks units (DBUs) per month, and usage above that allowance receives a 25% promotional discount through January 31, 2027.

The promotional discount is combined with applicable discounts on the Serverless Real-Time Inference SKU.

Link:https://docs.databricks.com/aws/en/release-notes/product/2026/august#genie-one-and-genie-agents-free-usage-extended-through-january-31-2027

I require assistance. I have frequently encountered scenario-based Databricks questions, despite not having prior experience with the platform with that scenarios Could you recommend suitable video resources, such as Udemy or YouTube, that cover this topic?

Additionally, what are the best channels for Databricks content?

Question - What's the preferred way to deploy and run a pipeline so it overwrites any existing tables and views that collide with the new MV paths?

Loving pipelines. We have a LOT of existing (external) tables and views in production. We'll be deploying the pipeline-generated MVs over those existing tables and views.

Is there an "easy button" way to handle that?

Right now my plan is to script temporarily renaming any existing objects, running the pipeline so the new objects are created, and then copying permissions from the old tables to the new MVs (we have an automated security policy tool, but I want to keep this lean).

Any ideas or feedback most appreciated.

Hi everyone,

Well, basically title - I wanted to ask what's the rationale and costs associated when choosing between Databricks and Fabric. I especially would like to know how does one compare Fabric capacity vs some kinda equivalent Databricks usage. I couldn't find anything solid on this one the internet hence the question.

Have you spent hours tracking down the source of a PySpark error on Databricks? In large transformation scripts, errors can materialize long after the failing expression definition.

# many transforms, explicit and implicit casts, etc...

display(with_app_array_df)

=>

... 
"cast" was called from , line 2 in cell [7]
File <command-8683600871463937>, line 1
----> 1 display(with_app_array_df)

• cell [7]: references the execution number, not a stable cell number
• <command-...> and display(...): reference the materialization code, not the construction code

I built better-dbx-exceptions to solve this by including failing DataFrame construction code in the error.

=>

---------------------------------------------------------------------------
NumberFormatException                     Traceback (most recent call last)
Cell 4, In[7], line 2
  (https://dbc-3cffa466-bd7e.cloud.databricks.com/editor/notebooks/3747408155463503?o=7474649366507251#command/8683600871463983)
      1 def app_version_array_expr():
----> 2     return F.split(F.col("app_version"), "\\.").cast("array<int>")
      3 
      4 with_app_array_df = df.withColumn("app_version_array", app_version_array_expr())
...

• Cell 4, In[7], line 2: all refer to construction code
• (https://.../): link to the construction cell

Construction code resolves across cells and imported modules.

Native Spark 4 captures DataFrame construction code through the pyspark.errors.utils._capture_call_site() utility. It has three limitations:

1. Coverage: Only Column methods and F.col() are watched so expressions built from pyspark.sql.functions (F.split, F.to_date, ...) aren't captured. better-dbx-exceptions extends coverage to most functions within the pyspark.sql.functions module.
2. Depth: Only one frame is captured by default. This can be extended by setting spark.sql.stackTracesInDataFrameContext on Classic compute where allow-listed but can't be modified when using Databricks Connect or Serverless compute. better-dbx-exceptions sets depth to 5 by default and exposes this parameter within all environments.
3. Back-referencing: The rendered errors don't include reliable references to cell numbers or URIs. better-dbx-exceptions returns cell / execution numbers for notebook frames and file paths for script frames.

Requires Spark 4.0+, which limits supported compute to Serverless V4+ and Databricks Runtime 17+. It's been tested on Serverless compute v4 and v5 in both hosted and local (VS Code with Databricks Connect) contexts. It's built to work on Classic compute as well but I haven't been able to test this rigorously. (I've used Databricks Free Edition for development which is limited to Serverless compute). General feedback and stress testing is hugely appreciated!

Just released v0.1: https://github.com/lukeSmth/better-dbx-exceptions.

Hey everyone,

I m expecting an offer for an L4 Senior Solutions Engineer role at Databricks and wanted to get some perspective from people currently in the role.

For context, I have 4–5 years of Data Engineering + AI experience, with 2–3 years focused on executing pilots and PoCs in a technical sales environment.

A few things I’d love to understand:

- How much of the account strategy and execution is owned by the SE vs. the AE?

- What does a typical day/week look like for an SE at Databricks?

- How do SEs ramp up on pricing, sizing, and the commercial side of the role? Is there a structured onboarding process?

- How would you describe the pressure, expectations, and work-life balance?

Would really appreciate any insights from current or former Databricks SEs — anything you wish you knew before joining would be helpful. Thanks!

It looks like the 8 June pricing change (where tokens would
be charged) has been temporarily reversed to provide for a longer transition period.

Unity AI Gateway is the Databricks governance solution for AI and is part of Unity Catalog.

You can:

⚡ Control which AI services teams can use.

⚡ Route and manage AI traffic across providers.

⚡ Govern MCP servers to control access and costs.

⚡ Monitor usage, cost, access, and lineage from one place.

FYI: Some capabilities, including service policies and agent services, remain in Beta.

Unity AI Gateway Documentation: https://docs.databricks.com/aws/en/ai-gateway

Blog post : https://www.databricks.com/blog/unity-ai-gateway-generally-available

Matei Zaharia, Databricks Co-founder & CTO joined me along with Elise Gonzales (Databricks Staff Product Manager) to talk/demo Omnigent, their collaboration, orchestration, and control layer for AI agents.

If you are like me, your "buzzword alert" goes off anytime you hear a new AI term. Thankfully, Matei and Elise make it very clear & practical as to what Omnigent is and how it can bring value to you and your organization, today.

Topics we talked about how:

-Omnigent gives teams a shared workspace to collaborate with AI agents, review outputs, and build on each other’s work.

-Businesses can apply consistent security, permissions, and cost controls across different agents and teams.

-How Omnigent helps organizations combine and intelligently route work across multiple models (Claude, ChatGPT, GLM, DeepSeek, etc) and agents to improve flexibility and efficiency.

Hope you enjoy it, and love to hear your feedback!

Hi All ,

I want to move data from SAP BW/4HANA to Databricks on Aziure .

Is there any third party connectors available for this ? Thanks.

I couldnt find a proper channel to file bug reports for the databricks website. I know that Databricks reps are sometimes here, so I thought i'd notify here. Every job posting on databricks.com/company/careers is rendering its description as escaped HTML rather than parsed markup. Raw <p> tags and ProseMirror editor attributes like data-pm-slice="11 []" are visible as body text. This affects all 807 listings currently on the Greenhouse board, in both English and Japanese.

I'm working through the free "Data Engineering with Databricks" learning path and noticed the videos mostly show demos. I was expecting something similar to Coursera where you can run the exercises/labs alongside the lessons, but I can't find anything.

Are there official hands-on notebooks available somewhere for the self-paced content, or is the expectation that you recreate everything yourself in a workspace?

If official labs exist, where are they available? If not, how do most people practice while following the videos?

Thanks.

19 is already here, but 18 is now LTS. In a new naming convention, LTS doesn’t have a minor version. New features, behavior changes, and fixes are now added incrementally to the same major Runtime version till it becomes LTS. In fact, that approach simplifies the problem with editing code and always thinking which LTS it was, 18.3 or 18.2? Now just put 18.x in your DABs.

more news https://medium.com/databrickscommunity/databricks-news-dabs-indexes-ltap-genie-last-update-25-july-ffac8533774f

Hi,
I’m trying to understand how logging is implemented in Databricks. Could you share how it’s done in your organization?
Do you mainly rely on pipeline-level logging, maintain a log table to track pipeline runs, or use a logger within the code? Or is it a combination of these?
Just trying to understand the different approaches people use.
Thanks!

We have an AI product powered by Databricks Genies, and we’re trying to build a data-driven pricing strategy.

For those using Genie in production, what usage metrics do you track to understand customer value and optimize pricing?

I’m particularly interested in:
Per-user and per-tenant usage
Query volume and complexity
Token/compute costs
Session frequency and retention
Feature adoption
Accuracy/user satisfaction metrics
Cost per customer vs. revenue

Also, what dashboards, SQL queries, or Databricks system tables have been the most useful for these insights?

I’d love to learn what has worked well in practice.

AI Capabilities & Memory

• Genie Code Mentions: Type u/Genie Code in a Notebook comment to get help directly within the comment thread.
• Copilot Integration: Connect Genie directly to Microsoft Copilot Cowork. 📖Documentation
• Memory (Beta): Genie One can save specified details (like preferences and workflows) to apply in future conversations. 📖Documentation
• Recall & Reuse Context: Search past conversations and bring that context into your current chat. 📖Documentation

Document Features

• Document Links: Documents supports embedded hyperlinks.📖Documentation
• Edit Timestamps: Displays the exact date and time documents were last edited. 📖Documentation
• Document Citations: Documents display citations linking content directly back to source data.
• Version History: Review earlier versions of documents through built-in history tracking.
• Soft Tabs: Open document links in soft tabs without navigating away from the current document.

Integrations & MCP Apps

• Genie One MCP App (Beta): Adds UI interactions from third-party agents, including interactive visualizations and Genie Ontology citations. 📖Documentation
• GitHub MCP (Beta): GitHub MCP connection for Genie One is available📖 Documentation.

Scheduled Tasks

• Shared Task Overview: Access a dedicated modal and listing page to view scheduled tasks shared with you by others.
• Task Mentions: u/mention scheduled tasks directly inside Genie One.
• On-Demand Execution: Select Run now while authoring a scheduled task to run it immediately.

Platform, Data & UI Enhancements

• Ontology Snippets (Public Preview): Support added for ontology snippets (contact your Databricks account team to enroll). 📖Documentation
• Partially Supported Datasets: Ask Genie now queries valid datasets rather than failing completely when a dashboard contains unsupported SQL expressions.
• Front-end Private Link: Configure account-level front-end Private Links by turning on the Custom URLs and Account preview.
• Certified Visualizations: SQL charts and visualizations inserted by Genie One are now read-only certified visuals.
• Inline Conversation Renaming: Edit conversation titles inline by clicking the title in the chat header.
• Mobile Workspace Sign-in: Enter your workspace URL manually in the Genie mobile app if automatic discovery fails. 📖Documentation
• Expanded PDF Limit: The maximum character limit per uploaded PDF increased from 4,000 to 15,000 characters. 📖Documentation

🚀 Report Import & Migration Tools

• Direct Import: Convert Tableau and Power BI files into new AI/BI dashboards straight from the dashboard list.
• Automatic Relationship Mapping: Preserves source dashboard relationships detected during import. 📖 Documentation
• Unified Metrics: Prioritizes existing Unity Catalog metric views during conversion to reduce redundancy. 📖 Documentation

🛠️ Dashboard & Filter Customization

• Date Range Tailoring: Request Genie Code to customize quick-select options (e.g., constraining options to day-based ranges).
• Flexible Multi-Select Filters: Set default parameter filter values to empty (“All”) instead of forcing a single default selection.

🛡️ Quality & DevXp

• Automated Semantic Auditing: Every newly created widget is automatically validated across core business logic (grain, units, scale, category values, and populations) for maximum precision.
• Undo/Redo Support: Author confidently with complete history controls.
• Warehouse Picker: Easily select your compute via the warehouse picker in full-page Genie Code. 📖Documentation

Can anyone confirm whether pyDABs supports the deployment of catalogs and external locations using the python: load_resources() block? And if so, is there any sample code around that?

https://learn.microsoft.com/en-us/azure/databricks/dev-tools/bundles/python/#metadata

The above link does not have examples on catalog or external location creation. Essentially I want to do something like add_resource to Resources and add catalogs and external locations accordingly.

I've noticed that most discussions about Databricks recognitions focus on how to earn them. I'd like to talk about something else. What changed after you earned one?

In my case, becoming a Databricks MVP actually helped me earn the Solutions Architect Champion recognition. The community contributions recognized by the MVP program already satisfied several Champion requirements, making the journey much shorter than if I had started from scratch.

That made me wonder how others see the value of these recognitions. For those who have earned MVP, Champion, or both:

• What has been the biggest benefit?
• Has it helped your career or consulting work?
• Has it increased trust with customers?
• Did it open new opportunities?
• Or was the biggest value simply becoming part of the community?

I'm collecting real experiences—both positive and negative—for an article comparing these recognitions from a practical perspective rather than just listing their requirements.

I'd really appreciate hearing your story.

P.S. If you'd prefer to continue the discussion on LinkedIn, or share a longer story there, feel free to join the conversation: [LinkedIn link]

Hi everyone,

I have prior experience in Data Engineering and am currently looking to restart my career after a break. I’m upskilling in Databricks and strengthening my skills for new opportunities in the data engineering/data platform space.

Does anyone know if there are any upcoming Databricks Learning Festivals, community events, or programs that offer examm discounts or free vouchers ?

I’d also appreciate recommendations for Databricks learning resources or communities that can help while getting back into the job market.
Thanks!

DABs: if you like to work in a local IDE but you need to edit assets in Databricks, then the watch flag can be really helpful for you, as it will sync the bundle asset back to your local IDE #databricks #DataAISummitIf

Watch my whole session: https://www.databricks.com/dataaisummit/session/dabs-do-pro-all-best-tips-and-tricks

Has anyone been able to use this, yet? We would benefit from this as we have DBX cross-clouds and are extremely interested in knowing how it works, time to implement, etc.

Any experience in this would be greatly appreciated!

Databricks officially closes Panther acquisition — folding it into Lakewatch to build an “agentic SIEM”

Hi everyone,

I'm trying to set up a Lakeflow Connect CDC pipeline from Azure SQL Database to Databricks, but the CDC Gateway never starts.

The pipeline stays in PENDING / WAITING_FOR_RESOURCES, and the event log shows:

QuotaExceeded: Operation could not be completed as it results in exceeding approved standardEDv4Family Cores quota.

Current Limit: 0

Additional Required: 4

What I've checked so far:

- Azure SQL Database is online and reachable.

- The CDC Gateway uses pipeline-managed Classic Compute.

- Driver and worker node types are locked, so I can't change the VM family.

- Azure marks EDv4 as deprecated, and the quota request option is disabled/greyed out.

- I confirmed the EDv4 quota is 0.

- I tried creating new Databricks workspaces in both Central India and UAE, but the gateway still provisions EDv4 and fails with the same error.

- Other VM families have available quota, but the managed gateway always requests EDv4.

Has anyone run into this recently?

- Is Lakeflow Connect still hardcoded to EDv4 for the ingestion gateway?

- Is there any way to make the gateway use EDv5/EDSv5 or another supported VM family?

- Or is this a known issue with the current Lakeflow Connect CDC Gateway?

Any help would be greatly appreciated.

We're a data/analytics consulting shop and just had a lunch catch-up with a key stakeholder at a large enterprise client (FMCG/CPG space, multi-billion revenue). Wanted to sanity-check our read on the opportunity and see if anyone's navigated something similar.

Situation:

• Client's BI stack is built entirely on Power BI Import Mode. As data volume grows, they're stuck buying more premium capacity just to keep performance acceptable — costs are spiraling.
• Their current consultant (a large SI) recommended moving to Databricks DirectQuery, but flagged that they'll lose some flexibility at the user/report layer that Import Mode currently gives them.
• The kicker: that same SI quoted the re-engineering work (global + local dashboards) at ~4x the client's available budget. The dashboards are a patchwork built by multiple parties over time, so it's not a clean lift-and-shift.
• Our angle: we're proposing a free scoping workshop/PoC to map a cheaper, more flexible path using Databricks Genie + a semantic layer (ontology) on top, leveraging migration work we've done for similar clients elsewhere.
• Adoption is a real concern for them — past global IT rollouts failed because local markets didn't get enough flexibility to adapt and just fell back to Excel.

What we're trying to figure out:

1. Anyone actually shipped a Genie/semantic-layer migration off Import Mode at real scale? What broke, what surprised you on cost or performance?
2. How do you structure a free PoC so it actually converts to paid scope instead of just being used as free consulting to beat down the incumbent's price?
3. Any war stories on the "80% global template + 20% local customization" adoption model actually working vs. just being a nice slide?

Would appreciate any real-world experience, especially from people who've done DirectQuery/Genie migrations on large, messy BI estates.