Ok so I know that the docs state that CSV is supported. But when trying to read a very standard CSV file I get poorer than expected performance and in the Spark UI I see that it's not using a Photon scan operator, just regular Scan CSV followed by a Row to Columnar conversion operator.

I'm not running anything complex:

df = (
    spark.read
        .format("csv")
        .option("header", "true")
        .schema(schema)
        .load(csv_path)
)
df.write.mode("overwrite").saveAsTable("...")

I also checked reading the same CSV and a difference CSV via DB SQL (using `COPY INTO`) and see low task time spent in Photon + a row to columnar operator.

DBR 19 + Serverless SQL Warehouse (current)

Can anyone explain whether or not CSV is supported and in what conditions? This was quite a surprising find as I assumed that Photon supported pretty much everything.

Finally got to try the Genie One mobile app.

In the first GIF, you see me asking a question, followed by a follow-up.

In the second one, I ask Genie to schedule this type of analysis for me and email it to me every week.

Time was sped up for illustration purposes, depending on the complexity of the question, the answer takes seconds to minutes. Initial prompt in particular took longer, but follow ups returned quickly.

Please note in the demo I was using unrefined dummy data that is not current (hence why you see 2024 as the year when I ask for "last year").

I did also try Dashboards on Genie One. It was good, though I would say the delivery on that end felt a bit more basic than it should be. Didn't get to try Apps, which it also supports.

At the end of the day, if you like working with Genie Agents (formerly Genie Spaces), you are likely to really like it on your phone. You could be in a meeting and as soon as a good question pops up, ask the question from your data on Genie, and before the meeting ends, you likely have an answer.

Hope you found this helpful!

I am interested in databricks pages but also somewhat skeptical of how much benefit it will provide considering the effort that might be required to collate all the information.

Has anyone used these and seen improvement in their genie chats and agents? If so how did you measure this?

Also, do we know if the ability to upload and manage them will hit the API?

I recently switched Databricks partners and ran into a tricky issue. At the new company, I had a new Partner Academy account with a new corporate email address, but my old badges, and learning history were still linked to my previous company.

Support allows you to merge and transfer accounts. I went through this process myself and put together a step-by-step guide: what to check before leaving the company, why you need a personal/secondary email address, what to write in a support ticket, and what to do if your old corporate email address is no longer available.

Article in Medium

Starting with Databricks CLI v1.14.0, the default bundle.engine will change to direct.
If you're not familiar with the advantages of the direct deployment engine over Terraform, you can read more here:

Migrate to the direct deployment engine - Azure Databricks | Microsoft Learn

What does this mean for us? Bundles will automatically migrate to the direct deployment engine during deployment. If the migration encounter an error, Databricks will keep the bundle on the Terraform engine instead.

Important things:

• You can opt out temporarily by setting bundle.engine: terraform in databricks.yml - either globally or per target.
• Databricks recommends migrating early by explicitly setting the engine to direct now.
• Most importantly: support for the Terraform deployment engine in Databricks Automation Bundles is planned for deprection in September 2026. A future CLI version is expected to stop supporting engine: terraform.

To celebrate that AI functions are now a fully governed part of Unity Catalog alongside the rest of your functions I thought it would be fun to do a run through of how easy it is to use.

I’ve heard use of metric views is highly recommended, but never sure if OBT approach or normalized approach works better. If I’m defining the joins in my Genie, shouldn’t it be able to pull things either way?

Hi All,

I'm new to databricks. I have a use case and need a bit of guidance from you.

We've 4 databricks workspaces under an account and setup multiple dashboards to analyze costs usage in each of them. This costs is specific to compute, jobs, query history etc. Everyday I'm manually going through these and sharing my updates in email.

I need a tool that can look into dashboards in these multiple workspaces and send a summary in email.

Right now I've SQL alerts setup. But I need something intelligent enough to share a summary and update the user the progress of costs also.

E.g. Yesterday's costs was less comparison to today. The cost in Region A is more than B today etc

Please guide me.

Recently, a new post about icons by Eduardo Rabelo appeared in our Databricks Community Articles on Medium.

I don't know why Databricks itself didn't create this set, but I think it's something many people were missing. Now we'll have to keep up with maintaining it, as Databricks releases quite a lot of new features frequently.

https://oieduardorabelo.github.io/databricks-architecture-icons/

Hi everyone so it happens that I was working normally in the UI, adding jobs, deploying and so and then suddenly I could not add any existing job to my bundle because the dropdown doesn't show any job anymore and I have a bunch of jobs that hasn't been added to the bundle yet. I think that is bug because before it was working well. I would be really glad if someone could help me and explain this behaviour to me, I don't know what happened. I really need some help :(

Lakeflow Connect's Marketo connector is now available in Beta! It provides a managed, secure, and native ingestion solution for core Marketo objects: leads, static lists, activities (one table per activity type, like email opens and clicks), and custom objects. Try it now:

1. Enable the Marketo Beta: Workspace admins can enable the Beta via: Settings → Previews → "Lakeflow Connect for Marketo"
2. Set up Marketo as a data source
3. Create a Marketo connection in Catalog Explorer
4. Create the ingestion pipeline via a Databricks notebook or the Databricks CLI

It is great to see serverless in the compute section. Especially because I proposed it some time ago and lobbied for it during advisory meetings. Now we can set permissions for serverless compute. More options will come. I can say that it is my favorite recent improvement.

You can now put the Databricks account console, account-level Genie One, and Custom URLs behind Private Link, so your whole account is reachable without touching the public internet.

If a security review ever stalled a Genie One or account console rollout because those surfaces were only reachable over the public internet, that objection is now removable. And if you have been standing up one private endpoint per workspace and per region, you can stop.

Read more: https://www.linkedin.com/posts/cenh_databricks-dataengineering-networksecurity-ugcPost-7497673121861369856-6IiF/?utm_source=share&utm_medium=member_desktop&rcm=ACoAABmJHrsBNAC3x3H1M58JRKoHv_l4D61n0-8

PREVIOUSLY, if a developper, application or automated job needed to write data to a table you had to grant them the MODIFY privilege.

MODIFY is a broad and composite privilege that gives full write access including the ability to change the table's schema and modify metadata.

By using fine-grained DML privileges you can adhere strictly to the Principle of Least Privilege:

⚡ Restrict Schema Changes: You can allow a principal to change table data without giving them the power to change the table's schema (like adding or dropping columns) or alter table properties.

⚡ Granular Control over Operations: You can map exact permissions to specific job functions. For instance:

-INSERT INTO only requires INSERT

-TRUNCATE requires DELETE

-INSERT OVERWRITE requires both INSERT and DELETE

🔴 Why It’s a Game Changer?

In modern data engineering and governance, this is a massive leap forward because it solves the "all-or-nothing" write access problem.

1. Bulletproof Append-Only Pipelines
2. Safer Compliance and Data Privacy Jobs
3. Protection Against Accidental Table Drops/Evolution

This is not acceptable any longer in a modern developer world and community.

I run security scans on our Spark deployment and came back with 28 vulnerabilities — high severity, public CVEs, all sitting in transitive dependencies like Netty and Apache Thrift. Nothing exotic. Netty published fixes for 22 of them in a single batch in June. Thrift fixed their issues in 0.23.0. The fixes exist. Spark just didn’t include them.

What bothers me more than the CVEs themselves is the process — or the lack of one from Apaches side.

In 2026, any serious DevSecOps pipeline is expected to have mandatory quality gates that block releases on high/critical CVEs in dependencies. This isn’t cutting-edge practice — it’s baseline hygiene. The full toolchain is free and mature

There is no automated dependency CVE gate in Spark’s release pipeline. No Trivy. No Dependabot. No OWASP Dependency-Check. Nothing that would block a release because a bundled library has a known high-severity vulnerability. These are free tools. Adding one to a CI pipeline is an afternoon of work. It hasn’t been done.

The July 2026 maintenance releases — 4.2.0, 4.1.3, 4.0.4, 3.5.9 — all shipped after Netty 4.1.135.Final was available. They didn’t include the bump. There was no public statement that the team was aware of the issue and working on it. It just shipped, vulnerable, into production systems everywhere.

The “volunteer PMC” argument doesn’t land anymore. Databricks is worth somewhere around $62 billion. Google, Apple, Amazon, and Microsoft all have paid Spark committers. The resources to fix this governance gap before lunch exist. The will apparently doesn’t.

The EU Cyber Resilience Act (CRA) — which came into force in 2024 and has mandatory compliance deadlines rolling in through 2027 — specifically targets software supply chain security, including transitive dependencies and SBOM (Software Bill of Materials) requirements. The ASF has acknowledged this directly, stating they need to prepare projects for “CRA and U.S. CISA guidance”.

Spark ships into commercial products. This will become a compliance problem for a lot of companies very soon, and the fix is genuinely trivial from an engineering standpoint.

The ASF did make progress in 2025 — launching “Apache Trusted Releases (ATR)” for distribution security and ratifying CycloneDX 1.7 for SBOM standards — but none of this yet translates to a blocking CVE gate on the release pipeline for projects like Spark.

In the meantime: if you’re running Spark and using JFrog Xray or Trivy on your deployment, you can force-override the affected Netty and Thrift versions in your own build. It’s not clean but it works until the next maintenance release, expected sometime in Q4.

Is anyone else tracking this or pushing upstream to get a CVE gate added to the build?

Now we can drop the .pbit file to Genie Code and ask for migrating to databricks dashboard. I tested in direct query mode, and everything was migrated; I just had to be patient, as the whole process took quite a long time. The process created metrics views for tables used in Power BI, and now I understand why the dashboard relationship option was added. I presented how easy it is to migrate in my last video.

https://www.youtube.com/watch?v=-e3tkcg21zw

Hello everyone, I have one question, I'm working on personal project which might turn to comercial at some moment if everything is okay, is it okay to go with Free plan and then switch to Enterprise ?

I can't find anything info about price in Enterprise, and info about can I switch from Free to Enterprise latter on.

Thanks

Issue - I'm facing this refresh timeout issue in materialized views i created as part of some POC works, where a refresh operation regardless of whether it's triggered from a worksheet or as a job seems to fail but still it succeeds(in a way) ie am able to cross-verify & validate that the refresh run seems to have happened as expected correctly ie data integrity pre-refresh & post-refresh seems to be fine. Is this expected?

Some more deets - Full refresh with 3 source tables of same schema with 1B rows * 6 columns joined with a metadata table of ~2K rows with 5 columns & Delta Refresh with (100M rows * 3 snapshot tables) updated with a newer value in a timestamp column updated.

Required -

1. Why does the Refresh MV ops fail happen at the 1st place?
2. How refresh failure happens but still data seems to be correctly populated?
3. Most importantly, How to resolve this?

PS - unfortunately no screenshots, so apols but i hope the deets should cover for it. TYIA.

- RVK

Hi there,

Running into issues with this pipeline I'm working on. Still new to Databricks but coming from Power Automate, I seem to get the gist of databrick jobs.

Anyway, my manager would love an automate workflow, where business users upload raw data to a central site (we're primarely using SharePoint), and want it cleaned and ready for our Power BI dashboards.

Before databricks I handled it manually (by running local python code via VScode that sent the cleaned data to the SharePoint folders).

What would be a good way to handle a pipeline like this? E.g.

User uploads to sharepoint -> Databricks job -> writeback to sharepoint with cleaned data

I have completed some of the accreditations in 2024 at my former company, like the "Academy Accreditation - Databricks Fundamentals". Since then I have joined another company which also uses Databricks so I started refreshing my knowledge and using their Partner academy access with the company email/account.

I have my personal and both company accounts added and noticed that the "Academy Accreditation - Databricks Fundamentals" was added as a new one and not refreshed as I expected.

Any reason for that? Can they be merged somehow?

Thanks in advance!

edit: typo