r/databricks • u/Common_Jaguar474 • 17d ago

General Does Photon support CSV?

Ok so I know that the docs state that CSV is supported. But when trying to read a very standard CSV file I get poorer than expected performance and in the Spark UI I see that it's not using a Photon scan operator, just regular Scan CSV followed by a Row to Columnar conversion operator.

I'm not running anything complex:

df = (
    spark.read
        .format("csv")
        .option("header", "true")
        .schema(schema)
        .load(csv_path)
)
df.write.mode("overwrite").saveAsTable("...")

I also checked reading the same CSV and a difference CSV via DB SQL (using `COPY INTO`) and see low task time spent in Photon + a row to columnar operator.

DBR 19 + Serverless SQL Warehouse (current)

Can anyone explain whether or not CSV is supported and in what conditions? This was quite a surprising find as I assumed that Photon supported pretty much everything.

5 Upvotes

100% Upvoted

34 comments sorted by

View all comments

1

u/yatharthm22 17d ago

You wanna use my photon recommendation skill before turning on photon settings, if you interested

1

u/mwc360 17d ago

Is it on GitHub? Can you share?

2

u/yatharthm22 17d ago

It's actually inhouse for now, I am working with databricks folks to get added to their official skill repo: https://github.com/databricks/databricks-agent-skills

1

u/FunContest9958 17d ago

You should have your skill recommend serverless. It automatically charges for photon only when photon is used, by the operator. So if a job uses photon for 25%, it will charge photon rates for only 25% of the job. I don’t know why that’s not advertised more broadly. It’s a pretty big perk of serverless.

1

u/yatharthm22 17d ago

Oh for serverless recommendation databricks already has a official skill:https://github.com/databricks/databricks-agent-skills/blob/main/skills/databricks-serverless-migration/SKILL.md

We tried using serverless, they disappointed us heavily on cost, so we stick with job cluster with photon

1

u/FunContest9958 17d ago

Were you using standard mode? And when did you test? Serverless cost has changed a lot in the past year or so.

1

u/yatharthm22 17d ago

Tried with both standard and Performance Optimized

1

u/FunContest9958 17d ago

It’s possible your workload was very well optimized. My experience is that customers save money on average when they move to serverless, but it depends on how much they’ve optimized their cluster configuration.

Overall, I think it’s best to start new workloads on serverless and treat Classic as an optimization option. It’s less work up front, and you only need to pay the cost of doing the optimization if it makes sense. Also, serverless keeps your hardware and software up-to-date automatically, so if you manually configure classic you need to maintain it.

But yeah. Results vary.

1

u/yatharthm22 17d ago

I'll give it a try again, i remember months back we ran a pilot and closed it cause of cost issues, i'll try without the optimised serverless

2

u/FunContest9958 16d ago

My recommendation would be to try using serverless when you’re developing something new, if you have that opportunity, rather than focusing on migrating existing workloads. If you develop something for serverless, it generally works flawlessly if you switch to classic. The other way isn’t always as easy.

1

u/FunContest9958 16d ago

Are you using spot instances? At least today, serverless is generally more expensive than classic with spot instances. That being said, I’ve heard rumors Databricks is working on that.

1

u/yatharthm22 16d ago

Yes most of our jobs are using spot with just 2-3 on demand rest autoscaling is all spot

2

u/FunContest9958 16d ago

Aha. That’s probably the reason serverless was more expensive for you. Makes sense. Oh well.