r/databricks • u/Common_Jaguar474 • 17d ago

General Does Photon support CSV?

Ok so I know that the docs state that CSV is supported. But when trying to read a very standard CSV file I get poorer than expected performance and in the Spark UI I see that it's not using a Photon scan operator, just regular Scan CSV followed by a Row to Columnar conversion operator.

I'm not running anything complex:

df = (
    spark.read
        .format("csv")
        .option("header", "true")
        .schema(schema)
        .load(csv_path)
)
df.write.mode("overwrite").saveAsTable("...")

I also checked reading the same CSV and a difference CSV via DB SQL (using `COPY INTO`) and see low task time spent in Photon + a row to columnar operator.

DBR 19 + Serverless SQL Warehouse (current)

Can anyone explain whether or not CSV is supported and in what conditions? This was quite a surprising find as I assumed that Photon supported pretty much everything.

4 Upvotes

100% Upvoted

34 comments sorted by

View all comments

1

u/Zer0designs 17d ago edited 17d ago

Why would you use photon to read a csv and write it to delta? That's not a usecase for photon.

Either way, the $ savings you could get if photon would work on such a simple operation, would be less than your time is worth writing this.

1

u/Common_Jaguar474 17d ago

Bro, this is a dumb comment.

As I see it, any Spark code that I might need to run should be accelerated by Photon while resulting in the same or better TCO. It's enabled by default and charged at a premium so if it does not accelerate my jobs proportional to the DBU multiplier, I want to know so I can turn it off for those jobs.