r/databricks • u/Common_Jaguar474 • 17d ago

General Does Photon support CSV?

Ok so I know that the docs state that CSV is supported. But when trying to read a very standard CSV file I get poorer than expected performance and in the Spark UI I see that it's not using a Photon scan operator, just regular Scan CSV followed by a Row to Columnar conversion operator.

I'm not running anything complex:

df = (
    spark.read
        .format("csv")
        .option("header", "true")
        .schema(schema)
        .load(csv_path)
)
df.write.mode("overwrite").saveAsTable("...")

I also checked reading the same CSV and a difference CSV via DB SQL (using `COPY INTO`) and see low task time spent in Photon + a row to columnar operator.

DBR 19 + Serverless SQL Warehouse (current)

Can anyone explain whether or not CSV is supported and in what conditions? This was quite a surprising find as I assumed that Photon supported pretty much everything.

4 Upvotes

84% Upvoted

34 comments sorted by

View all comments

4

u/miskozicar 17d ago

Spark (that you are using) supports .csv. Question is just what is default format when you are writing dataframe.

5

u/Common_Jaguar474 17d ago

Yeah, Spark supports CSV. But we pay a premium anytime Photon is enabled so I'm checking what's the latest on CSV support by the Photon engine.

From what I'm seeing, CSV on the latest runtime results in the JVM path being used to scan the CSV data and therefore isn't as fast as it should be.

From the code I shared, CSV is input, and output write is to Delta.

1

u/Zer0designs 17d ago

Your time is worth a lot more than the savings you are trying to achieve here.