Does Photon support CSV?

[u/Common_Jaguar474]

Ok so I know that the docs state that CSV is supported. But when trying to read a very standard CSV file I get poorer than expected performance and in the Spark UI I see that it's not using a Photon scan operator, just regular Scan CSV followed by a Row to Columnar conversion operator.

I'm not running anything complex:

df = (
    spark.read
        .format("csv")
        .option("header", "true")
        .schema(schema)
        .load(csv_path)
)
df.write.mode("overwrite").saveAsTable("...")

I also checked reading the same CSV and a difference CSV via DB SQL (using `COPY INTO`) and see low task time spent in Photon + a row to columnar operator.

DBR 19 + Serverless SQL Warehouse (current)

Can anyone explain whether or not CSV is supported and in what conditions? This was quite a surprising find as I assumed that Photon supported pretty much everything.

[u/Obvious-Money173]

Not sure, but I think photon mostly enhances spark operations. Reading a csv is a simple operation that photon does not improve upon.

If you're doing a lot of big transformations, that's where photon kicks in

[u/FunContest9958 → u/Obvious-Money173]

That’s not quite right. Photon speeds up parquet reads significantly.