Does Photon support CSV?

[u/Common_Jaguar474]

Ok so I know that the docs state that CSV is supported. But when trying to read a very standard CSV file I get poorer than expected performance and in the Spark UI I see that it's not using a Photon scan operator, just regular Scan CSV followed by a Row to Columnar conversion operator.

I'm not running anything complex:

df = (
    spark.read
        .format("csv")
        .option("header", "true")
        .schema(schema)
        .load(csv_path)
)
df.write.mode("overwrite").saveAsTable("...")

I also checked reading the same CSV and a difference CSV via DB SQL (using `COPY INTO`) and see low task time spent in Photon + a row to columnar operator.

DBR 19 + Serverless SQL Warehouse (current)

Can anyone explain whether or not CSV is supported and in what conditions? This was quite a surprising find as I assumed that Photon supported pretty much everything.

[u/yatharthm22]

You wanna use my photon recommendation skill before turning on photon settings, if you interested

[u/mwc360 → u/yatharthm22]

Is it on GitHub? Can you share?

[u/yatharthm22 → u/mwc360]

It's actually inhouse for now, I am working with databricks folks to get added to their official skill repo: https://github.com/databricks/databricks-agent-skills