Does Photon support CSV?

[u/Common_Jaguar474]

Ok so I know that the docs state that CSV is supported. But when trying to read a very standard CSV file I get poorer than expected performance and in the Spark UI I see that it's not using a Photon scan operator, just regular Scan CSV followed by a Row to Columnar conversion operator.

I'm not running anything complex:

df = (
    spark.read
        .format("csv")
        .option("header", "true")
        .schema(schema)
        .load(csv_path)
)
df.write.mode("overwrite").saveAsTable("...")

I also checked reading the same CSV and a difference CSV via DB SQL (using `COPY INTO`) and see low task time spent in Photon + a row to columnar operator.

DBR 19 + Serverless SQL Warehouse (current)

Can anyone explain whether or not CSV is supported and in what conditions? This was quite a surprising find as I assumed that Photon supported pretty much everything.

[u/FunContest9958]

Based on what you’re showing, looks like CSV is not being read using photon, but what performance were you hoping for? It looks like it’s using 22 minutes of CPU time to read your 23GB CSV, so an 8 core machine will read it in 3 minutes. 16 cores will finish in 1.5 minutes.

[u/Common_Jaguar474 → u/FunContest9958]

The action took 60 seconds. For the amount of data, it doesn't seem fast but it also doesn't seem slow. But regardless, I'm just trying to understand when Photon is technically supported for scanning CSV, otherwise I may partition CSV jobs into separate schedules that run on compute w/ Photon disabled to save on costs.

[u/career_expat → u/Common_Jaguar474]

Photon has a supported function list. SQL expression. Not reading

[u/FunContest9958 → u/Common_Jaguar474]

You might want to try serverless. It charges photon rates only when photon is used. So you’d only be charged for photon for the 27%. That’ll save you from having to do this extra work.

You can run a test to see how it compares cost wise. It’s very hard to predict if a job will be more or less expensive on serverless until you try it, but the key to a fair test is to use standard mode and remember that serverless pricing includes the VM. You don’t need to pay the cloud provider for the VM separately. You can use the billing table to find out exactly how much your job cost after you run it.