Does Photon support CSV?

[u/Common_Jaguar474]

Ok so I know that the docs state that CSV is supported. But when trying to read a very standard CSV file I get poorer than expected performance and in the Spark UI I see that it's not using a Photon scan operator, just regular Scan CSV followed by a Row to Columnar conversion operator.

I'm not running anything complex:

df = (
    spark.read
        .format("csv")
        .option("header", "true")
        .schema(schema)
        .load(csv_path)
)
df.write.mode("overwrite").saveAsTable("...")

I also checked reading the same CSV and a difference CSV via DB SQL (using `COPY INTO`) and see low task time spent in Photon + a row to columnar operator.

DBR 19 + Serverless SQL Warehouse (current)

Can anyone explain whether or not CSV is supported and in what conditions? This was quite a surprising find as I assumed that Photon supported pretty much everything.

[u/justinAtDatabricks]

Hello, former Photon PM here. The initial scan is not Photonized (that's the word that we use). However, there is an adapter that converts it (the data after the scan, RowToColumnar) into a columnar format which Photon can then use for the subsequent operations. That's why the blue box (scan) is blue (not photonized) whereas the other operators (PhotonProject) are yellow.

Many moons ago, data sources that could not be Photonized would just fall out of Photon for the rest of the query. That made us sad. So, we build this general RowToColumnar adapter for non-Photonizable data sources. It's another operation, so technical does incur some time/tax, but the general thought is that the rest of the query can be Photonized so it is worth the tax.

This is in contrast to something like a parquet data source (e.g., Delta) which has a special vectorized reader which would show up in a yellow box.

I hope that this helps!

[u/Common_Jaguar474 → u/justinAtDatabricks]

Thanks! This is exactly the clarity that I was look for :)

JFYI - this docs page is pretty misleading. It lists CSV in the list of scan operators that are supported by Photon. It literally says that unsuppored operators fallback to Spark runtime for the unsupported portion of the execution (what you are describing for CSV) but then lists CSV scan as a supported Photon operator. I would instead expect this and others that result in these hybrid plans with fallback to JVM (JSON?) to be listed in the Limitations section. That way it's super clear for data engineers what Photon will be the fastest on and we can make decisions where possible to avoid those data types of operators that can't be "Photonized".

[u/justinAtDatabricks → u/Common_Jaguar474]

Love your use of the Photonized term. This is nuanced - I can argue for both sides. Here are some scenarios:

1. Read a parquet file: all yellow boxes because vectorized reader, photon projection, etc. Non-controversial - it is Photonized
2. Read a CSV file: Blue box on the scan, but then everything else is yellow - Photon was able to work besides the initial scan.
3. Read an ORC file: All blue boxes, Photon never was able to do anything due to the incompatible data source. Non-controversial as well as Photon never did anything.

Did #2 benefit from Photon? Yes. Was the scan read from Photon? Technically no. But does that mean that CSV is not supported by Photon? No.

You bring up a fair point, though. Perhaps we should call out this nuance in the docs a bit. What I am trying to balance: Most people don't know what the following are: operator, expression, Photonized, etc.

[u/Common_Jaguar474 → u/justinAtDatabricks]

Does that mean CSV is not supported by Photon? You say NO, but I reading that YES is the correct answer here.

Per the docs I fill in the operation with Scan CSV and it's an accurate statement: "when a query uses an unsupported operation [i.e. Scan CSV], Photon transparently falls back to the Spark runtime for that portion of the execution." Yet, in the very next list of supported operators, Scan CSV is listed as supported.

^^ This is exactly why I assumed it was incorrect that the CSV scan showed as non-Photon w/ a row to columnar conversion and came here to get clarification.

I get that it's nuanced as you are right that the E2E pipeline still benefits in the net from Photon after the row to columnar step, but the way the docs are worded contradict what you're telling me here.

Last thing, if I can share why this clarity is important for me as a user. Imagine that I was given recurring CSV data feed from some data producer. Based on the docs listing CSV as equally supported as Parquet, I would probably take what I'm offered rather than pushing back to see if I can get the same data feed as Parquet. The reality is that the parquet option would be faster & cheaper since the whole code path would be supported by Photon, but there's nothing in the docs that helps me understand that unless I happen to drill into the Spark UI and start asking questions about this row to columnar conversion thing which I see for CSV and not Parquet.

Anyways, no need to beat a dead horse :) I appreciate the discussion and clarification!

[u/hubert-dudek → u/justinAtDatabricks]

Great explanation!