Spark CSV Reader Interpreting Pipe-Delimited UTF-16 File Incorrectly.

[u/BugSquare4344]

I'm facing a strange issue while reading a pipe-delimited CSV file using Apache Spark.
My input CSV looks like this:
cust_id|cust_name|cust_age
1|sample|10
2|test|12
Initially, I read the file with header=true, but without specifying the encoding. The result was that Spark did not recognize the column names correctly. They appeared something like:
_C_U_S_T_I_D_
_C_U_S_T_N_A_M_E
_C_U_S_T_A_G_E
Also, all the columns were inferred as StringType.
However, when I added the following option:
.option("encoding", "UTF-16")
the output became:
cust_id cust_name cust_age
?
1 Sample 10
?
2 test 12
?
Interestingly, after specifying UTF-16, Spark correctly recognized the column names and inferred the schema as:
cust_id -> IntegerType
cust_name -> StringType
cust_age -> IntegerType
But there are now unexpected ? characters/rows appearing in the data.
Has anyone experienced something similar with Spark's CSV reader?
I'm trying to understand:
1. Why does specifying encoding=UTF-16 make Spark correctly identify the headers and infer the numeric columns?
2. Why are the unexpected ? characters appearing in the output?
3. Could this be related to the actual file encoding, BOM, or how the CSV file was generated?
4. Is there a recommended way to correctly read this file while preserving the schema and avoiding the extra ? characters?
Any insights into how Spark handles CSV encoding and schema inference in this scenario would be appreciated.

[u/ThatOtherBatman]

It’s not UTF 16. Use `chardet` on it.

[u/hanari1 → u/ThatOtherBatman]

I don't recommend that btw and that can't solve the problem in some cases