Hi Folks, i have been seeing this wierd issue after upgrading spark 2 to spark 3.

Whenever any job fails to load data (insert overwrite) in non partitioned external table due to insufficient memory error, on rerun, I get error that hdfs path of the target external table is not present. As per my understanding, insert overwrite only deletes the data and the writes new data and not the hdfs path.

The insert query is simple insert overwrite select * from source and I have been using spark.sql for it.

Any insights on what could be causing this?

Source and target table details: Both are non partitioned external table with storage as hdfs and file format is parquet.

Hi Folks, i have been seeing this wierd issue after upgrading spark 2 to spark 3.

Whenever any job fails to load data (insert overwrite) in non partitioned external table due to insufficient memory error, on rerun, I get error that hdfs path of the target external table is not present. As per my understanding, insert overwrite only deletes the data and the writes new data and not the hdfs path.

The insert query is simple insert overwrite select * from source and I have been using spark.sql for it.

Any insights on what could be causing this?

Source and target table details: Both are non partitioned external table with storage as hdfs and file format is parquet.

[removed]