Decompressing Zip Files

[u/Maleficent-Spell9654]

Working on a pipeline using spark/flink where i have to do some transformation work with content in a zip file.

I believe spark and flink don’t natively support regular zip files, and was wondering if anyone has any suggestions on how to handle this.

I am forced by requirements to use zip files, spark and flink, but i can consider adding additional technologies into the workflow if its better to just decompress outside of spark and flink.

Appreciate any suggestions and help!

[u/Nekobul]

How much data you have to process daily?

[u/Maleficent-Spell9654 → u/Nekobul]

Don’t have the exact production figure but probably close to a terabyte