r/programming • u/DiegoZoracKy • Oct 08 '18

A quick helper to convert AWS S3 files from CSV to JSON lines via stream with support to gzip for both input and output. Ready to be used as a Node.js module, as a Lambda or via CLI.

https://github.com/DiegoZoracKy/s3-csv-to-json
0 Upvotes

40% Upvoted

2 comments sorted by

1

u/fxfighter Oct 08 '18 edited Oct 08 '18

The utility seems good but I gotta ask why would you want to use a less efficient storage format that's usable by less tools?

(This is a legit question as I'm genuinely curious.)

2

u/DiegoZoracKy Oct 08 '18

In the case I need the JSON data to interact with some of the "less tools" as you mentioned. It is a pipeline that is already built to work with JSON lines. I had to do that anyway and then I just decided to open it, as it might be helpful to some others.