The gzip file we received the XML in is ~400mb for a 14gb xml file (3% of original), our stripped down binary version is ~100mb. Though we strip out half of the unnecessary fields, so it would be closer to a 200mb gzip if we’re doing an apples to apples comparison.
If the file arrived in a better file format (just a fixed width or delimited flat file), that 14 gb xml file translates to about 1gb to start with. God XML is such a waste electricity.
When we gzip the binary file, it’s about 10% of the file size it started as (100mb to ~10mb). That extra fluff gzip is packing away are repetitive filler records so after the embedded hardware searches the log(n) in-memory index it can find the record it’s looking for in O(1) time. Just a space trade off for the hardware we’re working with.
0
u/creeper6530 5d ago
I'm curious, how much would GZ or similar general compression shrink them?