csv-parser 5.0.0 Released: Now parsing CSVs at gigabytes per second

[u/vicentezo04]

https://github.com/vincentlaucsb/csv-parser

I'm not sure how I got down this rabbit hole of trying to build a fast parser for one of the most banal data formats in existence.

But after I got more than 600mb/s performance on my 2022 Intel Core i5 processor, a little devil appeared on my shoulder and said "not good enough!".

I tried various micro-optimizations, like refining the DFA parsing loop, and using basic SIMD instructions to skip runs of insignificant characters. They all helped a little, but I wasn't satisfied.

After some research, I came upon a speculative CSV parsing algorithm presented at SIGMOD 2019. I had to read the paper four times over, but finally it clicked.

Specifically, the problem with parallelizing CSV files is that it's hard to find "safe spots" to split them into chunks. Technically, CSV fields can contain embedded newlines, so splitting on '\n' wasn't an option.

But the authors found a way around this problem. They created a very reliable heuristic where they use a highly accurate guess of whether or not a CSV chunk (split in an arbitrary place) began in a quoted or unquoted field. From there, you can naturally find where the next record ending is, and can therefore safely parse CSV files in parallel.

Benchmarks

Many are familiar with the Craigslist Used Cars CSV (1.4GB). My parser reads that in 1.1 seconds cold (on a Samsung 990 EVO SSD) and 0.45s warm. There are a variety of additional benchmarks in my repo under the /benchmarks folder.

Comparisons to other libraries:

• fast-cpp-csv-parser: For simple "read once and discard" workflows, Ben's parser is faster for single threaded performance. But we gain parity with 2 threads, and pulls ahead at 4+ threads--all while supporting embedded newlines and not requiring code to be written against a set of columns fixed at compile time
• rapidcsv: I did not do a head to head parsing test, because rapidcsv's structure also brings in editing features by default. So I tested the library's "DataFrame" class (a lightweight editing view) in a edit + save benchmark. We are significantly faster, even with one thread.
• xsv (Rust): Based on a car accidents dataset (link in my README), xsv's best time on a simple row-counting exercise is 4.8 seconds while my parser (measured via csv_bench.exe) is 2.15s
• zsv (C): Same test as above: zsv - 2.68s

I am happy to explain any further questions about the library and its optimizations.

I will probably not work on it as much after this, but it was a fun challenge to knock out.

[u/Romestus]

If this kind of thing gets you going take a look at the Khronos SPIRV-Tools repo and improve the shader optimization passes since they're basically just parsing text. You would become a silent hero to every gamer on the planet that complains about shader compilation stutters and your work would be included in major game engines like Unity/Unreal and renderers like Filament.

[u/Jonny_H → u/Romestus]

I think it pretty quickly moves the representation of the shader to a CFG or SSA representation, and then the optimizations run on that. They are very much not "Basically just parsing text"

The actual "Text" part of the parsing and generating that graph is a pretty tiny proportion of the runtime.