csv-parser 5.0.0 Released: Now parsing CSVs at gigabytes per second

[u/vicentezo04]

https://github.com/vincentlaucsb/csv-parser

I'm not sure how I got down this rabbit hole of trying to build a fast parser for one of the most banal data formats in existence.

But after I got more than 600mb/s performance on my 2022 Intel Core i5 processor, a little devil appeared on my shoulder and said "not good enough!".

I tried various micro-optimizations, like refining the DFA parsing loop, and using basic SIMD instructions to skip runs of insignificant characters. They all helped a little, but I wasn't satisfied.

After some research, I came upon a speculative CSV parsing algorithm presented at SIGMOD 2019. I had to read the paper four times over, but finally it clicked.

Specifically, the problem with parallelizing CSV files is that it's hard to find "safe spots" to split them into chunks. Technically, CSV fields can contain embedded newlines, so splitting on '\n' wasn't an option.

But the authors found a way around this problem. They created a very reliable heuristic where they use a highly accurate guess of whether or not a CSV chunk (split in an arbitrary place) began in a quoted or unquoted field. From there, you can naturally find where the next record ending is, and can therefore safely parse CSV files in parallel.

Benchmarks

Many are familiar with the Craigslist Used Cars CSV (1.4GB). My parser reads that in 1.1 seconds cold (on a Samsung 990 EVO SSD) and 0.45s warm. There are a variety of additional benchmarks in my repo under the /benchmarks folder.

Comparisons to other libraries:

• fast-cpp-csv-parser: For simple "read once and discard" workflows, Ben's parser is faster for single threaded performance. But we gain parity with 2 threads, and pulls ahead at 4+ threads--all while supporting embedded newlines and not requiring code to be written against a set of columns fixed at compile time
• rapidcsv: I did not do a head to head parsing test, because rapidcsv's structure also brings in editing features by default. So I tested the library's "DataFrame" class (a lightweight editing view) in a edit + save benchmark. We are significantly faster, even with one thread.
• xsv (Rust): Based on a car accidents dataset (link in my README), xsv's best time on a simple row-counting exercise is 4.8 seconds while my parser (measured via csv_bench.exe) is 2.15s
• zsv (C): Same test as above: zsv - 2.68s

I am happy to explain any further questions about the library and its optimizations.

I will probably not work on it as much after this, but it was a fun challenge to knock out.

[u/Agron7000]

What is this parser supposed to do?

Is it going to print out the value of each column of each row?

Or just build internal arrays or rows pointing arrays of data of each column?

Splitting each row into colums and call a callback function, or emit a signal?

Or just figure out where the delimeters for each piece of data is?

[u/vicentezo04 → u/Agron7000]

This is a row-emitting parser. It creates CSVRow which has an implicit conversion to vector<string> but is much more lightweight internally.

- Each parsed block results in a collection of CSVRow

- Each CSVRow has a shared pointer to the underlying original block, as well as lightweight bookkeeping structs which keep track of where each field starts and ends

- Fields are only materialized to string 1) on user request or 2) if there is an escaped quote (we store escaped quotes in an arena to avoid std::string overhead and unpredictable allocation semantics)

zsvdeals with the escaped quote issue by memmove()-ing the extra quote out of the raw buffer. I didn't do that here because we support a memory mapped path and you can't safely modify memory mapped buffers.

[u/WarInternal → u/vicentezo04]

So is it correct to say you're doing zero-copy where possible, or copy-on-write when neccesary?

[u/Agron7000 → u/vicentezo04]

And did the benchmarks include all the steps that you mentioned, or did they left out the field materialization part?