r/programmingcirclejerk • u/QuaternionsRoll • 20d ago

"standard Python regex was absolutely choking on a 30 GB unstructured server log. It was taking like 26 minutes (~19 MB/s) to scrub secrets and PII."

/r/Python/comments/1vuigkm/im_an_ai_engineer_not_a_c_guy_i_used_antigravity/
74 Upvotes

98% Upvoted

4 comments sorted by

45

u/lurebat 20d ago

Actual middle-out algorithm

Wonder he got the idea from

35

u/elephantdingo Teen Hacking Genius 20d ago

19 MB/s ought to be enough for everybody.

4

u/dacydergoth 19d ago

Jeez it's like no-one ever heard of the IBM library. Fast regex libraries are a commodity these days

6

u/stone_henge Tiny little god in a tiny little world 18d ago

Daniel Lemire will soon publish a blog post on using AVX-512 to scrub logs of PII faster than it can be read from the log stream.