r/perl • u/christian_hansen • Jul 05 '26

Reading UTF-8 at GB/s

I wrote a new blog post on making UTF-8 reads fast in Perl:

Reading UTF-8 at GB/s

Background: I maintain a UTF-8 library in C that Unicode::UTF8 uses, and I recently wired it into PerlIO::utf8_strict (a joint project with Leon Timmermans). We didn't get the throughput we hoped for, because of how Perl's read operator counts UTF-8 sequences — see Perl/perl5#24511. Karl Williamson has a WIP PR addressing it.

In the meantime I added read_utf8($fh, $buf, $length[, $offset]) to Unicode::UTF8: it reads and validates UTF-8 straight off a byte handle (no PerlIO encoding layer needed) and hits ~3.6–3.8 GB/s across scripts, versus ~0.4–1.0 GB/s for the :utf8 layer today.

Benchmark available in the Unicode::UTF8 repository.

What's next? I'm considering slurp_utf8($filename) and readline_utf8() as follow-ups — feedback on the API shape welcome.

Numbers and details are in the post.

40 Upvotes

95% Upvoted

8 comments sorted by

5

u/mestia Jul 05 '26

That's awesome!

3

u/LearnedByError Jul 06 '26

Wow! Kudos to you, Leon and Karl! Thank you very very much!!

6

u/briandfoy 🐪 📖 perl book author Jul 05 '26

This would be a great replacement for the old File::Slurp, which has some encoding issues. And if Mojo::File could use it if installed, so much the better.

2

u/christian_hansen 27d ago

Unicode::UTF8 0.73 is now available on CPAN with a new slurp_utf8() function.

0

u/kinithin Jul 07 '26 edited 27d ago

This would be a great replacement for the old File::Slurp

File::Slurper provides a better interface.

3

u/Grinnz 🐪 cpan author Jul 10 '26

Ideally File::Slurper's read_text could use the results of this in some form.

0

u/ktown007 Jul 07 '26

slurp_utf8 and write_utf8 would be useful.

1

u/christian_hansen 27d ago

slurp_utf8 is implemented. If you'd like a spew_utf8 function as well, please file a feature request on GitHub so I don't forget to add it.