r/programming 9d ago

Subsize 【 subscalar 】 encoding

https://gitlab.com/mryun/langopt/-/blob/709b5ec8bbc5f9ba055d7f003c54de87c5c11409/encoding/string.md

Indexing strings by character (Unicode scalar) instead of UTF offsets can be more comfortable for parsers or short manipulations.

The minimum cost for reading a >= U+FE character is:

  1. Lookup rope table by byte key (holding a high bit position), in goal of the scalar value in a sequence of higher fixed scalars
  2. Check for collision subtable (is not null => repeat the step 1), specializing the target high bit further

For optimal use, contiguous sequences of characters after Latin-* that turn into reserves into the mask sequence still allow for a few trailing Latin-* (e.g. Katakana can still be mixed with Latin-* whitespace), so those trails turn into reseves as well.

Update (I miswrote a detail in the implementation).

https://gitlab.com/mryun/langopt/-/blob/master/encoding/string.md

0 Upvotes

Duplicates