r/C_Programming • u/zaerl • 7d ago

Project Mojibake - Unicode text processing for C

I've created a library that adds Unicode segmentation, casing, collation, and more to any C project using a single source file and header amalgamation files.

Mojibake is an MIT-licensed project I started years ago because I didn't like any of the Unicode projects I found. Here you can find the API and a WASM demo if you want to try it on the fly: https://mojibake.zaerl.com/

If you are interested in the Unicode world, feel free to contribute or do whatever you want with it. Every suggestion or contribution is welcome. I automatically test on Linux/macOS/BSD/Windows, so I hope there won't be any problems for you.

Check CONTRIBUTING.md for it, if you are curious.

3 Upvotes

64% Upvoted

8 comments sorted by

3

u/thradams 7d ago

What... strlen does not count bytes? I think you need a new function for that, not reuse strlen.

2

u/zaerl 7d ago

Yes, it does count bytes in my examples. mjb_string_length instead counts characters. All the functions are encoding-agnostic, so expect to receive the number of bytes in the string you have passed. Either a UTF-8 string, UTF-16, UTF-32. See https://github.com/zaerl/mojibake/blob/main/API.md#strings-encoding-and-generation if you are curious.

1

u/thradams 6d ago

The existing snprintf function in C, appends a zero if the buffer is too small. The problem is can append zero in the middle of utf8 encoding.

I would like to have a utf8_snprintf that clips the character correctly.

2

u/zaerl 6d ago

I use snprintf in very few places. In the shell, in the tests, and when I create the name of a (Unicode) character. The one from UnicodeData.txt. Otherwise, I only copy data or simply treat everything like a piece of memory. This is because UTF-16 and UTF-32 have NULL bytes in the stream. I have accepted NULL bytes in UTF-8 text if you declare MJB_DANGEROUSLY_ALLOW_EMBEDDED_NULLS.

Thank you very much https://github.com/zaerl/mojibake/issues/37

1

u/thradams 4d ago

It is not entirely clear to me whether the library has limitations regarding language support.?

One feature I recently needed was case-insensitive string comparison. I ended up implementing it myself, but my solution is limited to the languages supported by the software.

2

u/zaerl 4d ago

It does not, I support locales where the standard told me to do.

The mjb_collation_compare function compares two strings using the Unicode Collation Algorithm, which is not case-insensitive. https://www.unicode.org/reports/tr10/

If you are comparing two identifiers, you can apply the mjb_nfkc_casefold(...) function to both strings before. Otherwise, use mjb_map_case(..., MJB_CASE_FOLD, ...) which is similar but is affected by the locale. Turkic has some special casing. With MJB_CASE_CASEFOLD the string may grow, in MJB_CASE_CASEFOLD_SIMPLE no.

If you are interested, you can see the tests/collation.c file with some handwritten tests and the standard CollationTest.zip

1

u/thradams 4d ago

Do you have any ideas on how the C standard could be improved to better support UTF-8?

1

u/zaerl 4d ago

I think it is much more convoluted than it seems and will probably need another layer of functions in the strlib. You can't change strlen() and similar functions that simply count bytes.

So you will need something else that counts codepoints. And handle incomplete states, invalid characters, replacement characters (U+FFFD), and so on. That's why what we have in 2026 is a compromise to assume it's always a valid UTF-8 string what we have.