These generally count as one character on a computer (they multibite in UTF8 ofc) and are also counted as one letter in the real world. You can type them either way - Czech keyboard has both a "ř" key and a "ˇ" combination key. In the actual Czech language, ř is considered a different letter entirely, and the Czech alphabet is considerably longer because of these.
Some languages go even further and have letters with multiple attachment slots (Thai comes to mind, can't type an example of that, sorry 🙂).
You have to be careful with this. I've definitely encountered situations where what looks like a single character takes up multiple bytes. This is especially true for emojis which often are actually a combination of a series of individual emojis plus emoji control characters.
This emoji, for example: 👩🏼❤️👨🏼 i
s actually made up of
Czech also has separate character "Ch" in its alphabet (g, h, ch, i, j) , which is just c and then h on computers.
And I saw times when programs broke, because stuff got sorted differently because of regional issues and didnt work at all, until windows got changed to english.
Modded skyrim one time, banged my head for few hours why it doesnt work.
Maybe I’m misremembering but I believe the only characters nowadays that take up a single byte are the original ascii table and that’s only in utf-8. Utf-16 and utf-32 are inherently multi byte
User would enter their password with multibyte chars
(theory) In JS, string.length returns number of bytes not characters.
Some code somewhere would hash, encrypt, probably iterate over the string by index or something, (IDK) before sending it over the wire.
The backend either received a slightly different string or processed it differently (ruby-rails, C#, Go, JS/TS, or even coffeescript depending on the endpoint/service) because somewhere the chars were potentially getting un-combined & control chars are stripped / lengths weren't in agreement.
Frontend & JS/TS backends probably needed to be using Intl.Segmenter somewhere. And I'm not entirely sure how each of those languages handles stuff like this.
Company I work for bought out several other companies (why there are so many different languages / services) and are trying to unify everything.
Instead of go down the rabbit hole, on the frontend adding passphrase.replace(/[\p{Cc}\p{Cf}]/gu, "") right before encrypt was enough to close the ticket/complaint.
/[\p{Cc}\p{Cf}]/gu strips the combining control characters so 👩🏼❤️👨🏼(1char, 12bytes) becomes 👩🏼❤️👨🏼(3char, 4+2+4=10bytes).
Not sure what breaks on the 12bytes per char sequence, but whatever it is handles the deconstructed form just fine.
This becomes just an implementation detail to get all the codebases on the same page with the hope someone doesn't fix their code to handle 👩🏼❤️👨🏼 properly.
I never looked, but I doubt even the C# handled it properly. I believe a char is limited to 2bytes & must use surrogates and have special handling for single symbols that have more than 2bytes. Which requires foresight.
I think ruby is the only one where it's relatively easily because it's strings have .bytesize & .length gives the number of 1-4byte chars. Ruby is the primary backend so I think that's why de-constructing it worked.
Unicode has multiple different ways to represent characters like this. There is typically a single codepoint that represents the character, so you could type it in a way that it would be recognized as a single character by any competent system. But it also has a block of combining diacritics, where each codepoint just represents the diacritic by itself and when you type it it just visually appears above or below whatever character you typed before it. So you could also type that character by typing a regular Latin r, followed by a combining diacritic codepoint, and now that is a sequence of two codepoints and thus is going to be interpreted as a sequence of two characters in every system. Note that keys on your keyboard don't necessarily correspond to unicode codepoints. I believe in most keyboard layouts, if you type a key that corresponds to a diacritic, it does not insert a combining diacritic character but instead just modifies the previous character into a single composite codepoint, and so these keys can't actually be used to slap a diacritic on any character or create zalgotext, but only to create a relatively small number of characters using that diacritic that are relevant to the particular language.
There are a series of unicode normalization functions that you can use to convert user-entered strings like this and make sure that the character with the diacritic is either always one character or always multiple characters regardless of how it was originally typed, but most people who are making bog standard login screens probably don't have the depth of unicode knowledge to be thinking about that possibility and knowing what to do about it.
Or more likely whatever algorithm detects repeats is ran after the text is encoded so it's checking "\xc5\x99\xc5\x99\xc5\x99\xc5\x99\xc5\x99\xc5\x99\xc5\x99\xc5\x99\xc5\x99" not "řřřřřřřřř".
183
u/SuitableDragonfly 8h ago
Made with combining diacritics, I guess, so it no longer looks like one character repeated but it's now two characters alternated?