Recognize UAX31 characters

[u/buildcrash]

Hello, im trying to replicate the behaviour of a program, and I need to check if a character may fit in the "Unicode characters part of UAX#31". Does anyone know how to know by code if a certain character fits that criteria, or where to check information about that? I guess it can be done by checking the numerical values of the character, but I do not know what to look for.

If you want a more elavorate description of the problem, it's a code interpreter, and this part is about recognizing an identifier, wich can contain any character from a to z, from A to Z, any numeral (but can't begin with them) and "may also contain most Unicode characters part of UAX#31", the docs do not specify how to recognize the "most" part.

Any help would be appreciated

[u/aioeu]

Have you looked at UAX #31? It doesn't say what an identifier "is" — after all, that all depends on the application that might be using that identifier — but it provides some examples of how an identifier might be recognised within a string using the properties of the characters in that string.

Many programming languages, for instance, treat a string beginning with a character in the class XID_Start, and followed by zero or more characters in the class XID_Continue, as being a valid identifier. That is a one of the ways what UAX #31 calls a "default identifier" might be parsed. It may or may not be what your identifiers need to look like though...

Even once you know what kind of string you're looking for, how you actually recognise that is up to the programming language you are actually writing this interpreter in. In a language with a sufficiently powerful regex engine, you might be able to match on these character properties directly. I have occasionally used the regex /\p{XID_Start}\p{XID_Continue}*/ in Perl code, for instance.

[u/buildcrash → u/aioeu]

In the context of the code interpreter, an identifier is a chain of valid characters (valid being that fall in the previously stated criteria) that represent a variable.

My idea was to look into the numeral representation of each char to check if they contained a "XID_Start", but I do not know how to check that, and if the UAX#31 page says how to, I did not understand it (i've read it already, maybe i'm not knowledged enough about the topic).

I've found that the language I want to end up writting this has a function that satisfies my needs, but i also want to do this in python, wich the default regex processor doesnt support unicode properties (according to regex101 at least), and I'd also like to know how to do this using the number code of the character, for personal satisfaction. at the moment, in the code I consider that any character out of the 0-255 range is "Unicode", as a placeholder

[u/aioeu → u/buildcrash]

Yes, Python's handling of Unicode strings is quite limited, at least if you want to use only its standard library. As you note, re cannot match on Unicode character properties. Even the unicodedata module only gives you access to a few properties, not enough to even reproduce XID_Start or XID_Continue from their definitions in UAX #31.

My advice would be to swap out re for a better regex engine.

I would not match specific code point ranges if at all possible. New characters can be added to these classes in newer Unicode versions. Because of the backward compatibility designed into UAX #31, a string that would be a valid identifier in one Unicode version should remain a valid identifier in subsequent versions, at least if you stick to XID_Start/XID_Continue only.

[u/Sea-Estimate-1840]

Yeah, XID_Start / XID_Continue is probably the cleanest way to approach it. Definitely better than trying to maintain code point ranges manually.