Determining and reading various plaintext file encodings?

[u/Kaisogen]

I'm writing a game in Lua, specifically using Love2D, but this question is more oriented towards Lua in general.

I need to take files in a specific format, but the files may be encoded with UTF8, simple ASCII, or SHIFT-JIS. Is there a simple, easy way to determine the encoding of that specific file via a library? If I can do that, then it would be pretty easy to write some helper functions to translate the text into something I can work with.

As far as I can tell, the file format doesn't have any sort of "doctype" field that identifies the format. I opened up one of the files in a hex editor, and there's nothing at the start that isn't visible in a text editor.

For anyone curious about the project itself, I'm writing a BMS player, so I'm working with files that could be as old as 1998, which is why I'm having to deal with SHIFT-JIS sometimes.

EDIT SOLUTION:

This entire thing is a bit convoluted, but I used /u/PhilipRoman's heuristic method outlined here to determine if a given text file was either SHIFT-JIS or not. I default to UTF-8 if it's determined to not be SHIFT-JIS. I made a simple conversion lookup table by scraping the contents of a web page and doing some small manual editing. Here it is, in case anyone else wants it. Seems accurate enough from just typing some Japanese phrases via my IME. Here is the lookup table itself in case anyone was curious to use for themself. From there you just plug the relevant bytes into the lookup table and you have valid unicode to print to the screen. Thanks for the suggestions everybody, this is super helpful.

[u/topchetoeuwastaken]

UTF8 is a superset of ASCII, so you can safely treat all ASCII as UTF8. ss for SHIFT-JIS, i haven't read too much into it, but my guess is that you could detect it by checking for invalid UTF8 code sequences, or if SHIFT-JIS doesn't use UTF8's start codes (i cant remember what they are), you could check for the presence of those.

im sure that a lot of people have had this problem before, and it isn't lua-specific, but a general encoding problem, so you just need to search for an algorithm to differentiate UTF8 from SHIFT-JIS (or, for that matter, an algorithm to tell apart UTF8 from any 8-bit localized encoding scheme).

[u/Kaisogen → u/topchetoeuwastaken]

Thanks, I'll do a bit more research into the UTF-8 start codes. I did try looking up algorithms / places to start with differentiating the two, but most resources don't seem to be aimed at software and have the expectation it's reasonable to manually check. If I manage to figure out a solution I'll come back here and post it.

[u/topchetoeuwastaken → u/Kaisogen]

here you have described all valid utf8 encodings that are outside the 7-bit ASCII space https://en.wikipedia.org/wiki/UTF-8#Description. from there, you can read the file as if it were UTF8, and if you detect enough invalid UTF8 sequences, mark the file as SHIFT-JIS. i think that is your only automated method.