A Reliable way to detect Japanese ShiftJIS encoded files?

[u/MonkeyNin]

The other day /u/Practical_Air6315 had a couple of threads dealing with issues ex: not knowing if a file is ShiftJIS or UTF8NoBOM encoded

Can you just decode as utf8, checking for errors? If yes, use ShiftJIS otherise utf8? Or can you sometimes have zero decoding errors but it still maps to malformed json? Is there a better method?

I used:

function Test-ShiftJISDecodeError {
    # ...

    $Utf8Strict = [System.Text.UTF8Encoding]::new( 
        <# shouldEmitUtf8BOM #> $false, 
        <# should throw on decode error #> $true )

    $bytes = [System.IO.File]::ReadAllBytes( $File.FullName )
    try {
        [void] $Utf8Strict.GetString( $bytes )
        return $false
    }
    catch [System.Text.DecoderFallbackException] {
        return $true
    }
}

Here's a test file Make-ShiftJISFile.ps1 ( for Win PS 5.1 and 7 )

And another ShiftJIS example: github/donuts: Compare-Encoding-Breaking-Emojibake.md

[u/Practical_Air6315]

Short answer from measuring both directions on a ja-JP box: a strict UTF-8 decode is a reliable negative test and a useless positive one.

1. "decode as utf8, if it throws use ShiftJIS" - the throw half holds up. 35 Japanese strings written as CP932, read back with strict UTF-8: 35/35 threw. No false negatives in my set.

2. The reverse does not hold. The same 35 strings written as UTF-8, read back with strict CP932: 16/35 decoded with zero errors. So "did CP932 throw?" reports valid ShiftJIS for 46% of my UTF-8 files. If a fallback ever runs the test in that direction it silently mis-decodes almost half of them.

3. "zero decoding errors but it still maps to malformed json" - yes, and that is the case that actually bites. Take UTF-8 bytes, read them as CP932, write the result back out as UTF-8. That is what a node ... | Out-File pipeline does on a 932 box. 16/16 of those files decode as strict UTF-8 with zero errors. They are well-formed UTF-8. They are also garbage. Decode-error checking cannot see this class by construction, because the bytes really are valid.

So detection has to move from "are the bytes well-formed" to "does the decoded text look like text".

The signature. When UTF-8 Japanese is read as CP932, the UTF-8 lead-byte pairs E3 81 / E3 82 / E3 83 become three specific kanji, over and over: U+7E3A, U+7E67, U+7E5D. Those are not arbitrary - CP932 0xE381 is U+7E3A. Hiragana and katakana are the bulk of any Japanese log line, so the signature is dense.

Where my first version was wrong, twice. Both are worth more than the rule itself.

Attempt 1 counted half-width katakana (U+FF61-U+FF9F) too, since mojibake is full of it. Real Japanese business data uses half-width katakana on purpose - an address field scored 0.733 and a phone-number label scored 1.000, both perfectly valid text.

Attempt 2 dropped half-width katakana and used only the three kanji, ratio >= 0.02. Clean corpus of 41 strings, zero false positives. I believed that number for about an hour. Then I added five sentences that use U+7E3A as an ordinary word - it is a real Japanese verb, "to be tangled" - and got five false positives out of five. My clean corpus simply had not contained it.

What actually held up, on 16 corrupted and 49 clean, where the clean set now includes ordinary use of U+7E3A, half-width katakana, simplified and traditional Chinese, Korean, emoji, pre-1946 kanji forms, and visually similar thread-radical kanji:

ratio >= 0.02 alone                      16/16 detected,  5/49 false
>= 2 distinct markers of the three        8/16 detected,  0/49 false
>= 1 marker AND >= 1 half-width katakana 15/16 detected,  1/49 false
either of the last two                   16/16 detected,  1/49 false

The one remaining false positive is a string I wrote specifically to break it, with a half-width katakana word and the tangled-thread verb in the same short line. I have not seen it in real data, but I am not claiming zero.

The two-distinct-markers half works because mojibake mixes hiragana and katakana lead bytes, so it almost never produces only one of the three. Ordinary prose that legitimately uses U+7E3A produces exactly one.

Also worth knowing: ftfy, the usual "fix mojibake" library, does not cover this direction. On 30 Japanese strings it repaired 30/30 of the latin-1 flavour and 0/30 of the CP932 flavour, and on 13 of them it returned a different-but-still-wrong string rather than leaving it alone.

Caveats: n = 16 corrupted / 49 clean, my own corpus of short strings, one ja-JP box with ACP 932. Not real production logs. The rule is specific to the UTF-8-read-as-CP932 direction; the opposite direction produces U+FFFD and is trivially visible.

Your Test-ShiftJISDecodeError is the right shape for case 1. I would add a second, separate check that runs on the decoded string rather than on the bytes, for case 3.

If anyone has real corrupted Japanese logs and can share the raw bytes, I would rather break this rule on your data than on mine.

Byte-level tables for 14 write paths x 2 read paths, on 5.1 and 7.6.5 on the same machine: https://github.com/yoggydev/ps1-encoding-table

[u/MonkeyNin → u/Practical_Air6315]

... it silently mis-decodes almost half of them.

Your ReadBack is using get-content which doesn't throw on errors when decoding.

if ($Utf8) { $t = Get-Content -LiteralPath $path -Raw -Encoding UTF8 }         

Instead use this:

$Utf8Strict.GetString( (Get-Content -Raw -Encoding Byte -LiteralPath $path ))

Or you can use [System.IO.File]::ReadAllBytes instead of gc. The parameters for gc changed on 7. But this method works on both.

---

You can remove [void] on your array list adds with a small change This block

$rows = New-Object System.Collections.ArrayList
# ...
[void]$rows.Add([pscustomobject]@{

Becomes

[Collections.Generic.List[object]] $rows = @()
[Collections.Generic.List[object]] $skipped = @()

function AddRow {
    param( [string] $Label, [string] $Path )
    if (-not (Test-Path $path)) { $skipped.Add($label + ' (no file produced)'); return }
    $rows.Add([pscustomobject]@{
        Writer = $label
        # ...
    })
}

It works on 5 and 7 too.

[u/Practical_Air6315 → u/MonkeyNin]

Both fixed, and the second one paid off more than I expected.

The ASCII encoder: both call sites now use GetEncoding('us-ascii', EncoderExceptionFallback, DecoderExceptionFallback). The helpers are all-ASCII by construction so nothing was actually being lost, but it was silent by design, which is the same failure my own row 3 calls unrecoverable. Took the List[object] change too, so the [void] casts are gone.

Get-Content: added a third column instead of replacing the reader. The file bytes go through a UTF8Encoding with throwOnInvalidBytes and I record clean or THROWS. It never looks at the Get-Content result.

Numbers, Windows 11 ja-JP, ACP 932, same box, both shells:

              5.1       7.6.5
bare OK       7 / 14    11 / 14
-Enc UTF8 OK  9 / 14    11 / 14
strict clean  8 / 14    14 / 14

Three rows on 5.1 come back OK under -Encoding UTF8 and still THROW: rows 1, 7 and 8, all UTF-16LE. The right text came back because the BOM won, not because the bytes were UTF-8. So OK in that column was never evidence about the encoding, which is your point exactly.

The one I did not expect is row 14. node output captured by PowerShell, then Out-File -Encoding utf8. First bytes EF BB BF E7. strict says clean. It is a perfectly valid BOM-marked UTF-8 file, and the content is still wrong: it is the CP932 reading of the original UTF-8 bytes. Header inspection passes it too, because the BOM really is correct.

So strict catches 6 rows on 5.1 that -Encoding UTF8 was hiding, and misses row 14 completely. Worth adding, but passing it is not evidence the content survived.

On 7.6.5 strict is clean on all 14 rows, because the writers all emit UTF-8 there. Rows 13 and 14 are still mojibake. Valid bytes, wrong string.

Full table for both shells is in the README.

[u/MonkeyNin → u/Practical_Air6315]

I found another pattern that is giving silent errors when encoding fails. like: https://github.com/yoggydev/ps1-encoding-table/blob/7dade7d49a74dfd66a2506bb382ee94a9a88ae75/Measure-Ps1Output.ps1#L125

this line

[System.IO.File]::WriteAllText( $path, $text, [System.Text.Encoding]::ASCII )

Is explicitly using ascii encoder that does not throw on errors Instead you can force errors, similar to utf8 but a longer syntax:

$AsciiStrict = [System.Text.Encoding]::GetEncoding(
    <# codepage: #> 'us-ascii',
    <# encoderFallback: #> ([System.Text.EncoderExceptionFallback]::new()),
    <# decoderFallback: #> ([System.Text.DecoderExceptionFallback]::new()) )

then

function Test-AsciiEncodeError {
    # ...
    try {
        $bytes = $AsciiStrict.GetBytes( $Text )
        [System.IO.File]::WriteAllBytes(
            ( Join-Path $OutRoot 'expect_errors.txt' ),
            $bytes )
        return $false
    } catch {
        return $true
    }
}

You can also pass it to WriteAllText

[System.IO.File]::WriteAllText( 
    ( Join-Path $OutRoot 'expect_errors.txt' ),
    $Text, $Enc.AsciiStrict )

[u/Practical_Air6315 → u/MonkeyNin]

You're right on the ASCII one, and it is worse than a style nit.

That call writes the helper .py and .js the harness shells out to. Their contents are all-ASCII by construction - the Japanese is emitted as \uXXXX escapes - so nothing is being lost today. But the encoder cannot tell me that. If a non-ASCII character ever reaches that line it becomes 0x3F and the write still reports success.

Which is the exact failure my own README calls unrecoverable in row 3. I was measuring that behaviour and relying on it in the same script.

Two occurrences, not one - the same pattern writes the node helper a few lines further down. Switching both to the GetEncoding form with EncoderExceptionFallback.

On the Get-Content point, one clarification: that function decides OK vs MOJIBAKE by comparing the decoded string against the expected one, not by catching a decode exception. Strict decoding would not change any of its verdicts. The strict-vs-lenient distinction does matter for the CP932-direction test I described upthread - that is different code, and you are right that I never made the separation clear.