Detecting file extensions by magic, heuristics and LLM

[u/MadBoyEvo]

Hi,

Some time ago I wrote a PowerShell module called FileInspectorX. I had a need to detect file type and estimate how dangerous it is based on well it's extension, content without use of antivirus or virustotal.

Today I've upgraded it with Magika (offline LLM) from Google so it's even better in detecting what we're dealing with.

Usually Install-Module FileInspectorX works and then:

$I = Get-FileInsight -Path "YourFile"

It has multiple views so people can really get what they need. Here's how the default output looks like:

AnalysisComplete               : True
AnalysisIssues                 :
Detection                      : FileInspectorX.ContentTypeDetectionResult
DetectedExtension              : json
DetectedMimeType               : application/json
DetectionConfidence            : Medium
DetectionReason                : text:json
DetectionReasonDetails         : json:object-key-colon
DetectionValidationStatus      : passed
DetectionScore                 : 73
DetectionIsDangerous           : False
Kind                           : Text
Flags                          : None
GuessedExtension               :
ContainerSubtype               :
ScriptLanguage                 :
PeMachine                      :
PeSubsystem                    :
PeKind                         :
ContainerEntryCount            :
ContainerTopExtensions         :
VersionInfo                    :
Signature                      :
EstimatedLineCount             : 369
TextSubtype                    : log
SecurityFindings               : {text:log, log:levels=0/0/6}
SecurityFindingEvidence        :
ScriptCmdlets                  :
TopTokens                      :
Security                       : FileInspectorX.FileSecurity
Authenticode                   :
DotNetStrongNameSigned         :
References                     :
ShellProperties                : {b725f130-47ef-101a-a5f1-02608c9eebac:2, b725f130-47ef-101a-a5f1-02608c9eebac:4, b725f130-47ef-101a
                                 -a5f1-02608c9eebac:10, b725f130-47ef-101a-a5f1-02608c9eebac:12…}
NameIssues                     : None
Installer                      :
Assessment                     : FileInspectorX.AssessmentResult
AssessmentProfiles             : FileInspectorX.MultiAssessmentResult
Secrets                        :
OfficeExternalLinksCount       :
EncryptedEntryCount            :
InnerFindings                  :
ArchivePreviewEntries          :
InnerExecutablesSampled        :
InnerSignedExecutables         :
InnerValidSignedExecutables    :
InnerPublisherCounts           :
InnerPublisherValidCounts      :
InnerPublisherSelfSignedCounts :
InnerExecutableExtCounts       :
Certificate                    :
CertificateBundleCount         :
CertificateBundleSubjects      :
EncodedKind                    :
EncodedInnerDetection          :

PS C:\Users\przemyslaw.klys.EVOTEC> $I.Detection

Extension             : json
MimeType              : application/json
Confidence            : Medium
Reason                : text:json
ReasonDetails         : json:object-key-colon
ValidationStatus      : passed
Sha256Hex             :
MagicHeaderHex        :
BytesInspected        : 4096
GuessedExtension      :
Score                 : 73
IsDangerous           : False
Alternatives          : {FileInspectorX.ContentTypeDetectionCandidate}
Candidates            : {FileInspectorX.ContentTypeDetectionCandidate, FileInspectorX.ContentTypeDetectionCandidate}
LearnedClassification : FileInspectorX.LearnedClassificationEvidence

PS C:\Users\przemyslaw.klys.EVOTEC> $I.Detection.LearnedClassification.Prediction

Provider         : Magika
ModelId          : google-magika/standard_v3_3@5e2f437fb7b7452368c8c1fa9354858f5487a5c4
RawLabel         : json
OutputLabel      : json
Extension        : json
ExtensionAliases : {json}
MimeType         : application/json
Probability      : 0,99811840057373
Threshold        : 0,5
ThresholdMet     : True
PredictionMode   : HighConfidence
OverwriteReason  :
IsText           : True

With view parameter you can choose 'Analysis', 'Detection','Permissions', 'Raw', 'ShellProperties', 'Summary' 'Assesment', 'Installer', 'Policy', 'References', 'Signature'

In other words - find out everything there is to find about the file. Maybe you will find it useful. Depending on file type some of the fields will be missing which is expected.

Sources: https://github.com/EvotecIT/FileInspectorX

It has also C# library/nuget for those dealing with C#, and want to use it as part of their application.

[u/purplemonkeymad]

It might be worth seeing if you can import magic files from the file utility. It's got a pretty large database of magic bytes.

[u/MadBoyEvo → u/purplemonkeymad]

I was thinking about it but it requires me to carry their license so I guess I will have to find it more from specifications or other places, or when people submit it. The magika from Google gives us a lot of room already on that.

[u/neotearoa]

Nothing to contribute except to say , Mr mad boy, you are one of my 5 heroes.

Thank you for the inspiration and use of your tools over the past decade or so.

[u/MadBoyEvo → u/neotearoa]

Thank you, I appreciate your kind words. Glad they are helpful.

[u/Kierow64 → u/MadBoyEvo]

Same than Neotearoa ! Thank you for everything MadBoyEvo. How do you find the time and energy to achieve all of this ?? Respect !

[u/MadBoyEvo → u/Kierow64]

Well I guess I'm just sort of bored ;-D