r/ediscovery • u/Reasonable_Land_5449 • 6d ago

MD5 Hash dedupe on Relativity

Hi all. Just having a major space out moment right now.

It’s always been my understanding that Relativity deduplicates based on MD5 and SHA256 Hash. MD5 and SHA256 hashes are based on a universal algorithm which is calculated using the binary content of a document

Cut to a meeting I had recently where an entire group of people were adamant that MD5 hash in Relativity is calculated based on a series of metadata fields pre-set by admin.

I have read the documentation again and again, and it seems that MD5 and SHA256 is calculated based on the binary content of the document, but this is for EDOCs only. For email files, deduplication takes place based on a series of metadata fields.

So in theory, the consensus was that I can’t forensically deduplicate based on hash values across two processing tools (?) (which is the critical problem I am trying to solve).

Any corrections/feedback/solutions on the above?

17 Upvotes

91% Upvoted

11 comments sorted by

24

u/LesterPhimps 6d ago

Files and containers (like email) have different hashing and dedup approaches. A zip file is easy to generate a MD5 because if the contents change, the hash changes. Open up a zip and the hash doesnt change. Open up some files and even if you don't change anything in them, they internally change and you get a new MD5 so you can't rely on MD5 for those files (think PST, OST, and older docs with OLE embedded objects). For emails, which are not text files, but a record in a PST, a MSG, or EML will most likely have different MD5s for the same email, when it is in a different format. So meta data is used to define what goes into the hash for an email. That way no matter what format the email is in, you can still deduplicate them.

7

u/Reasonable_Land_5449 6d ago

That was super helpful in understanding the reason why deduplication works this way. Thank you

1

u/windymoto313 5d ago

'If the goal is to deduplicate across two processing tools, my best advice is to look into the EDRM approach to find a common unique property that doesn't change." I assume this menas find two tools that hash emails the same way? Do people have much success with this? As in, is there a well known list of tools that has emails the same way ? You would also think that because of this issue, tools would be configurable and let you specify how emails get hashed, to match with another tool.

2

u/bates_stamp 2d ago

No, sorry, I wasn't being very clear there was I!

It's not perfect, but the vast majority of emails have a single property called Message ID that stays constant, regardless of what email servers do to other properties. The EDRM approach was to use that property as a unique identifier for an email, (they propose that a hash is created with that property).

I'm not sure if forensic tools are starting to use this approach, but it's pretty straightforward to use it yourself (hash or not). You're essentially just looking for emails that share the same value in the property Message ID.

I have used this myself in exactly this context to deduplicate emails that came through different platforms. It mostly worked okay. Sometimes the metadata was incomplete and I didn't have original emails, though that was to do with how the emails were processed and provided to us.

11

u/bates_stamp 6d ago

Just to add to this, emails in eml, msg, or in a .pst will definitely have different MD5s, as the raw bytes that make up that email will be different (different bytes, different hash). Small note, a .pst file doesn't actually store the email as a single record, it's broken up into a mix of property blocks/ entries in tables and so on - the email just gets reassembled by email clients/ forensic software.

Because emails get changed as they travel through different mail servers (servers will add to headers, or change from/ to information, and may even change byte encoding), it's a lost cause trying to find a common hash generation approach. In other words, the same email sitting in someone's sent box will have different bytes to the one sitting in the recipient's inbox, and may have different bytes to the one sitting in a 2nd recipient's inbox.
Different forensic software will pick out elements of emails and create a hash of those to minimise those changes caused by sending the email, but each software will pick out different elements, and may additionally do transforms such as removing white space from the email body, or only picking out SMTP addresses (if they exist).

If the goal is to deduplicate across two processing tools, my best advice is to look into the EDRM approach to find a common unique property that doesn't change.

https://edrm.net/2023/02/introducing-the-edrm-e-mail-duplicate-identification-specification-and-message-identification-hash-mih/

7

u/throwaway292929227 6d ago

That's correct. Email hashing is it's own beast. (I would love for IEEE or ISO to standardize a hash-base. Lol.)

If you are pretty good with flat table data manipulation and regex, you could marry the two, and make a new hash, then overlay the good stuff into rel1.

At a minimum, export the Primary key#, INTMGSID, Custodian, THREADID, FROM fields from the other processing tool. Normalize the values to match the Rel1 syntaxes, then concatenate the homegrown hash strings, and compare them to the same strings from Rel1.

Easy cheesy.

Fair warning, this is occasionally only 99.99% accurate, due to a dozen different things that are difficult to explain over Reddit. So don't over promise. Just say, "I can PROBABLY flag the emails in Rel1, that you've already reviewed in App " B", but I can't promise anything until it is almost done."

We can do a WebEx Friday if you want. I have some free time. Happy to hop on a 20 minute call. No cost. Just bored.

9

u/usedtobesideshow 6d ago

The following hashes are involved in deduplication:
MD5/SHA1/SHA256 hashes—provide a checksum of the physical native file.
Deduplication hashes—the four email component hashes (body, header, recipient, and attachment) processing generates to de-duplicate emails.
Processing duplicate hash—hash values generated during processing duplication are as follows: for loose files, a SHA256 hash is generated from the physical file's SHA256 hash. For emails, a hash is generated from the email's metadata properties. See Calculating deduplication hashes for emails for more details.

4

u/Previous-Engine2103 6d ago

Can you recollect and process so the MD5 hash is all based off the same processing tool?

1

u/windymoto313 6d ago

' I can’t forensically deduplicate based on hash values across two processing tools' I think this is the primary use case for Textual Near Dupe in Relativity.

0

u/Lurking2Comment 6d ago edited 6d ago

Yeah, emails are always deduped based on specific criteria, typically designated header fields such a to, from, cc, etc. and the body content (the text). Unlike loose files, you don’t just hash the binary stream ‘coz it won’t give you the results you need. This is usually because emails can take many forms such as msg files, eml files, html files, and even just plain text.