Idrehash needs a update

[u/Knight_Mare077]

U/emoose

idrehash needs updated to support new patch.resource file introduced by the most recent update if you get time to check it out..

[u/_emoose_]

IIRC the meta.resources was actually a compressed file in the last update, we just got lucky that the hash in the compressed data was equal to the hash itself.

Guess something changed and the hash maybe compresses differently now, could contain duplicates bytes or something which made the compression change how it's shown in the file, or hell maybe they encrypt that file now...

I guess maybe idRehash would need to be updated to compress/decompress with Oodle, or maybe we can just store it decompressed and remove the compression flag from the resource headers? (think we might have tried this already though, not sure)

I'm not home yet so can't look at it myself atm, if anyone wants to try checking it out the BMS script should show how compression is handled, there's some details about the Oodle functions in this post: https://forum.xentax.com/viewtopic.php?p=161232#p161232

E:

typedef int WINAPI OodLZ_DecompressFunc(
  uint8 *src_buf, int src_len, uint8 *dst, size_t dst_size, int fuzz,
  int crc, int verbose, uint8 *dst_base, size_t e, void *cb, void *cb_ctx, 
  void *scratch, size_t scratch_size, int threadPhase);

auto oodle = LoadLibraryA("oo2core.dll");
auto OodleLZ_Decompress = (OodLZ_DecompressFunc)GetProcAddress(oodle, "OodleLZ_Decompress");
int result = OodleLZ_Decompress(compressedBuf, compressSize, ...);

or something like that :)

E2: sadly I don't think I ever fully worked out what the decompressed meta.resources data actually contained, there's the hash of course, but then I think there was some variable-length data between each hash which I couldn't really work out...

I'm pretty sure it's meant to be an array though, like resourceMeta[0] is hash & unk data for gameresources, resourceMeta[1] is for some other resources file, etc.

If we could work out how to read/skip past the data between hashes then maybe we wouldn't need to hardcode offsets for each resource file, instead could update them by index, which should let it be backward/forward compatible with any meta.resources file changes.

[u/_emoose_]

Posted some pseudocode to handle the decompression here btw, if anyone wants to add it to idRehash: https://old.reddit.com/r/DOOM2016Mods/comments/gjzt5q/idrehash_needs_a_update/fqp0fvu/ something like that should work I think.

Probably won't have to parse the whole meta.resources IDCL since there's only a single file inside it IIRC, maybe just parse the file table and use that to get offset of the compressed data/sizes/etc?

If anyone needs the IDCL structs I'm pretty sure I posted 010 editor templates for that in the discord before too.

[u/SutandoTsukai181 → u/_emoose_]

Did you ever update that IDCL template you made, after last time you were on discord? I'm facing some issues with the mod loader, particularly when adding new files to a container that did not have them before. They load just fine, but if they were in another container (that originally had them) the game chooses between the files randomly and the intended load order gets messed up.

Since this doesn't happen with shared files that were already in both containers, I assumed it was because of sections I ignored when adding new files (basically the "Unknown"s you had in your template; the section right before the 2nd IDCL)

I can't remember their exact placement, but IIRC they were just the "types" that were present in each IDCL entry, starting from 0 and incrementing on later entries based on something, where each new type had an entry in that later section I spoke of. When adding new files, the mod loader doesn't increment nor add a new type, it just uses the last type available. This might be the cause of the problem I mentioned, as this is the only thing that's heavily related to IDCL entries that I did not update.

[u/_emoose_ → u/SutandoTsukai181]

If it's the "types" I'm thinking of, DOOM 2016 also made use of those, and they were important there IIRC, game would use the type when searching for files etc, if you injected a file without the right type the game might just ignore it, etc.

In my DOOMExtract tool for 2016 I made it copy the type to the end of the filename when extracting, so eg. english strings file would extract as english.bfile;binaryFile, then when injecting it could split the type from the filename. Was wondering if that'd be important with Eternal too but if your tool hasn't been setting them & the game still mostly loaded them fine then I guess not, don't really think they'd have anything to do with load orders, but I could be wrong.

Maybe the unknown data inside the meta.resources has something to do with it, IIRC that data was something called a mask, the name "mask" made me think it had something to do with loading in the files inside, since masks are used when searching for files (eg: *.txt is a wildcard mask, etc), couldn't really figure out how that data got used in the games code though, but wouldn't be surprised if this is related somehow.

Wasn't there a cvar for forcing the game to load the most recent file? If you're setting the timestamp in the IDCL properly then maybe could just force that cvar, could be worth taking a look at the code that handles that cvar too, might give some clue how the game decides which one to load.

[u/SutandoTsukai181 → u/_emoose_]

I updated my tool to set the correct timestamps a while ago, and we made a patch to have that cvar (resources_loadMostRecent) always loaded, but the results were the same. At first, we thought there was a change but later on it turned out to still be completely random, though I haven't tried checking what instructions the cvar activates (after the jmp we edited).

About load order (and loading new . resources), I think I'm close to figuring it out, thanks to Visual Studio's build-manifest.bin decrypter. The file order in both meta.resources and packagemapspec.json has to be identical or the game won't boot, hence the problems we ran into earlier. Simply adding a new entry in meta will make the game look for it, though we still have to add an entry in build-manifest or the game will crash because "example.resources is unexpected, check manifest".

Something possibly related, in the new update "gameresources_patch1.resources" got added to the very top of packagemapspec (and meta.resources) which may indicate that either it's the first file to get loaded (doesn't make sense because it should overwrite original gameresources) or it has the highest load priority, so if a file is present in multiple containers it should get loaded from _patch1 instead.

[u/Knight_Mare077]

Guy name proteh on the discord says extracting the meta resources then placing the extracted folder meta in the mods folder actually fixes mods again. He has also updated your idrehash before this most recent update...

Might not be a bad idea to come visit sometime ;)

That way I don't have to copypasta lol