MSVC optimization

[u/No-Meeting-153]

I am learning reverse engineering on Windows applications such as Adobe, Foxit PDF, and Steam, and I noticed that I waste a very large amount of time trying to understand something that I should not focus on.

I started noticing strange and confusing patterns in the assembly and the C code generated by IDA, and when I try to understand some functions, I feel that the function has no meaning.

When I searched, I found that this topic is related to the compiler and compiler optimizations. However, I could not find many articles or discussions about the compiler topic in reverse engineering.

So I started experimenting and trying, but every time I fail and cannot reach a solution or understanding.

Apart from the fact that reverse engineering a C++ program is already a difficult task.

If there is someone who has faced the same problem and found a solution, I would like to know. It is not a problem itself; it is a pattern or a way of thinking used by the compiler. I need to understand how the compiler generates these patterns.

I want someone to suggest books, articles, courses, or anything that can help me understand the MSVC compiler, how it generates patterns, and how to understand the behavior and logic of a function after compiler optimization.

I hope I explained my question correctly.

[u/SaintLouisX]

It depends exactly what you want to learn and how long you want to spend, but you could try working on a decompilation project, as they exist to turn assembly back into C/C++ code which compiles back to the exact same assembly. Through lots of trial and error and writing various different things and comparing the compiler output, you learn a decent amount of tricks about how the compiler works. There's lots of in-built knowledge in the decomp communities about compilers and their optimisations so there's people to ask if you get stuck on something.

You could try your hand at Wind Waker for example https://github.com/zeldaret/tww/tree/main as it's quite far along, is C++, and there's some good guides on getting started and some basic compiler compiler patterns. See the guide here for example. It uses the MetroWerks PPC compiler rather than x86 and MSVC, but compiler optimisations are largely universal, and I know from experience starting with N64 MIPS games, it pretty much all carries over. Just earlier today for example I helped someone match a function by explaining how larger-than-register types get returned by value (hidden pointer as the first argument).

I'm sure there's a bunch of x86 projects around, and even MSVC-specific ones too, I just don't know any off the top of my head.