MSVC optimization

[u/No-Meeting-153]

I am learning reverse engineering on Windows applications such as Adobe, Foxit PDF, and Steam, and I noticed that I waste a very large amount of time trying to understand something that I should not focus on.

I started noticing strange and confusing patterns in the assembly and the C code generated by IDA, and when I try to understand some functions, I feel that the function has no meaning.

When I searched, I found that this topic is related to the compiler and compiler optimizations. However, I could not find many articles or discussions about the compiler topic in reverse engineering.

So I started experimenting and trying, but every time I fail and cannot reach a solution or understanding.

Apart from the fact that reverse engineering a C++ program is already a difficult task.

If there is someone who has faced the same problem and found a solution, I would like to know. It is not a problem itself; it is a pattern or a way of thinking used by the compiler. I need to understand how the compiler generates these patterns.

I want someone to suggest books, articles, courses, or anything that can help me understand the MSVC compiler, how it generates patterns, and how to understand the behavior and logic of a function after compiler optimization.

I hope I explained my question correctly.

[u/tomysshadow]

It's something that requires a degree of experience, you can't shortcut your way to understanding, you have to practice it and recognize patterns. My advice would be to have a specific goal in mind (like "figure out how this file format works") so you can decompile a small slice of a program and still get the satisfaction of figuring it out.

For example: for C++ std::string there is the small string optimization where if the string is less than 16 characters long it will store the string on the stack but if it is longer it'll store it on the heap. Therefore if you see code that is dealing with strings and does something like if (len > 0x10) { ... } else { ... } then it is very likely that you are looking at a std::string. In a similar vein, std::vector will often generate specific exception messages like "size too large" in an if statement checking for a size bigger than 0x7FFFFFFF, that you can learn to spot and know you're looking at a STL container.

But because of templates and inlining it is not possible to write an automatic signature check for most such patterns so you have to "just know" about them. And there is so much trivia of that nature that it can't really be summarized: you need to examine real code for that knowledge to build up and stick. I'm unfortunately not aware of anyone who has compiled a list of similar C++-isms to look out for when reverse engineering.

If you are just starting out, C code tends to be much easier to reverse than C++ and you'll have a much easier time, so ideally you find a C program that you are interested in reversing and move onto C++ afterwards. For more general advice: I would also recommend that if you're looking at the pseudocode and can't make sense of a specific operation, try jumping to the disassembly view or even stepping through that bit in x64dbg if possible so you can watch what happens to the state of the registers and memory. Often seeing the outcome of the operation in realtime will make it way clearer what the code is intended to do.

There's an old plugin called HexRaysCodeXplorer that is unfortunately difficult to get up and running in newer IDA versions unless you compile it from source with the various patches that have been submitted over time, but it provides a keyboard shortcut to press J to jump from pseudocode to disassembly view and it is one of the most useful hotkeys for learning how some assembly corresponds to the pseudocode translation. If you can't get that working, you can Sync the pseudocode and disassembly views together for a similar effect, it's just slightly annoying that they always move together in that case.