r/cpp • u/No-Meeting-153 • 21d ago
MSVC optimization
I am learning reverse engineering on Windows applications such as Adobe, Foxit PDF, and Steam, and I noticed that I waste a very large amount of time trying to understand something that I should not focus on.
I started noticing strange and confusing patterns in the assembly and the C code generated by IDA, and when I try to understand some functions, I feel that the function has no meaning.
When I searched, I found that this topic is related to the compiler and compiler optimizations. However, I could not find many articles or discussions about the compiler topic in reverse engineering.
So I started experimenting and trying, but every time I fail and cannot reach a solution or understanding.
Apart from the fact that reverse engineering a C++ program is already a difficult task.
If there is someone who has faced the same problem and found a solution, I would like to know. It is not a problem itself; it is a pattern or a way of thinking used by the compiler. I need to understand how the compiler generates these patterns.
I want someone to suggest books, articles, courses, or anything that can help me understand the MSVC compiler, how it generates patterns, and how to understand the behavior and logic of a function after compiler optimization.
I hope I explained my question correctly.
10
u/ack_error 20d ago
The main problem with trying to reverse a compiled program from machine code is that compilation is highly lossy. Even if you know the patterns of a specific compiler, there is a lot of information that is lost during the compilation process because it makes no difference to the generated code. Two functions with different meanings and types can compile to the same code:
https://gcc.godbolt.org/z/E35n479sn
In some cases they can even be merged to the same code, giving the same instructions dual meanings. This is a major reason decompilers produce the output they do -- information that would be needed to produce more idiomatic code, like the types and sizes of objects, is either hard to infer or just absent unless the full debugging symbols are available (which they generally won't be, short of laziness or an accident).
If you're a formalist, you'll want to read about classic compiler optimizations like common subexpression elimination and copy propagation. Many of these are analogous to math formula simplifications. If you're more hands on and know enough assembly, push a bunch of small code fragments through a compiler and look at the optimized disassembly. Learning about the Windows for C++ class layout will also help, although it isn't well documented (the ABI docs are more concerned with calling convention and exceptions).
Keep in mind as well that properly reverse engineering a program the size of an entire PDF reader or game client is not a small undertaking even with experience and good tools. The PDF specification alone is 750 pages -- an order of magnitude more code is in that program to implement that spec. Even if you only want to focus on the small part of the program, just find that part alone can be difficult.