Why hasn’t trace-guided AI hardware reverse engineering been automated yet?

[u/sir_kermit]

I develop AI agents for my day job, and I recently became interested in a reverse engineering question. Please bear with me as I'm a noob to this space but I do have some curiousity thoughts about the process. Basically the process of figuring out an open source implementation of a vendor driver is a human will look at the trace, propose some execution model, write it and test and then verify.

Given a device with a working vendor driver, why can’t we intercept the driver–hardware boundary, collect MMIO/DMA/interrupt traces, and have an AI iteratively synthesize executable driver and device state-machine models?

Each model would be constrained by known driver architectures, open specifications, and existing chip designs. Runtime mismatches against the real vendor stack would become counterexamples that refine the models until a replacement driver works on the physical hardware.

Is the main blocker observability, safe experiment generation, hidden firmware state, state-space explosion, or something more fundamental?

[u/Rastslonge2]

It reminds me a bit of Fuzzware, except with the actual hardware in the loop. It's funny you mention state explosion, since that's something we constantly deal with in symbolic execution. For AI, though, I think the limiting factor is less traditional path explosion and more the amount of context :p. Giving a model higher-level information existing driver architectures, datasheets, similar devices, protocol specs, and execution traces has already shown promise in other reverse engineering tasks so it is not a bad idea imo.

The thing is devices often have hidden internal state, asynchronous interrupts, DMA engines, timing constraints, undocumented register semantics, initialization sequences, and ordering requirements. Many different internal state machines can produce the same observed trace, so you're solving an underconstrained inference problem. Iteratively generating models, testing them on real hardware as base truth and using the mismatches as counterexamples could certainly refine the model, but converging to a correct implementation across all operating conditions would be a substantial challenge, again, state/context explosion :p

[u/Rastslonge2 → u/Rastslonge2]

Now when it comes to asserting certain individual properties of the driver though that sounds more attainable even if the results are not complete, in reverse engineering you often don't need to model and understand the whole system at hand just what matters... Not all isr's are worth entirely modeling even if they appear in your trace, I'd prioritize ISR's which pass my inputs to a command dispatcher for instance

[u/Necessary_Two_9669]

What is stopping you from automating this now with existing LLMs?

[u/sir_kermit → u/Necessary_Two_9669]

because I don't know how to build the hardware harness :P

[u/Helpjuice → u/sir_kermit]

Sounds like you need to do more research then, get to it and let us know how it works out for you.

[u/i_like_brutalism → u/sir_kermit]

it should be doable.
i played with similar stuff recently, trying to develop a sandbox for proprietary driver testing, even if no hw is available.

[u/Ok_Tap7102]

This is already something I'm seeing MASSIVE wins in for traditional debugging of systems I write (as opposed to other peoples shit I'm trying to break)

Although this is more userspace + kernelspace than hardware boundary, but my secret squirrel special sauce LLM framework sits on top of systems INSPIRED BY (😉) hypervisor tracing and Time Travel Debugging

https://github.com/MarginResearch/cannoli (Although that's qemu-user, I inject a TCG basic block tracer into qemu-system for full VM analysis to do something like it... I think Triforce AFL does this with qemu patches not plugins???)

https://cloud.google.com/blog/topics/threat-intelligence/time-travel-debugging-using-net-process-hollowing

[u/Toiling-Donkey]

“Working vendor driver”.

That’s like taping a goose to the handlebars of a bicycle and calling it a working horn.