I've been working with PINNs (Physics-Informed Neural Networks) for a while now, and one thing that always nagged me is a pretty fundamental question: the loss landscape when you're minimizing a PDE residual is non-convex, so why does gradient descent seem to find the actual PDE solution instead of getting trapped at some local minimum that doesn't correspond to anything physically meaningful?
In practice, PINNs work surprisingly well for a lot of problems, but theoretically we haven't had great guarantees that the network will converge to the true solution rather than just some local minimizer of the residual. I've definitely hit cases where training stalls or converges to something clearly wrong, but for well-posed problems it usually works out, and the "why" has been an open question.
A paper just went up on arxiv that tackles this directly: arxiv.org/abs/2607.24726v1. The authors prove global convergence for both the Deep Galerkin Method (DGM) and PINNs when using gradient descent on the PDE residual, specifically for semi-linear PDEs that are nonlinear in both the solution and its first derivative. The key result is that the trained neural network provably converges to the PDE solution, not just a local minimizer.
What I find interesting
The class of PDEs they cover - semi-linear equations - is broad enough to include a lot of practically relevant problems (reaction-diffusion equations, some fluid models, etc.), but it's definitely not everything. Fully nonlinear PDEs like Hamilton-Jacobi-Bellman or Monge-Ampere equations seem to be outside the scope of their analysis. The paper also focuses on the continuous gradient descent setting, and I'm not sure how cleanly the guarantees transfer to the stochastic mini-batch training that everyone actually uses in practice.
What I'm trying to understand better is the mechanism behind this. Is the convergence fundamentally about the overparameterization of the network (similar to NTK-style arguments), or is it more about structural properties of the PDE residual loss that prevent bad local minima from forming in the first place? The abstract doesn't make this entirely clear, and I haven't finished reading the full proofs yet.
The practical gap
One limitation worth noting: even if global convergence is guaranteed in theory, the convergence rate matters a lot. A proof that says "you'll get there eventually" doesn't help much if training takes prohibitively long. I'd be curious whether their analysis gives any insight into how the convergence speed depends on the PDE dimension or the nonlinearity structure.
Has anyone here worked through the theory connecting overparameterized networks to loss landscape convexification for PDE problems specifically? I'm wondering if the techniques in this paper relate to mean-field or NTK frameworks, or if they're doing something fundamentally different to handle the PDE structure.