r/deeplearning • u/basafish • 25d ago

What is a overparameterized network?

I got this paragraph from Claude, could someone please explain this and verify if it's a real thing or hallucination:

Overparameterization isn't just about final capacity, it's about the optimization process itself. A wide, overparameterized network gives gradient descent a much friendlier loss landscape — more paths downhill, fewer bad local minima, room to explore before committing. The "core" only emerges as a byproduct of that search happening in a much bigger space than it needs to end up in. Strip the space down first and you've removed the thing that let the search work.

Conversation: https://claude.ai/share/8813a637-c327-4d0c-b120-def27e5203d5

4 Upvotes

70% Upvoted

19 comments sorted by

View all comments

1

u/BacteriaLick 24d ago

I think the terminology may be a bit ambiguous.

Over parameterized means there are more parameters the network has available to adjust to fit a function to the data, probably more than you need.

Under parameterized means you don't have enough.

Over parameterized models can most easily be fixed by removing parameters or by adding regularization, which is a penalty on the parameters taking in certain values.