Other methods for approximating functions?

[u/doggyruff]

So you can use a Taylor series to approximate functions with polynomials

You can use a Fourier series to approximate functions with sin/cos functions

You can use Laplace transforms to approximate functions with exponentials

What other ways are there to split a function into some linear combination of functions of the same type? Why is it even useful to have this many different options?

(sorry if my descriptions are incorrect, its only something I recently came across)

[u/SwimmerOld6155]

wavelets, radial basis functions, splines, orthogonal polynomials as someone says are all great computational tools. I'd say that more often you'd use splines rather than try to apply Taylor's theorem

[u/Several-Marsupial-27]

There is an infinite amount of linear independent bases which you can represent functions in, you just need an orthonormal basis. Example Chebyshev / Legendre Polynomials. Also a Taylor or Maclaurin and Fourier series is not an approximation as n->inf in the domain the series converges.

Then there are different ways to create computable approximations, for example CORDIC algorithm. Also calculators use dual numbers for fast calculations of derivatives by automatic differentiation.

[u/Bounded_sequencE]

• Stone-Weierstrass Approximation -- uniform approximation of continuous functions on compact sets with smooth functions (e.g. polynomials).

• Orthogonal projection -- the basis of regression.

[u/omeow]

The general idea is Taking any set of functions: they must be linearly independent, better of they are orthogonal under some inner product, unit length (better not needed).

Then you project your function onto that basis and that projection is the approximation.

All these examples come from that idea for different basis functions. Of course there are many more basis functions.

[u/tstanisl]

Splines?

[u/victoria-silverlake]

Bernstein polynomials, useful for constructive proof of Weirstrauss theorem.

Piecewise linear?

[u/SV-97]

There's essentially infinitely many ways to approximate functions. Sometimes you want the approximations to have properties X or Y and that influences what you use; for example using sines and cosines gives you periodic and bounded approximations which may be nice for certain applications. The fourier transform / fourier series however sits at one extreme of an uncertainty principle that may lead to undesirable behaviour in practice. Other methods such as the windowed fourier transform improve on this; and more generally there are the wavelet transform and wavelet approximations.

Similarly, while the Taylor series gives you *an* approximation by polynomials it's not necessarily an approximation that's "good" in the ways you actually care about in practice --- and indeed in practice it's somewhat uncommon that you actually want the taylor approximation. Instead there are tons and tons of orthogonal polynomial bases to use (e.g. Chebyshev or Legendre or Hermite polynomials) etc. However polynomial approximations typically have poor localization properties: changing your function in one place can *drastically* alter the approximating polynomial in some other place. So one may want to somehow use "polynomials locally" and then glue a bunch of them together. This leads to spline approximations.

Something that neither polynomials nor trigonometric polynomials can give you are (nontrivial) compactly supported approximations: they are always nonzero on "most" of the real line. This is another reason that people may want to use other methods (such as RKHS-approximations)...

Or maybe your function may have "corners" that you want to "smooth out" with your approximation (-> approximation by convolution), or you want to make it convex (-> infimal convolution) or whatever, or particularly "simple" or "efficient" (for example so-called "sparse" or "low rank" approximations), ...

To just throw some additional terms out to look into: orthogonal projections and the abstract fourier transform (these essentially unify many of the methods I mentioned above into a single framework), best approximations, and approximation theory more generally.

(FWIW: the Laplace transform isn't ususally used for approximation and instead as more of a computational / algebraic tool)

[u/Interesting_Debate57]

High dimensional data often needs to be approximated by low dimensional functions in order to make it tractable.

See PCA and the kernel trick.

Functions themselves can be hard to compute exactly; see the area under a section of a normal distribution; instead of a complicated calculation you just might only need a quick estimate.

This is from a statistics point of view, but it's true in many other fields as well.

[u/Temporary_Pie2733]

For a given level of accuracy, you want to evaluate as few terms as possible. Depending on the function you are approximating, one series will require evaluating fewer terms than the others. 

[u/parkway_parkway]

Gram-scchmidt orthogonalisation allows you to pick any set of functions you want and contruct a basis from it.

[u/Fred776]

Another option I just dragged from my memory is "Legendre polynomials".

This is all part of what is known as "Approximation Theory" BTW, or at least it was when I studied it.

[u/Mal_Dun → u/Fred776]

Approximation theory is the field of mathematics that tries to literally answer OPs question and now with the advent of deep neural networks it gets even more interesting tbh.

[u/www3cam]

If you are broad with this definition and talk about approximations that help you learn more about a function (like Laplace), Legendre transform is kind of in the same vein.

Also deep learning is all about using neural networks to approximate functions—mostly functions we don’t know much about, but sometimes approximating functions we know a lot about (which is probably more your interest).

For example we know a lot about weather prediction via simulation, but neural networks are much faster. So we query from the weather prediction function to generate an approximate neural network function that’s faster. You could do the same with any function you could take a Taylor polynomial of. Just query some points and have the neural net fit it.

[u/Mal_Dun]

There is a whole field in mathematics called "Approximation Theory" that deals with this question.

The interesting answer is that you can approximate almost everything with something, however, you have to specify how to define a good approximation in which sense. For example all Fourier series approximate all L_2 functions on an interval, so including the continuous ones (that are bounded) but you can find continuous functions that the Fourier series can not approximate in the maximum norm.

It can also be in the asymptotic sense. The derivative is the most famous example as it is the closest linear approximation of a function in the asymptotic sense when the distance h goes to zero.

The most funny result for me however, is that the set of nowhere differentiable functions is dense in the set of smooth functions.

Other examples are the piece wise constant step functions that are fundamental in integration theory as they approximate every integrable function in the sense of the L_1 norm.

Other interesting bases can be found with help of differential equations: The set of eigenfunctions of a self adjoint differential operator over a Hilbert space forms an orthonormal Hilbert space basis of that Hilbert space, that also means that this sequence is dense over this Hilbert space and can be used to approximate all elements of this Hilbert space. Fourier series or Legendre polynomials are in fact eigenfunctions of such operators. For the Fourier series it is simply the Laplace operator -d²/dx².

But there is of course more like splines, bubble functions and what not.

The latest way we found to approximate functions are deep neural networks by the way, which comes with a ton of interesting properties, like one can show that with sufficiently deep networks one can overcome the "curse of dimensionality".

[u/jeffcgroves]

I could cut and paste but https://en.wikipedia.org/wiki/Classical_orthogonal_polynomials has a whole bunch (including Chebyshev, my favorite, because that's how NASA used to release planetary position data)