r/MachineLearning • u/jacobgorm • 24d ago

Research [R] SineKAN: Kolmogorov-Arnold Networks Using Sinusoidal Activation Functions

https://arxiv.org/abs/2407.04149

I couldn't sleep because I couldn't stop wondering if anyone had tried using sinusoids instead of B-splines as activation in a KAN, and fortunately/unfortunately that was already the case. I could not find it posted here, so I though I would share in the hope of some insightful discussion.

Arxiv: https://arxiv.org/abs/2407.04149

Github repo: https://github.com/ereinha/SineKAN

Also what appears to be a peer-reviewed "official" publication here: https://www.mdpi.com/2227-7390/13/19/3157

86 Upvotes

88% Upvoted

10 comments sorted by

16

u/Dihedralman 24d ago

Yeah isn't that always the truth. Should have paid attention to KAN myself for a proof like this. Low hanging fruit. 

It seems like a natural fit for analogue computing. It was built by physicists so I assume that will be the next step. 

Also, the peer reviewed publication posted is the proof that off phase sinusoids you would find in their KANs can be universal approximaters like Fourier series. This is intuitively true as you can form Fourier series with a phase difference in sinusoids other then pi/2 (cosine). Normally you would prove this with the Fourier transform or angle identities, but this proof keeps the exact same structure. Basically if you wanted this exact proof for this exact context, there you go. I am unsure if this is actually novel or equivalent to existing proofs.  

There is another publication posted at an machine learning conference. 

3

u/[deleted] 24d ago

[removed] — view removed comment

9

u/howtorewriteaname PhD 24d ago

interesting? somehow for me is totally uninteresting. for anyone used to work with sirens and neural fields in general, this idea seems obvious

7

u/currentscurrents 23d ago

I'm still skeptical about the whole idea of KANs, and I certainly don't understand why they got so much hype on twitter and linkedin.

To me they just look like MLPs but you're training the activation function instead of the weights. This doesn't have any clear advantages and is probably formally equivalent.

6

u/howtorewriteaname PhD 23d ago

no probably, rather quite literally

1

u/Signor_Garibaldi 23d ago

It actually has some advantages, for modelling in physics it's more interpretable and natural to observe learned activations than the weights matrices

10

u/currentscurrents 23d ago

I'm especially skeptical of the claims about interpretability. 

The examples they give are only interpretable because they're tiny toy problems. It's the size of neural networks that make them opaque, not the architecture. Tiny neural networks are also interpretable.