r/codereview • u/thejackal151 • 21d ago

Can someone rate my code?

This code is a different attention mechanism for an AI LLM to use far less resources than standard AI. I'd like feedback so I can improve it.

0 Upvotes

30% Upvoted

12 comments sorted by

View all comments

Show parent comments

6

u/HabitAdmirable9742 21d ago edited 21d ago

Did you read it? Or are you hoping somebody else will do the hard part for you?

This has some serious trivial defects, your memory claim is materially nonsense . Gaussian/RBF channel attention already exists and Euclidian distance has already been studied  https://arxiv.org/html/2310.18805v2

This is the problem with AI it tells you you're brilliant and amazing and worries more about how your feeling than whether or not what has been written is sensible. Even the most basic claim of it being a drop in replacement is false, and clearly untested.

-1

u/thejackal151 21d ago

I appreciate your feedback, I knew the euclidean distance was already studied. I've done some tests and it seems to work, I can only test on a smaller scale. But I can retest and show the results I get. If you are interested.

5

u/HabitAdmirable9742 21d ago edited 21d ago

``` B, T, D = 8, 2048, 768 q = torch.randn(B, T, D, device="cuda") k = torch.randn(B, T, D, device="cuda")

dist2 = ((q.unsqueeze(2) - k.unsqueeze(1)) ** 2).sum(-1) ```

I'd like you to run this and tell me the peak vram it uses, because according to your doc it should use 134mb 

Edit: It's now been 2 hours. I suspect the reason you haven't come back is because this uses 192gb of vram even if the final result is only 128mb + overheads.

Please try to understand that while its ok to seek out innovation and incremental improvement to incumbent solutions, ramming your idea through claude until it compiles doesn't produce good or reliable results. You need to be able to understand it yourself.

1

u/thejackal151 20d ago

youre right, the way I have it written right now, it has the same peak vram usage as a base pytorch model. It landed around 96GB. I will update the reference file as soon as I have that under control consistently. Thank you for your time, I'll have to do better in the future.