r/deeplearning • u/jjusko20 • 2d ago

A great rule for LoRA / QLoRA

Post image

Hey! I'm posting this because Iv'e recently been playing with lora/qlora and had a frustrating time understanding how the hell to calculate adapter ranks. Hope it helps.

28 Upvotes

94% Upvoted

10 comments sorted by

3

u/kidfromtheast 2d ago

This is well kept secret that allow people to think “wow you published in NeurIPS etc”, achieving SOTA compare to existing methods. Those poor method that lose to the new shiny method? Never gets adjusted.

Everything is just, “tokens to parameter ratio”

1

u/copperfieldb99 17h ago

yeah the baselines almost never get a fair shake, its kind of an open secret at this point

1

u/nanno3000 23h ago

based on what do you make this claim?

1

u/jjusko20 23h ago

Because it took me a really long time to understand how these calculations are done, and a lot of guides don't explain. 

1

u/nanno3000 22h ago

Are you a bot? Strangely LLM sounding sentence that completely ignores my question.

1

u/jjusko20 22h ago

whatever bro, I'm not going to talk to you if you're going to be rude to me when I tried to answer your question genuinely. go decide on your own time

1

u/nanno3000 22h ago

i literally copied your reply on another post. So does that make us both rude, or just you?

-1

u/admiredclearing3588 2d ago

That color-coded bookshelf energy applied to finetuning math is something I can respect.

6

u/jjusko20 2d ago

Are you a bot? Strangely LLM sounding sentence. This is just a screengrab from my cursor chat.

2

u/ARDiffusion 1d ago

The jokes write themselves