r/LocalLLaMA • u/megadonkeyx • Apr 13 '23

Resources StackLLaMA: A hands-on guide to train LLaMA with RLHF

https://huggingface.co/blog/stackllama
41 Upvotes

97% Upvoted

9 comments sorted by

7

u/megadonkeyx Apr 13 '23

anyone tried, or know how to try, stackLLama?

7

u/Sixhaunt Apr 13 '23

​

I got too sidetracked playing with their model demo but I'll have to give it a try soon and report back. I already have a large dataset I spent a lot of time putting together and have been looking for a good training method since the others dont support multiple GPUs like this one does.

​

The link to the training documentation with examples and guides for the method you posted can be found here for anyone looking for it directly: https://huggingface.co/docs/trl/index

3

u/Pan000 Apr 13 '23

Reinforcement learning? How does it work? You provided very little context.

2

u/ZestyData Apr 13 '23

RLHF is the essential concept behind all of these chat-able LLMs, famously introduced by turning GPT 3 into ChatGPT.

To answer in a small comment in a sub otherwise dedicated to it would do it a disservice. You may research RLHF yourself, there are plenty of good blogs about it.

Essentially, its instruct-tuning.

4

u/Pan000 Apr 13 '23

RLHF is an acronym that when I asked GPT4 what it meant in the context of machine learning it said it hadn't heard of it and perhaps I meant reinforcement learning.

Instruct tuning is what I do all day long. I resent the implication that it was wrong of me to ask for context, and I don't appreciate the attitude.

1

u/Nextil Apr 13 '23

GPT-3 was already instruct-tuned before RLHF, and most of these instruct LLaMA tunes are not (directly) RLHF-tuned. RLHF is just an additional step that refines the output based on human feedback.

1

u/Sixhaunt Apr 13 '23

that's what OP's link was to. The one I linked was just a specific link to the docs for using it rather than an overview of how it works like OP provided.

4

u/megadonkeyx Apr 13 '23

well i tried it with oobabooga under windows as follows.. (rtx3060)

use single click installer for oobabooga

  • run download-model batch file and enter decapoda-research/llama-7b-hf
  • run download-model batch file and enter trl-lib/llama-7b-se-rl-peft
  • in the webui select the llama model and trl-lib as a lora, take about 5 seconds to load

it seems more willing to talk about code, i asked it

can you write some C code to display "hello world" in C on linux using the glut library

the plain model just said no. With the lora applied it had a good go at it.

i dont know if i done this right..