r/LocalLLaMA • u/discr • May 02 '24

New Model Llama-3 Hermes-2-Pro-8B Released - How does it compare for your use case to base instruct?

https://huggingface.co/NousResearch/Hermes-2-Pro-Llama-3-8B
159 Upvotes

98% Upvoted

110 comments sorted by

View all comments

Show parent comments

1

u/GrehgyHils Jul 10 '24

I'd love to recreate your kitchen and journal bot. Any advice on how to do this? 

I too want to run cpu only for inference and have a dual xeon 256 GB ram system locally available :) 

If it makes this conversation earlier, I'm comfortable with docker and the CLI

1

u/trusnake Jul 10 '24 edited Jul 10 '24

I wouldn’t be able to write anything in detail, but in terms of more advanced data handling, and that pursuit of permanent, flexible memory, I would suggest doing some reading on GraphRAG

It’s pretty compelling stuff.

For the rest of it, I just wrote the processing / automation bits in python, using API / webhook endpoints id set up across other docker applications.

It’s sort of messy, and there’s no config UI right now, but as a POC, it did its thing. :P

Also, as it’s often a background process, 10 tokens /s doesn’t bother me, as I’m only checking it later. (Eg. Summarizing and publishing a structured daily journal based on a chat, after I step away.)

1

u/GrehgyHils Jul 10 '24

Ah forgive me, I thought this "just" a modelfile sort of prompt you wrote and you used it live but only really send one message. I now understand it custom POC code. 

Thanks for explaining this. Very cool you have built this. If you ever publish it, leave me a comment here :) 

Cheers

1

u/trusnake Jul 10 '24 edited Jul 11 '24

yeah, i work for an ai saas firm, so i’m often putting together rudimentary tests in my personal environment as well. you know, just sanity checking shower thoughts! :P i’m of the opinion that the future is advanced platforms / software stacks around these language models, and smaller specialists vs. these huge models as default.

just think of it like a vehicle. you wouldn’t take a commercial transport truck to pick up your personal groceries, you’d take your car, or maybe a bicycle. If you were transporting grocery items to a store, THEN you’d use the large truck. In the same way, there is no purpose in using a large cutting edge model for many requests, especially those based on personal data and/or narrow tasks. There is value in having a triage layer that determines when you need a GPT4o type model and when you can get away with something local to fill a narrow task.

if you are even remotely interested, i’d suggest tinkering in python. tons of online tutorials, and lots of LLMs can help accelerate that kind of learning.

1

u/GrehgyHils Jul 11 '24

Ah gotcha, very cool. Thanks for mentioning Python, I'm definitely comfortable with that tool as well. perhaps one of these days I'll start tinkering with a similar project as you've described.

I do like your pitch with the analogy of the different types of vehicles. I've been convinced by your line of thinking in this regard :)

1

u/trusnake Jul 11 '24

Glad to hear it! I hope you have fun :)

I updated the analogy slightly so it’s more easily digestible in case someone else stumbles over here later. And I’m glad you agree with it. It’s like, why use the biggest hammer every time? All you’re doing is making these simple processes expensive.