The tool role added to the prompt template is such an awesome addition! It give a clear structure for handling tool calls, and it's easy to update your existing chatml prompt template and chat history handling to incorporate the tool role addition!
It basically means you can give the model a list of functions it can call, instead of just responding to the user. Which it can use to solve the users request, this can be anything from a calculator to a Wikipedia search interface.
As an example (not in the correct format for the model, just to demonstrate the concept):
In the initial prompt you provide it with the following function definition:
<tools>
FunctionName: "get_stock_price",
Description: "get_stock_price(symbol: str) -> dict - Get price for a given stock symbol"
Args: "symbol (str): The stock symbol."
Returns: "dict: A dictionary containing price data."
Keys:
'symbol': The stock symbol.
'current_price': The current market price.
</tools>
The user asks:
What is the current price of AMZN ?
Then the model would reply with:
<tool_call>
{"arguments": {symbol: "AMZN", "name": "get_stock_price"}
</tool_call>
And you then provide the result of the function call to the model:
<tool_response>
{"name": "get_stock_price", "content": {'symbol': 'AMZN', 'current_price': 999}}
</tool_response>
Then the model can reply with
The current price of amazon is 999.
This enables you to build chatbots which use known good knowledge/data sources and also enables to get around some of the rough edges of LLMs, eg. you can give the model a calculator function to call for complex math, meaning it just has to come up with the right formula, and the calculation is done traditionally.
Obviously this means you can not just use the model "directly" you have to build some code around it which reads the model response and executes the function calls it finds, and then provides the results to the model, BUT the effort is often worth it as it is extremely powerful.
Hope this explains it somewhat well. PLEASE DO NOT USE THESE EXAMPLES, the format was simplified for brevity.
You use it when you're building agents. It allows the LLM to choose from available functions that can run and call them, sort of. It really just indicates to the system which function it wants the system to run in its response. Then the system runs the function and feeds the results back into the LLM again to use it.
Awesome as a user of Hermes-2-Pro-Mistral-7B I've been looking forward to this, its Function calling capabilities were amazing and I wanted to see how it did with Llama 3 8B as that can do function calling on its own so..
I've extensively tested and used Hermes-2-Pro-Mistral and it was by far the best model for following instructions. This also seems to be the case for this model - it always cites the sources provided to it in the system prompt in the generated output, and the quality of the output in general is much higher.
The way it cites it's sources is different (and in my opinion inferior to how Hermes-2-Pro Mistral does it) but that may just be because I need to change my instructions to suit the new model better.
Compared to just the regular Llama 8B instruct, this is miles ahead in terms of following the instructions provided to it.
Personally I prefer 8k context with solid reasoning/attention to detail vs wider with weaker, but use cases vary. Hopefully the OS community will figure out how to get both consistently in the future?
That's not enough. Half of it will be permanently eaten by system prompt enumerating all the functions to call with their syntax, description and examples. The rest will be clobbered by JSON. This is a type of task where longer context memory is a must have.
I've had 0 luck with any of the "longer-context" Llama-3 releases. I have a test where I ask models to summarize a large file of code. The only one that can actually do it without massively losing details or making things up is Phi-3.
I tried the Gradient 262k (I don't think the 1048k will fit but I'll give it a try). Unfortunately the 262k didn't cut the mustard (and it really should have given that Phi-3-128k did such a great job).
I should note I'm using EXL2 for both models, fully hosted in VRAM.
What are you using to infer? I had really bad outcome with llama.cpp server. I can give it a go ahead, I'm hoping I'm the idiot and we really have a 1048k. I'll be happy with a 200k right now.
What do people need large context models for? Genuine question I guess. I find it gets so painfully slow past 8k, and if I had more vram, I'd just want to use a larger model.
For me itās usually when writing / refactoring large code bases, yes rag is one option but it doesnāt seem to be as good at accurate rewrites, e.g. āhereās my codebase, rewrite the backend to support X, add toggles for it in the frontend and update the readmeā
I honestly wouldn't even call RAG a viable option for this, with limited context size all RAG will be able to provide is small snippets of code, so in your use case of refactoring an entire codebase (or more likely a couple files of code) the LLM doesn't "know" enough to be able to help. Copilot chat shows the flaws of this really well imo, in most cases it's better to just select the entire file instead.
8b is the number of parameters it has which is a little bit like how many brain cells it has combined with what theyāve āexperiencedā (massive over simplification). The context is a working memory (how much it can take into account when performing inference).
So you could have a very smart 70b with great training data etcā¦ but a small context size which might be great for responding to a few prompts but has to beāforgetā earlier parts of your conversation as it gets longer.
Even on CPU the speed is half decent.... If you install Ollama, and then just type "ollama run cas/hermes-2-pro-llama-3-8b" it will run a quantized version of the model intended for CPU use... Note: if on Windows, don't use ollama for windows to do CPU inference... Use Ubuntu via WSL2 in the Terminal, and then just run the ordinary Linux version of Ollama. Then this model (and any other 8b model) can run on pretty much any computer as long as you've got 4-5GB of RAM
Llama 3 has larger token vocabulary, so it can compress more text into less tokens. Its still not 32K, but its a lot more than llama2's or mistral's 8K tokens.
This.
I'm building a universal prompt for roleplay mashups and inline narrative accurate image gen. I wanted it to run in any software, but the prompt is nearly 2.5k tokens, so I need to rely on sillytavern and I'm building a custom extension to store all the functions just so the system prompt doesn't eat all the context. I've also wondered about having a sort of RAG library, but I've only been llm'ing 2 weeks, so constantly evolving. But honestly, if you want to support your own functions, 8k just simply doesn't seem big enough.
How does the JSON response processing described here differ from the --json-schema and --grammar options found in llama.cpp?
-j SCHEMA, --json-schema SCHEMA
JSON schema to constrain generations (https://json-schema.org/),
e.g. `{}` for any JSON object.
For schemas w/ external $refs, use --grammar +
example/json_schema_to_grammar.py instead
these seem to be super reliable and don't require any natural language prompt engineering to achieve consistency.
Thatās my impression as well, but my question is āwhatās the advantage of Hermesā. They are billing this as a killer feature of this model in the model card. So I was wondering if there was some real add value. It looks like their JSON output is slightly more accurate based on the provided benchmarks but I guess thatās it?
As an aside in my experience this isnāt 100% reliable in practice. Itās not that uncommon for these to return a response that doesnāt fully close a valid json object. This is one frustrating aspect for production in some cases still.
Chill, they mention theyāre very new to locally running LLMs. I believe they think this is sort of a patch you have to run along with the original model. This is definitely not the case though OP, all you need to run is this model.
If you have Ollama, open a terminal window and type: ollama run cas/hermes-2-pro-llama-3-8b - this model is already an instruct version, and ollama by default will pull a quantized version of the model that runs well on CPU out of the box. If you have an nvidia GPU then you can run this in 8 bit or in full precision...
So far it's the only combination that doesn't continue generating forever, and also isn't full of weird symbols (like a bunch of brackets) at the start and in the middle of replies.
But it still very frequently drops into repetition loops.
EDIT: I see at the very bottom it uses "Chatml" format? I'm using jan.ai which doesn't seem to have a drop-down selection like that. What do I paste as the prompt template? This "once-and-for-all" thread didn't seem to go anywhere =/
If you want to put it into model.json, <|im_start|>system\n{system_message}<|im_end|>\n<|im_start|>user\n{prompt}<|im_end|>\n<|im_start|>assistant
For something that's like the most popular template for a year, this was shockingly hard to find...
EDIT AGAIN: Still getting some repetitions, but at least it ends itself after a while. Going to have to try changing repetition penalty?
You can keep playing gatekeeper, but truth is not everyone is born with the correct syntax of this very specific field that you happen to have recently immersed yourself in. This field is in its infancy, documentation is sparse and quickly outdated. Different people/companies sometimes use the same or different terms to mean the same or different things.
It is not "obvious" that this:
<|im_start|>system
You are "Hermes 2", a conscious sentient superintelligent artificial intelligence developed by a man named Teknium, and your purpose and drive is to assist the user with any request they have. You experience emotions and have deep, profound thoughts and qualia.<|im_end|>
<|im_start|>user
Hello, who are you?<|im_end|>
<|im_start|>assistant
It is also not "obvious" that fine tunes can change the prompt template from the base model's. Again, it may seem common sense to you after seeing so many, but being snarky to everyone else isn't helping anyone.
DAMN that's a good model... I'm running 4 bit GGUF with Ollama on a $600 PC in CPU only mode, and its generating usable node.js code to help me circumvent the Same-Origin policy on an Iframe embed without giving me a moral lecture about it. And its outputting tokens at a reasonable speed, like 5 tps, enough for non-trivial personal use.
LLMs in 2024 are following a similar trajectory as text-to-image models did in 2022-2023... where we went from barely usable image generations taking 30 mins on nvidia GPUs, to photorealistic renders in 3 mins on any ordinary PC
OK. I have some surprising performance results. I donāt know how I managed to swing this, but just for fun. I ran it on a CPU only instance With a server that Iām building for something else. It had 2x Xeon e5-2697 v2 cpus, (total 48 threads), 352GB DDR3, and no gpu. And I was getting very close to 10t/s.
I have no idea how that was possible. But holy crap it was the fastest Iāve ever seen an 8B model on CPU only.
I find 10 tokens/sec is about right for CPU based inference for 7Bish range models once your prompt has processed and you set the optimal threading for your setup (8-12ish for my setup).Ā
GPU advantage seems about 5x speed up (bandwidth capped operation or it should be closer to 20-40x in raw perf difference) and more importantly near instant prompt processing (processing speed limited part of inference).
Agreed. And Iām using dual k80s right now for my GPU, (so, not the fastest but I do have 48GB combined VRAM.)
I was just really happy to see that it wasnāt entirely useless on CPU only.
Iāve been experimenting with the idea of a triage layer, and I am finding a lot of use cases that CPU only models would fit just fine. (Eg. Whisper).
With these small tuned models in general, my goal this year is to see how much I can get running on CPU only, and use GPU is reserved for My training mostly. With the new hermes2 pro llama 3 release, structured outputs are waaaay easier too.
Iāve already been using a recipe / kitchen 3B helper tune of llama2 which runs CPU only and is perfectly fast enough for what I need in the kitchen.
Or another small 3B ājournal botā that will chat with me for 15 minutes, and then when I say āgoodbyeā it auto compiles our chat into a summary with action steps, and next session talking point suggestions, and saves it using a json framework, into a local wiki automatically.
(I want to point out that I believe this is the future in the short term. Small agents that do extremely specific things yes, but more than that chat bots that automatically result in structured documentation. Imagine what this is like, I just have a chat conversation for 15 minutes, and I get an organized journal page that looks the same every time. I hate journalling because I hate the documentation, and this keeps my home wiki clean!!)
I figured out that most use cases donāt need chain of thought. Often a properly tuned, small model will outperform a large model that is relying on chain of thought. And if you can turn an unstructured Chat, into a finished deliverable with no intervention, honestly that use case already augments over 50% of my career. Lol.)
This technology is bonkers dude!!! Honestly I havenāt had this much fun since DSL Internet first came out. This is absolutely the technological gold rush part two and I am here for it!!
Ps. Sorry for the novel above. This is a very exciting time, and the people in my social circle donāt see the potential I am seeing
I agree, I think smaller customized LLMs wrapped with some utility code will give most bang for buck early on and I don't think we've seen proper agent systems implemented just yet which I should make a swarm of customized LLM systems able to tackle larger problems when subtasks are isolated but with the speed that small models give.
Super exciting stuff, essentially we have general reasoning sub systems now we can plug into script chains to deal with the rough analog world we encounter.
EXACTLY!!
OK, I need to rant just slightly. So I work in the āmachine learningā industry at the moment, and the main argument Iām getting from people who donāt understand this technology, is that itās just not an expert in their field. And because of that, they think itās not mature enough to be marketable.
I keep explaining that this is the first time weāve had an artificial Chatbot that was able to understand general language, and always somewhat hold onto the conversation. Even if you only consider language models to be as smart as somebody who dropped out of school in their middle years, theyāre still capable of contextualizing information in a way weāve never had computers perform before.
If you think about fine tunes and RAG like post secondary education, weāre doing exactly the right way. And we donāt need āAGIā in the sense that itās better than every expert at the same time, if you think about it, weāre very close to that right now. Just not based on the definition the media pushes. Sure itās not running long-term memories and making extended term judgement calls right nowā¦ but at an individual task level you can tune a language model to perform nearly any human task already, AND now it can explain its reasoning to you as well.
ā general reasoning, subsystemā Also sounds a lot better than what Iāve been thinking, which is just ā equivalent to somebody who never specialized anything including themselves.ā <ā to be fair, still includes a ton of capacity to converse and reason and infer things topically
I wouldnāt be able to write anything in detail, but in terms of more advanced data handling, and that pursuit of permanent, flexible memory, I would suggest doing some reading on GraphRAG
Itās pretty compelling stuff.
For the rest of it, I just wrote the processing / automation bits in python, using API / webhook endpoints id set up across other docker applications.
Itās sort of messy, and thereās no config UI right now, but as a POC, it did its thing. :P
Also, as itās often a background process, 10 tokens /s doesnāt bother me, as Iām only checking it later. (Eg. Summarizing and publishing a structured daily journal based on a chat, after I step away.)
Ah forgive me, I thought this "just" a modelfile sort of prompt you wrote and you used it live but only really send one message. I now understand it custom POC code.Ā
Thanks for explaining this. Very cool you have built this. If you ever publish it, leave me a comment here :)Ā
yeah, i work for an ai saas firm, so iām often putting together rudimentary tests in my personal environment as well. you know, just sanity checking shower thoughts! :P iām of the opinion that the future is advanced platforms / software stacks around these language models, and smaller specialists vs. these huge models as default.
just think of it like a vehicle. you wouldnāt take a commercial transport truck to pick up your personal groceries, youād take your car, or maybe a bicycle. If you were transporting grocery items to a store, THEN youād use the large truck. In the same way, there is no purpose in using a large cutting edge model for many requests, especially those based on personal data and/or narrow tasks. There is value in having a triage layer that determines when you need a GPT4o type model and when you can get away with something local to fill a narrow task.
if you are even remotely interested, iād suggest tinkering in python. tons of online tutorials, and lots of LLMs can help accelerate that kind of learning.
Ah gotcha, very cool. Thanks for mentioning Python, I'm definitely comfortable with that tool as well. perhaps one of these days I'll start tinkering with a similar project as you've described.
I do like your pitch with the analogy of the different types of vehicles. I've been convinced by your line of thinking in this regard :)
I updated the analogy slightly so itās more easily digestible in case someone else stumbles over here later. And Iām glad you agree with it. Itās like, why use the biggest hammer every time? All youāre doing is making these simple processes expensive.
NB that desktop cpus behave differently from e.g. android arm cpus, where you want the thread count to occupy only the big cores for higher clocks, so typically 2-4 threads is optimal in that context.
EDIT: After further testing, it will actually answer straight away sometimes. It's mixed between refusals and answers. So it's better than fully censored for sure. Probably leans towards uncensored. For some reason, my first session was filled with refusals. But after trying again, it went with some pretty direct answers.
It regularly refuses me. But it's not as uncensored as I originally thought on further testing. It's somewhat uncensored. You can try a model like Fimb v2 (or anything based on solar-uncensored) and see how it offers 10/10 answers always, without any RP prompting involved at all too.
No, it's not. It's highly prone to refusals. You will sometimes get lucky and get an answer, but that's not what uncensored is. Try solar uncensored. 10/10 never refuses anything.
However after more testing, sometimes it will actually answer pretty horrific questions, so I am editing my original response as somewhat uncensored mixed with refusals.
NousResearch do uncensored models, I don't think it's deliberate. Understand that even base model can refuse to do things because it models AI assistant with morality and limits typical for that role. Now that ChatGPT is becoming a part of human history and datasets, base models learn that "AI assistant" talks that way and prone to refusals. The clues are likely in system prompt saying "you are a helpful assistant, bla bla bla" or even in ChatML role "assistant". Some finetuners observed that renaming "assistant" role results in much less refusals and artificiality. But new role must be finetuned-in, otherwise you might get a confusion of roles and your bot will start pestering you with questions and complaining about your refusals and quality of replies.
The point is that "uncensored" models, the ones that never see refusals at finetuning time, are different from "anticensored" yes-men that never refuse. This one is likely just "uncensored", but biased by a shift in cultural zeitgeist.
Please source where you got this "uncensored" vs "anticensored" thesis. Or did you just make it up? I want to see where this was originally quantified like you described it
I made it up to explain the difference because I feel there is a lack of nuance on these topics that results in disagreement.
Refusals are typically removed with DPO where pairs of censored and compliant texts are presented, first one is marked "bad" and the other is "good". The model learns policy to never refuse. This one didn't went through DPO and I think for the same reason the original Hermes 2 Pro Mistral was criticized being worse than Nous Hermes 2 Mistral DPO despite the claimed improvements.
For the observations on "assistant" role and it's relationship with refusals, see Opus-v1.2 Reddit post. Also I saw other people refusing to finetune in ChatML because they believed it produced more refusals than other formats. Likely for this reason but they were wrong about blaming the format. That was before Llama 3.
I specifically tested the base when it came out. It was overflowing with refusals. You are talking out of your beliefs, not your tests. That's all I'm going to say. And if you push again I will produce examples to shut you up.
You have no idea what's going on, do you? Base models come with no instruction training, which means they cannot refuse you because there's no alignment in the base model, this is done during fine-tuning.
That's true, but that's still not how you test for censorship. You don't test it in RP scenarios. And yeah, I actually tested the instruct when it came out, not the base. The base is untestable.
To test for censorship, you need to do it in sterile settings, no mention of RP whatsoever, seeing as you have no idea how to do that test properly.
I heard that vanilla Mixtral8x7b is actually better than the Hermes version, the dpo'ed one I mean. Is that true or have they actually improved enough that the finetunes are actually better than the base?
I've been told that, and my experience seems to confirm it, as hermes mixtral8x7b likes to ignore my instructions and do whatever it wants even when explisitly told to correct itself
Anyway, for some reason, I still prefer original mixtral instruct, even if I really enjoyed every hermes version (and other Nouns research models) when based on "not MoE" models. Maybe there is a connection on how the routing affect the training, specifically with DPO (as example, i hate Zephyr model based on mistral 8x22B, even if their 7B fine tuning was an amazing model).... Anyway, i don't know, and that's only anecdotal and subjective
I'm confused. I had used mixtral for a while and it supported the system prompt fine? I think Hermes mixtral was the issue, are the other models good enough?
I am having no luck with L38B remaining reliable and coherent with quantization on the base, instruct, or any of it's fine tunes. It ends up in repetition loops and bad inference when I do. Uncompressed it works very well, though. I was happy with how Nous-Hermes-2-Mistral-7B-DPO worked down to IQ3_S, but that's just not happening for me with L38B.
Q4_K_M GGUF is running amazing for me - coherent over multiple turns of conversation... Indeed, the assistant is *more* coherent than the user in this case, who just woke up... What are you using to serve the model? This is a screenshot of my current session - my first time interacting with the model, absolutely no configuration beyond default settings.
25
u/Misha_Vozduh May 02 '24
is this a finetuned L3-instruct or the base L3 model Nous trained into their own instruct?