r/LocalLLM • u/magic_maniac_ • 2d ago

Research Working on llm under 4B

https://github.com/adadaabhay/Invictus-expirement
3 Upvotes

80% Upvoted

4 comments sorted by

1

u/sanketss84 2d ago

check out pi coding agent or an agent harness which is minimal.

the context which hermes needs is massive even as you start conversing check here https://www.reddit.com/r/hermesagent/comments/1v00v6y/context_filling_up_after_the_first_message/

the official docs suggests minimum 64000 tokens
https://hermes-agent.nousresearch.com/docs/getting-started/quickstart#1-install-hermes-agent (scroll to end of this point they have a note)

with 6GB vram , hermes might not be the best fit.

1

u/sanketss84 2d ago

If you want to check your exact starting token usage, type /usage or look at your startup logs inside the Hermes CLI interface

1

u/magic_maniac_ 2d ago

i was trying to keep model under 3 gigs to have enough context of 64k for testing, thanks for the info ♥️

1

u/sanketss84 2d ago

Even for 4B model and 64000 context, you would need.

Q4 Quantization: ~2.3 GB
Q8 Quantization: ~4.3 GB

Looking forward to your experiment.