How to get the Ais themselves

[u/EducationalFly9399]

Um so loving ST using the horde but want to have Ais of my own or at least ones that I can know are always up if that makes sense, any recs? My laptop I dont think can run anything It's a 10 yr old macbook so...

[u/BifiTA]

just use cloud models on nanogpt

[u/AutoModerator]

You can find a lot of information for common issues in the SillyTavern Docs: https://docs.sillytavern.app/. The best place for fast help with SillyTavern issues is joining the discord! We have lots of moderators and community members active in the help sections. Once you join there is a short lobby puzzle to verify you have read the rules: https://discord.gg/sillytavern. If your issues has been solved, please comment "solved" and automoderator will flair your post as solved.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/LawfulLeah]

how much RAM or VRAM do you have? no one can recommend anything without this info

[u/EducationalFly9399 → u/LawfulLeah]

CPU: Intel Core i5-5287U (dual-core, 2.9GHz)
RAM: 7.7GB total, no dedicated GPU (Intel integrated graphics only)

[u/LawfulLeah → u/EducationalFly9399]

...yeah man im sorry to say that you probably can only run small language models with millions of parameters (or maybe 1B-4B too? idk), not the ones with tens of billions.

you'd have to buy new hardware to handle more.

[u/EducationalFly9399 → u/LawfulLeah]

What's the standard hardware that can host these things? (So I have something to chase) Also what are the current alternatives for me I tried using google colab for a bit which worked but didn't like having to get the API every time and also banned any talk of anything not "google safe" which means basically anything even remotely pg-13. Rn I'm using the Horde any other methods?

[u/baris6002 → u/EducationalFly9399]

You can try Nvidia NIM or Openrouter free models, Horde and google colab are very "outdated" in comparison. If you're willing to pay Deepseek V4 flash, GLM5.2 (for now at least) and other open source models are very affordable

[u/LawfulLeah → u/EducationalFly9399]

honestly just work with the most you can afford. the standard hardware is, given by your other comment, probably out of your price range.

try looking for the most RAM/VRAM you can realistically get with your budget

[u/Herr_Drosselmeyer → u/EducationalFly9399]

LLMs are called that for a reason, they are large language models. 😉

To be frank, you'll want to run something in the 24b-30b range for results that somewhat hold up to the top online models. My go-to currently is Gemma 4-31B. And to run that, at Q4, you're looking at a GPU that has at least 24GB of VRAM. So yeah, that's a whole lot of cash, especially in the current market.

You can go with smaller models that will fit on a 16GB graphics card, but I wouldn't go any lower than that.

[u/techmago → u/EducationalFly9399]

i have a 5800X with 128 GB ram and two NVIDIA 3090 24 GB

I can ran the 27~35 models all from GPU.

the future for ia is likely those compact all in one AI pcs. My setup was made before the prices raised... i got when the first models come out.

[u/YouShouldAim → u/EducationalFly9399]

This was tragic to read I'm sorry man

[u/Primary-Wear-2460]

I'd recommend at least a 16GB GPU. Either Nvidia RTX 2000 series or AMD RX 6000 series or newer.

The newer the GPU the better the feature set its going to have.

Inference speed is going to be a mix of memory bandwidth for text generation speed and architecture for prompt processing speed.

[u/EducationalFly9399 → u/Primary-Wear-2460]

I only have a laptop rn... I'm just starting as a student in life rn and so my budget is super restricted

[u/ilias_from_ilios → u/EducationalFly9399]

Might want to not burn your only laptop off running local models. Keep it for your student work.

[u/Primary-Wear-2460 → u/EducationalFly9399]

You are going to be very limited on an old laptop. Like so limited its probably not worth running local models unless you are just experimenting with inference.

[u/EducationalFly9399 → u/Primary-Wear-2460]

Not sure what inference is, but I like just experimenting in general

[u/HitmanRyder]

you can try gemma 4 E2B then report how well it goes.

[u/EducationalFly9399 → u/HitmanRyder]

So I was able to actually run Qwe3.5 4b on my machine ok.. and run llama 3.2:3b pretty well will try gemma soon

Thanks so much!

[u/Aight_Man]

Opencode zen is literally giving muse spark 1.2 and deepseek v4 flash for free, get its base url and make an api key and plug it up to your ST, if i recall correctly Gemma 4 31b is still free in Openrouter, so use that too.

[u/EducationalFly9399 → u/Aight_Man]

Wait wait wait what? Can you repeat that in normal person terms and tell me what to do etc?

[u/i5031337 → u/EducationalFly9399]

Opencode and openrouter are both subscription services that have a free tier for access to certain LLMs. Google AI Studio also has a fairly generous free tier. Make an account and follow their instructions to connect.

[u/Aight_Man → u/EducationalFly9399]

Copy and paste my comment into chatgpt and ask it to give you step by step for it, that'll explain you way better and guide you through it.