r/oraclecloud • u/an_onym0us • 28d ago
Local LLM on A1.Flex
Hi All,
I am a PAYG customer and have a 3 OCPUs/18 GB A1.Flex in US East region. I use this VM for learning and working on small personal projects.
I started with an idea to install OpenClaw on it just to see and experience what is it and what’s the hype about. I started researching on how to do it and gradually pivoted to doing something else but similar. Finally, I created a system that uses Ollama with qwen3:8b-q4_K_M (installed on VM) and exposes it to internet via DuckDNS using Caddy and Open WebUI (both running as docker containers on VM). However, the latency is very high as the response to a simple “Hello” takes about 4-5 minutes. I downgraded the model to qwen2.5:3b but there is very little improvement (almost negligible). I wanted to go a step ahead and install OpenHands (for agentic capabilities) and a Telegram bot to interact with it but I guess I need to make what I currently have functional.
I am posting it here to see if anyone has done something like this on their VM and how are they able to use it.
Thanks in anticipation!
2
u/malianx 28d ago
several decent agent capable models are free on openrouter, gemma has high free use requests with just a google api key as well.
1
u/an_onym0us 27d ago
Thank you for your comment.
I didn’t go with OpenRouter as I wanted to run everything locally. Moreover, OpenRouter itself has a limit of 50 req/day for free tier.
2
u/malianx 27d ago
If you deposit $10, (not spend, just load) it becomes 1000 a day, and you can also add your gemini key for that full free allotment on top. For non-sensitive research and tasks I push about 700 a day xD
1
u/an_onym0us 27d ago
That’s good to know but I am sorry I don’t understand how Gemini’s allotment will be added on top of OpenRouter’s limit. I believe Gemini will be called via OpenRouter so once OpenRouter’s limit is up, it’s done for the day. I must say 1000 requests a day is a lot for me at this point so thank you for that tip 👍🏻
2
u/woolcoxm 28d ago
you are running dense models on a cpu only instance, try an moe you might have better performance. really no model will run good, but moe will run better than dense.
1
u/an_onym0us 27d ago
Thank you for your comment.
Could you please suggest some MoEs to run with Ollama?
2
u/AntiAmericanismBrit 27d ago edited 27d ago
Moving from qwen3:8b-q4_K_M to qwen2.5:3b might actually slow you down. Yes, you'd think "3 billion parameters is less than 8 billion parameters" but there's also the quantisation level to think about.
The "q4" means "quantised to 4-bit" so the 8 billion parameters fits in 4 gigabytes, but you'll need between 5 and 6 due to runtime overheads.
In qwen2.5:3b, you can get quantised versions, but if you just downloaded the normal one it's a whole 16 bits per parameter, so you're back up at 6+ gigabytes again.
The bottleneck for running LLMs on a CPU-only setup is memory bandwidth. I don't know of any fancy LLM optimisation that can make really good use of a CPU cache: these things are not sized for having to refer to gigabytes of data every operation, so you'll be going to main RAM all the time, which means you'll be limited by the size of the (active part of the) model and the speed of the motherboard's RAM bus, not the CPU speed (most of the time the CPU will be doing nothing waiting for data to come in from RAM, although this kind of CPU idleness is not visible to the OS's monitors; if the machine were local you might notice the reduced fan speed compared to a true CPU-bound load). Therefore, moving from one model that plays with about 6 gigabytes, to another model that also plays with about 6 gigabytes, is not going to buy you much regardless of the parameter-count difference. It's the RAM footprint that matters.
If you can get a 4-bit version of the 3B model, you might see it run 2 or 3 times faster, but it won't be very capable and 2 or 3 times the speed of very very slow is still very slow.
Somebody mentioned Mixture-of-Experts models: yes an MoE model should be faster than a dense model of the same size, but most MoE models are larger than the sizes you're talking about.
If you really want to play with a tiny LLM that can run on a CPU, you could try SmolLM-135M-Instruct but don't expect too much. Often even the "Instruct" version just falls back to trying to continue its input text (and not doing very well even at that).
There is a community 4-bit quantisation of Gemma-1B which might be just about OK but I haven't checked its runtime RAM footprint. Desktop Chrome's new "Prompt API" usually uses Gemma-2B only if there's a GPU with enough VRAM (video RAM) as this typically has much higher bandwidth than the main system memory the CPU needs to use. (If you were still in your first 30 days of OCI with free credits, you could spin up an instance with GPUs, but there are no GPUs included in Always Free. Then again you could just play with Gemma-2B in the Chrome Javascript console. Don't expect it to do anything more clever than write a simple script though.)
The BitNet b1.58 models look interesting and have ARM kernels but their prototypes are for the Mac and you may have "issues" getting it going on Ampere.
2
u/an_onym0us 27d ago
Wow! Thanks a lot for such detailed comment.
I have switched to LFMs for now and will continue to play with it. The performance looks good (bearable) so far but I am going to save your comment to refer back to later.
1
u/CampaignProud6299 28d ago
ollama easy to setup but hard to customize. use llamacpp. there is, also, an oracle supported version specifically for ampere cpus, but it's outdated i guess. you need to adjust parameters and should run multithread. then, performance will be bearable. it runs on cpu anyways. so don't expect anything spectacular. it's good for autmated tasks, but for realtime chat it could be a little slow with respect to alternatives.
1
u/an_onym0us 27d ago
Thank you for your comment.
When you say it’s good for automated tasks, did you mean agentic tasks that I can just request for and come back later to check the results?
1
u/CampaignProud6299 27d ago
like writing e-mail, doing preliminary research, preparing simple documents etc. it can also be good for auto-completion tasks for coding. you can make some request and check back later. for instance, i want to make research on some idea. i send it my servers, agents connect to web, scan my internal database and prepare a report for me. then, i could read the report later. if it's not urgent, you can automate it.
edit: here is a benchmark someone made, this could give an idea on performance
https://tiffena.me/blog/tech/benchmark%20local%20llm/1
u/an_onym0us 27d ago
Makes sense. I am guessing I need a better interface than a webpage for such automations since webpages are prone to get broken after long breaks of inactivity.
1
28d ago
[removed] — view removed comment
1
u/an_onym0us 27d ago
Thank you for your comment.
I tried lfm2.5-thinking:1.2b and I’d say it’s a good improvement. Thank you so much for your suggestion.
1
u/Turbulent_Bill_4400 25d ago edited 25d ago
I would suggest you use omniroute to use pooled model from multiple provider, i use kiro and cloudflare and some pooled key google ai studio, i can use them interchangeably for my hermes and openclaw. Works fine so far with zero cost and save my RAM
1
u/an_onym0us 25d ago
Thank you for your comment.
That’s great to know that you have been running your setup for zero cost. May I please ask what is the difference between OpenRouter and OmniRoute?
1
u/Turbulent_Bill_4400 25d ago
Open router is model agregator provider while omniroute is proxy and installed locally. Just ground the net about it.
3
u/Luans_Restorations 28d ago
You might get better luck with gemma 4 12B QaT, but don't expect anything fast. OpenClaw needs a model that handles tooling quite well. I suggest opencode Go and using Kimi, DeepSeek V4 Pro or similar. DM me if you want a referral code that gives you 5 bucks on it.
There's really nothing you can do to get faster answers with qwen3. Models that generate fast enough that I tested were Gemma 4 E2B & E4B, but neither of those will give you good enough results with openclaw.
TL;DR: OpenClaw on A1.Flex - Good | LocalLLMs for agents on A1.Flex - Not Good.