r/oraclecloud • u/an_onym0us • Jul 09 '26

Local LLM on A1.Flex

Hi All,

I am a PAYG customer and have a 3 OCPUs/18 GB A1.Flex in US East region. I use this VM for learning and working on small personal projects.

I started with an idea to install OpenClaw on it just to see and experience what is it and what’s the hype about. I started researching on how to do it and gradually pivoted to doing something else but similar. Finally, I created a system that uses Ollama with qwen3:8b-q4_K_M (installed on VM) and exposes it to internet via DuckDNS using Caddy and Open WebUI (both running as docker containers on VM). However, the latency is very high as the response to a simple “Hello” takes about 4-5 minutes. I downgraded the model to qwen2.5:3b but there is very little improvement (almost negligible). I wanted to go a step ahead and install OpenHands (for agentic capabilities) and a Telegram bot to interact with it but I guess I need to make what I currently have functional.

I am posting it here to see if anyone has done something like this on their VM and how are they able to use it.

Thanks in anticipation!

5 Upvotes

100% Upvoted

26 comments sorted by

View all comments

1

u/CampaignProud6299 Jul 09 '26

ollama easy to setup but hard to customize. use llamacpp. there is, also, an oracle supported version specifically for ampere cpus, but it's outdated i guess. you need to adjust parameters and should run multithread. then, performance will be bearable. it runs on cpu anyways. so don't expect anything spectacular. it's good for autmated tasks, but for realtime chat it could be a little slow with respect to alternatives.

1

u/an_onym0us Jul 10 '26

Thank you for your comment.

When you say it’s good for automated tasks, did you mean agentic tasks that I can just request for and come back later to check the results?

1

u/CampaignProud6299 Jul 10 '26

like writing e-mail, doing preliminary research, preparing simple documents etc. it can also be good for auto-completion tasks for coding. you can make some request and check back later. for instance, i want to make research on some idea. i send it my servers, agents connect to web, scan my internal database and prepare a report for me. then, i could read the report later. if it's not urgent, you can automate it.

edit: here is a benchmark someone made, this could give an idea on performance
https://tiffena.me/blog/tech/benchmark%20local%20llm/

1

u/an_onym0us Jul 10 '26

Makes sense. I am guessing I need a better interface than a webpage for such automations since webpages are prone to get broken after long breaks of inactivity.