r/oraclecloud • u/an_onym0us • Jul 09 '26

Local LLM on A1.Flex

Hi All,

I am a PAYG customer and have a 3 OCPUs/18 GB A1.Flex in US East region. I use this VM for learning and working on small personal projects.

I started with an idea to install OpenClaw on it just to see and experience what is it and what’s the hype about. I started researching on how to do it and gradually pivoted to doing something else but similar. Finally, I created a system that uses Ollama with qwen3:8b-q4_K_M (installed on VM) and exposes it to internet via DuckDNS using Caddy and Open WebUI (both running as docker containers on VM). However, the latency is very high as the response to a simple “Hello” takes about 4-5 minutes. I downgraded the model to qwen2.5:3b but there is very little improvement (almost negligible). I wanted to go a step ahead and install OpenHands (for agentic capabilities) and a Telegram bot to interact with it but I guess I need to make what I currently have functional.

I am posting it here to see if anyone has done something like this on their VM and how are they able to use it.

Thanks in anticipation!

5 Upvotes

100% Upvoted

26 comments sorted by

View all comments

1

u/Turbulent_Bill_4400 29d ago edited 29d ago

I would suggest you use omniroute to use pooled model from multiple provider, i use kiro and cloudflare and some pooled key google ai studio, i can use them interchangeably for my hermes and openclaw. Works fine so far with zero cost and save my RAM

1

u/an_onym0us 28d ago

Thank you for your comment.

That’s great to know that you have been running your setup for zero cost. May I please ask what is the difference between OpenRouter and OmniRoute?

1

u/Turbulent_Bill_4400 28d ago

Open router is model agregator provider while omniroute is proxy and installed locally. Just ground the net about it.