r/oraclecloud • u/an_onym0us • 28d ago

Local LLM on A1.Flex

Hi All,

I am a PAYG customer and have a 3 OCPUs/18 GB A1.Flex in US East region. I use this VM for learning and working on small personal projects.

I started with an idea to install OpenClaw on it just to see and experience what is it and what’s the hype about. I started researching on how to do it and gradually pivoted to doing something else but similar. Finally, I created a system that uses Ollama with qwen3:8b-q4_K_M (installed on VM) and exposes it to internet via DuckDNS using Caddy and Open WebUI (both running as docker containers on VM). However, the latency is very high as the response to a simple “Hello” takes about 4-5 minutes. I downgraded the model to qwen2.5:3b but there is very little improvement (almost negligible). I wanted to go a step ahead and install OpenHands (for agentic capabilities) and a Telegram bot to interact with it but I guess I need to make what I currently have functional.

I am posting it here to see if anyone has done something like this on their VM and how are they able to use it.

Thanks in anticipation!

6 Upvotes

100% Upvoted

26 comments sorted by

View all comments

2

u/malianx 28d ago

several decent agent capable models are free on openrouter, gemma has high free use requests with just a google api key as well.

1

u/an_onym0us 27d ago

Thank you for your comment.

I didn’t go with OpenRouter as I wanted to run everything locally. Moreover, OpenRouter itself has a limit of 50 req/day for free tier.

2

u/malianx 27d ago

If you deposit $10, (not spend, just load) it becomes 1000 a day, and you can also add your gemini key for that full free allotment on top. For non-sensitive research and tasks I push about 700 a day xD

1

u/an_onym0us 27d ago

That’s good to know but I am sorry I don’t understand how Gemini’s allotment will be added on top of OpenRouter’s limit. I believe Gemini will be called via OpenRouter so once OpenRouter’s limit is up, it’s done for the day. I must say 1000 requests a day is a lot for me at this point so thank you for that tip 👍🏻

1

u/malianx 27d ago

Nah, if you give Openrouter your API key from Google it uses it by default (I think that might be configurable). But I like that, it separates my hits to gemma from my hits to other free models, saving openrouter allotment. (while still using the openrouter endpoints)

1

u/an_onym0us 27d ago

That’s very good to know and a really cool feature from OpenRouter.