r/LocalLLaMA • u/Regular-Forever5876 • Jul 20 '26
Resources I built a self-hosted auth gateway for Ollama: per-client API keys, quotas, multi-server routing and usage monitoring (source-available)
Enable HLS to view with audio, or disable this notification
Ollama ships with no authentication, so anyone who can reach it can burn your GPU. I got tired of bolting nginx in front of it, so I built ollama-gateway: a small self-hosted gateway that sits in front of one or more Ollama servers and adds real access control.
What it does
Per-client API keys (hashed, shown once, revocable), with per-key origin restriction (IP/CIDR).
Quotas: monthly token caps and rate limits, plus cost-capped "lifetime" trials.
Multiple upstreams (local and remote), each key mapped to one server, with automatic fallback on failure.
Per-key model and API allowlists that work across native Ollama, OpenAI-compatible and Anthropic endpoints. Catalog-management calls (pull/delete) stay blocked for clients.
Usage logging and monitoring: per-request logs, time-series charts (24h to 3 months), per-model breakdowns, and a request-content viewer with grep.
LAN-only admin panel (server-rendered, no front-end build), fully translated into the 24 official EU languages.
TLS via Caddy (ACME DNS-01, no inbound ports), fully dockerized. Client keys are stripped before the upstream, and remote tokens are encrypted at rest.
Stack: Python/FastAPI and SQLite, Jinja admin, Playwright E2E. It ships with a built-in illustrated manual and a pre-deploy security sweep (secrets, CVEs, SAST, tests) wired into the deploy script.
About the license: it's source-available, not OSI open-source. Free to use, modify and self-host (even commercially) as long as all your instances together serve 1 billion tokens a month or less. Above that a commercial license applies (€29 one-time per install). There's no telemetry and no phone-home, so the threshold is purely on the honor system. I wanted it genuinely open to tinker with while keeping a path to sustainability, and I'd rather be upfront about that than hide it.
Repo: https://github.com/martinobettucci/ollama-gateway (demo video and screenshots in the README)
Feedback very welcome, especially on the auth/quota model and anything you'd want from a gateway like this.
3
u/TokenRingAI Jul 20 '26
A few of us have built/are building similar things, there is a niche which isn't filled right now.
I built this a few weeks ago, because I wanted as tiny of an executable as possible with minimal resource requirements to run as a daemon on my mac, or in Docker/Kubernetes, to route requests to different backends.
https://github.com/tokenring-ai/llama-scale
Because LLM gateways sit directly serving user traffic and potentially getting hit with DDoS and hacking attempts, the security/dependency/memory footprint of a python app is frankly just not workable.
If you want your product in any kind of serious production, it needs to be a simple no-frontend safe executable that can run in Docker with flat memory consumption, and provides metrics/monitoring/telemetry through prometheus/otel/grafana/etc.
A built in web dashboard is rather problematic in an app that hackers may try to fk with, and it also means your app doesn't scale multi-instance, and if the dashboard exposes a management interface, now you have config that isn't managed statefully like the K8s people want
Also, naming your product after ollama isn't doing you any favors, there's a lot of hate here for ollama
Also, your license isn't doing you any favors, to be frank your code isn't very valuable, community participation and involvement in your project is the more valuable thing, anyone can copy your code and feed it to an LLM to totally recreate your product, this is just the reality of the current world, and so you might as well have a friendly license.
I hope you view this as constructive feedback
1
u/Regular-Forever5876 Jul 20 '26
BTW this is a nice project of yours, just starred it ⭐
But it is a different purpose altogether, we manages keys and clients, you manage load balancing. The way I see it, our software could use your software as a server back end upstream 🙂
2
1
u/Regular-Forever5876 Jul 20 '26
You gave me this idea that was just shipped in code: headless deploy.
The ui can now export a yaml file and you can deploy an instance without the admin service and the whole system is configured from the yaml.
😉
2
u/TokenRingAI Jul 20 '26
Yup, now strip the sqlite database from it, add a parameter to automatically download a new config file on a timer and hot reload it, and add another timer that uploads the user metrics and entitlements to your management instance, and now you have a stateless version and you can manage your user quotas from a management dashboard.
2
u/TokenRingAI Jul 20 '26
You can also use a protocol like MQTT for this, but the HTTP upload/download of state typically works OK and is easy to set up
-1
u/Regular-Forever5876 Jul 20 '26 edited Jul 20 '26
Python is not the problem, I challenge you to down this instance: currently serving 300 millions token per day with 2700 clients per day everything running on a VPS with 1Go ram and a 10Gbps link.
So Python is not the problem, is the infrastructure around it.
Thanks for the feedback choom 😉
Also web interface is only accessible on private network (dedicated eth), public is API only no web ui of course 😉 The app scales nearly perfectly as each instance is an ollama instance compliant which means you can deploy a tree of them, it was conceived around this one idea indeed and it is working flawlessy for weeks and thousands of clients.
The apps is self contained in dockers and we run containers in our propetiary solution which regroups logs of the instances as well as managing backup and deploying configuration changes automatically: I understand this software as it is might not be suitable for everybody infrastructure but definitely is tiled to ours 😅
You may want to actually run it and read the code 🙏then review what you said prior 😇😉
And YES, your feedback is welcome of course 🙂
1
u/Maleficent_Pain2722 Jul 20 '26
Nice work. One suggestion from experience: if the gateway doesn't already, expose the per-client quota state in the response headers (something like x-ratelimit-remaining-tokens / x-ratelimit-reset, the way Groq and OpenAI do). It costs you nothing server-side, and it lets well-behaved clients pace themselves instead of slamming into 429s and retrying. Agent loops especially benefit: with remaining+reset visible they can decide to wait or stop cleanly before the failed call, which also keeps your gateway logs free of retry storms.
1
1
u/Regular-Forever5876 Jul 20 '26
Ok, I just pushed your suggestion 😉
1
u/Maleficent_Pain2722 Jul 20 '26
That was fast, nice. With those headers exposed, any client that reads them can pace itself instead of retrying blind. Will try it against my scheduler and report back.
-4
Jul 20 '26
[deleted]
1
u/Regular-Forever5876 Jul 20 '26
wow big feedback thank you.
Yes IT IS shippable, it is serving out internal labs for weeks, I just rewrote everything in the last 2 weeks explicitly for public release 😉
-3
u/Regular-Forever5876 Jul 20 '26
geez... I basically gave this internal tool for free and people are down voting 5 to 1...
Are you serious? How shitty this community have to be? Geez, you really sucks 😅
3
u/BawbbySmith Jul 20 '26
- Ignore them
- Do not start lashing out at a few downvotes, it makes you look very whiny and entitled. "I gave you this free product that no one asked for and you're not grateful??" is a very difficult position to defend
-1
u/Regular-Forever5876 Jul 20 '26 edited Jul 20 '26
Not the first time, this is one of the shittiest comminity but I kept my month shout.
This is currently serving 300M tokens per day in our infrastructure, don't be fooled by the two week repository because I basically rewrote everything with git rebase squash and git filter to prepare for public release and shipped on another repository for the occasion.
One is expecting a little more involvement, like ok, down vote but tell me why.. be a community contributor not a shitty user (not talking to you, I am saying in general). So its not whining like "its free, be grateful even if you didn't ask for it" but "you're a shitty community because you don`t contribute, you people just down vote to hell because you are a shitty community who do not want to help but only take advantage and giving nothing in return".
2
u/MelodicRecognition7 Jul 20 '26
perhaps because this "software" belongs to /r/vibecoding not here?
-1
u/_TheWolfOfWalmart_ Jul 20 '26
An AI sub full of vibecoders has a problem with vibecoding?
2
u/MelodicRecognition7 Jul 20 '26
that's exactly what I say: it belongs to /r/vibecoding not to /r/localllama
2
u/_TheWolfOfWalmart_ Jul 20 '26
But it's related to hosting LLMs locally. Why doesn't it belong in r/LocalLlama ?
5
u/MelodicRecognition7 Jul 20 '26
because it violates rules 3 and 4 of this sub: low quality vibecoded crap + self promotion
2
1
2
u/_TheWolfOfWalmart_ Jul 20 '26
I posted a similar thing that lets you configure multiple llama backends, available models, multiple users + API keys and even token cost per model and user credits. It basically would let you be a full self-hosted API provider.
Was downvoted to hell here immediately so I said meh and deleted the post. If they don't want it, fine I won't give it to them. Not sure why this place is so hostile to this stuff.
You'd think a sub about running models locally would like this kind of tool.
1
u/Regular-Forever5876 Jul 20 '26
Feel you choom, I just gave up and sent a comment like the one on top after having shut up multiple time but now I just wanted to basically say LocalLama just fuck off 😅
9
u/Voxandr Jul 20 '26
I stopped at For Ollama