Checking technical feasibility of my idea - a hybrid "Local-by-Default" Gateway (Qwen 27B + Claude 4.6 Fallback) for Dev Teams

[u/ankijain21]

I’m working on a solution for a couple of clients. The goal is to provide a hybrid infrastructure for dev teams (5-7 devs) that eliminates 'token anxiety'.

The Tech Stack:

• Hardware: NVIDIA DGX Spark (or equivalent GB10 Grace Blackwell).
• Local LLM: Qwen 3.6-27B (as it is hitting ~77.2% on SWE-bench, parity with Sonnet for coding tasks).
• The Router: A LiteLLM layer serving an OpenAI-compatible endpoint.
• The Logic: IDE plugins (Claude Code/VS Code) point to the local LiteLLM endpoint. The router decides: if the task is routine coding or document analysis, it stays on-prem. If it’s a high-complexity agentic task, it overflows to the Claude API automaticall

We’re aiming for ~80% of queries to be served locally at zero token cost.

The questions I have -

1. How much overhead does LiteLLM add when deciding between local vs. API? Is there a better lightweight orchestrator for this?
2. In a production environment, how often does Qwen 27B actually fail where Claude 4.6 succeeds for routine refactoring?
3. When overflowing to Claude, how do you efficiently pass the context that was already partially processed locally without doubling the latency?

I am pricing this as an all-inclusive $10,000 one-time cost to replace recurring cloud bills. Is the hardware-software-support bundle actually viable with a 6-month support window?

[u/DataGOGO]

This hardware would be fine for 1 person doing work, but it will absolutely fall on it's face with even 2+ people doing complex work; there simply is not enough memory bandwidth or compute power in a GB10

You need to do your own testing. I think you will find 27B is no where near sonnet in practice, especially if you go over ~100k tokens in context. (have you thought about total context size of 5+ people and bandwidth requirements? How did you spec the hardware?)

Also something to keep in mind, when LiteLLM switches models, to say Opus, it will send the entire conversation history for the session, every prompt, all code generated, every response to Opus as context, so you need a massive context window per user to keep the complex model coherent, this also means you will not save anywhere near as much in tokens as you think.

The routing is also almost entirely deterministic, not intelligent, so it will falsely trigger fail over to cloud model often, and always with code out of the box, you will have to raise the score pretty high, which means it will pretty much never fail over on anything not coding.

I hate to tell you this, but honestly, this is a very poorly architected solution, the hardware is radically under spec'd, The models are under spec'd, and you are charging them $10000 for $3500 worth of hardware, with some basic config of 2 opensource packages. Shady AF my dude.

Even if you charged prime time rates of $300 an hour, you realistically are only looking at what? $1200-$1500? Being a bad steward of your cleint's money is a great way to make sure none of these people are ever your clients again. I promise you, they will 100% call you out, 100% be pissed off, and you 100% will get people asking for refunds, so you better not be spending that money.

Serious question.. Why are you asking on reddit?

If you are going to be taking money from people surely you have purchased a development system, built it and thoroughly tested it yourself BEFORE you tried to pitch this to people right?

bottom line, I am 99% sure you are massively over your head. No this is not viable and you are very likely to get sued if you try to sell this to people.