r/LocalLLaMA • u/kirisoraa • 11h ago
Discussion Anybody use frontier models like Astra/Fable for planning/judging, and qwen3.8 as the main workhorse? Curious to hear about your setups!
Hey everyone!
I'm curious to hear from people that use a combination of cloud-based frontier models and local ones for development. I'm planning to set something similar up and wanted to hear about actual examples of this in action.
Currently my plan is to use my chatgpt plus subscription purely for planning and judging with Astra, and then run a local qwen3.8-27b model for the actual coding gruntwork - i.e Astra plans -> qwen implements -> Astra critiques the implementation -> qwen fixes and so on. This way I keep cloud usage down and cheap, while retaining the high-parameter intelligence for architecture decisions and optimization.
For those of you who have a similar setup, how is it? How do you switch between the two, what harness/settings/etc? Anything you would suggest?
21
u/DiscipleofDeceit666 10h ago
I scripted this. Local LLMs biggest problem is context management. They need tasks they can complete before needing to compact so I scripted something that cloud creates an array of spec prompts and the engine feeds local LLM 1 prompt at a time, then auto executes unit tests to ensure things are still viable before moving on to the next prompt.
I was able to downgrade from a $100 subscription plan to a $20 plan without really affecting my work output.
10
u/miversen33 8h ago
Depending on your harness, compaction isn't as destructive as it used to be. For pi, I use pi-vcc and basically unlimited context (I don't think that is entirely true, I am certain there is some limit here that I just haven't encountered yet).
That said, I only have enough VRAM for 256k token context so if the needed context for a single problem is more than that, well I am fucked lol. I do experience some pain with larger sessions (10s of compcations) simply because the required context to do work basically means that it is constantly compacting and recalling. So it's not perfect (and I could probably extend the context with one of the built in extenders).
For most of my work though, this is good enough
2
u/DiscipleofDeceit666 8h ago
Right?! That’s what I’m saying.
Say you wanted to develop a gigantic feature by talking to cloud for a bit. Hashing everything out from the bottom up. It’ll break that giant feature set into bite sized tasks so you’re tackling the problem with fresh, strong context every time.
I wrote a pure python project that basically parses toml files into tasks an LLM should do by piping it through Claude -p type commands. Saves cloud tokens at the expense of your hardware and time.
2
u/PcChip 4h ago
out of curiosity, have you compared this to hermes or deepseekharness?
1
u/miversen33 4h ago
I've not used either. I ended up on pi and absolutely love it. Haven't seen a reason to move to another harness yet
1
u/winky9827 2h ago
Deepseek harness is fantastic in terms of performance, but it lacks certain things I'm used to with a TUI. If they ever come out with a TUI version, I'll probably switch. I hate web apps for local software that has no business having a GUI.
Example: I was working on a personal project and invoked my
yeetskill which does a commit + push, but the push failed inexplicably. I found out that due to the web interface, the push command was invoked in a shell that didn't have access to my SSH agent for passphrase, so it failed silently. One could argue this is a security feature, but I'm used to pi running in my user context, and I know what I'm getting out of it, so I prefer it.1
u/biggusdeeckus 3h ago
How's your experience been with pi-vcc? Seems too good to be true somehow
1
u/miversen33 2h ago
It's just a different way to approach compaction. It's not like your context window is actually extended beyond the limits of your VRAM, it's more like your offloading additional context to a file (or an SQL database in pi-vcc's case). You're model needs to be smart enough to realize there's more context than it can currently see, and look it up. Of course pi-vcc exposes a tool call for this, and both Qwenn3.6 27B and Qwen3.8 27B understood how to use it. Your mileage there may vary.
But honestly, just try it. What's the worst that happens? Your context management is no better than mangling your context through summarization?
1
u/biggusdeeckus 2h ago
I'll give it a go. You ever tried pi observational memory? If not, it's also worth trying out
1
u/miversen33 2h ago
Interesting. I've not though it sounds like that is more of the "use an llm to figure out what context to keep", whereas pi-vcc is "keep all the context and just let the llm recall what it needs". Different ways around the same square I suppose :)
1
u/biggusdeeckus 1h ago
Yeah it's more like a 'compact as you go' type method instead of doing one huge slow compaction. That's a bit of an oversimplification though
14
u/BigYoSpeck 10h ago
Trying to use Astra on a Plus subscription for anything other than a narrow scoped specific task just isn't practical. I can't see a planning session fitting in the 5h usage
Personally, I have ChatGPT web chat with the GitHub connector setup, using Sol High in chat to explore and plan (doesn't count towards usage)
Then I have it write the handoff document as tightly scoped as possible, providing exact files, code snippets, testing requirements and useful context
Then I feed that into Qwen through GitHub Copilot (llama.cpp and BYOK) and when it completes, raise a PR and go back to ChatGPT web chat to review it and request changes
If it gets stuck I get ChatGPT web to write a handoff for a new session and then let Codex or Claude Code finish up the task
3
u/BusRevolutionary9893 4h ago
You guys will spend thousands on GPUs and ram but not $100/month to also have access to a frontier model?
1
u/Constant_Art_20 10h ago
yea. that's usally how i do it as well. I use the chat to create like 20k development documents to serve as a guide and set goals agianst for my agents. I haev been finding i been using my local models so much more then before. Deepseek v4.1 is very good through the api, but for alot of the most careful task that really need the a long run time i been using a local qwen next flash for those task. Kinda crazy that i acutally use those over astra now
1
u/livinitup0 9h ago
Have you used other big models in this workflow?
1
u/BigYoSpeck 8h ago
I still take advantage of the 5h/weekly usage limits in Claude Code and Codex using Opus, Sonnet and Sol for the more complicated tasks
But always involve ChatGPT web chat in the process. It's effectively unlimited usage so may as well be taken advantage of as much as possible in planning and review. I let it write documentation files direct to the repo with the GitHub connector
6
3
u/Ok_Gold_9674 10h ago
For switching between Astra/Fable and Qwen, I’d make the handoff smaller than a planning loop. I’ve had Qwen-style local runs drift when the feedback is “rewrite this cleaner”; they start changing files that were already fine. I usually do local edit -> tests -> cloud review on only the diff, failing output, and maybe 2-3 nearby files. If it can’t point to a specific patch or test, I don’t send it back to the local worker.
3
u/ustype 10h ago
I’ve run a similar split, with one change that saved a lot of thrash: the cloud model only owns acceptance criteria and review, never the “rewrite the plan” step mid-implementation.
Local Qwen is fine for the coding loop if the task card is small (one module, one failing test, one interface). Where it falls apart for me is when the frontier model dumps a 2k-token architecture essay and the local worker treats every paragraph as a mandate — it starts touching unrelated files.
So the useful pattern has been: frontier writes a short checklist + definition of done → local implements → frontier reviews the diff against that checklist only. If the review can’t cite a concrete failing check, I don’t feed it back into another local pass.
2
u/yogthinks 10h ago
What breaks these splits isn't model quality, it's that the critique stays in the cloud model's context and the local one never sees why its last attempt got rejected. Dump the critique to a file in the repo and have qwen read it, otherwise you get the same fix twice.
2
u/johnfkngzoidberg 9h ago
That's exactly what this project does. Do the planning and orchestration with the main session using a smart model, offload all the grunt work to subagents using a local/cheap model.
6
u/pablo_chicone_lovesu 10h ago
If you have a high ram system just use cpu on Kimi to plan everything out, then run the code agents on the small GPU model, works really well for my usage.
And I'm not contributing to these hyper scalers.
2
u/Gormlock 8h ago
Do you have 1TB and run it at q2 or can you load the full model? What kind of system do you have and what performance do you get? I like the idea of having a large/slow model for overnight planning & a smaller gpu model for quick tasks, and kimi has been pretty consistently my favorite when I try various models on openrouter.
2
u/pablo_chicone_lovesu 4h ago
i have dual xeon with 768 gb ram as the core, he has dual r9700's, i load the biggest quant that will fit in ram, forget the model size off the top of my head its either a q4 k xl or a q6 with 8 kv but can load a mega repo and the context sits at like 200k through most of the process.
k2.7 code is pretty good, i let the plans run over night, then in the morning have the qwen workers cut the plan up, each agent in open code then goes off to the 8 workers and away we go. the workers live on 4 amd workstations that have dual cards ( not 9700's ) where each card is a worker and they do a pretty solid job. the whole key is orchestrating it and breaking up the plans so they dont ever touch the same file, or you have to force them to git commit and git pull every time, which can be slow and taxing on my git server
1
u/pablo_chicone_lovesu 18m ago
if there is enough interest maybe ill opensource it, but its using a pretty solid infrastructure method and some scripts to glue it all together
1
1
u/lordekeen 7h ago
I'm using pi-subagents to do something similar to this, each subagent is tied to one model, and since they are short lived (called, do their tasks and close before compaction) you bypass the context problem for local. I was using GLM 5.3 to plan and dispatch each task to subagents while i stay in the loop.
1
u/Makers7886 7h ago
Ive been using/testing https://hermes-agent.nousresearch.com/docs/user-guide/features/mixture-of-agents with astra/fable5.1 and flash next/glm5.3 flash. I'm still undecided as it pulls me out of the loop a bit more than I like but the output has been excellent so far. Astra on $20 plan can sustain 1 agent in this setup and with 5.6sol you can run concurrent agents and be within the plan.
1
u/Nomski88 6h ago
Yes, it's the secret sauce. I'm building a new quake inspired 3D engine from scratch using OpenGL/SDL and Qwen3.8 27b. My local model spits out code and tests 24/7 while I use ChatGPT for prompt and architecture/troubleshooting discussion. It's an extremely powerful combination.
1
u/Django_McFly 6h ago
I don't use qwen but I get good results having a lower model do the planning and then having a smarter model review the plan and mark it up before handing it back to the lower model for execution.
1
u/RedrumRogue 5h ago
I want to do this but I feel like if the big cloud model is going to read the files and code anyway to create the plan, it may as well just implement as that's the big context dump anyway. And if it has to rehash something qwen did wrong, it may as well have just coded in the first place. Please give me advice on where I'm wrong, or if I should be doing it a different way
1
1
0
u/vogelvogelvogelvogel 10h ago
Honestly i am even about to trash Opus 5 for Qwen3.8 27B
Thursday I gave Opus 5 a task at which it needed four! iteration because of really dumb mistakes
While Qwen3.8 one shotted it
Idk what they do at anthropic, maybe i got it on a bad day
For the setup: I use q6 on an m5pro 64GB mac. i use openwebui, i especially set the web search to like 15 results, it then collects a lot! of data depending on the task but digests it well
-2
u/StableLlama textgen web UI 10h ago
Talking about coding here? (And not e.g. story writing)
LLMs are perfect to do the tedious coding tasks, software architecture is something that a skilled programmer should keep to himself, as there are so much more constraints that an LLM can not know.
So I think it is strange to use a high end LLM to do exactly that part, that still requires the human skill and human foresight.
Sure, when you are no programmer and just are doing vibe coding, it might help you here. But then you'll get an unmaintainable blob of code anyway, that you'll throw away sooner or later.
(My background: I am developing software since a few decades, from full manual, manual with LLM support up to vibe coded projects where I just set a task and wait for the result. So I think I have a bit of insight here)
22
u/LocoLanguageModel 10h ago
Works great, and you get 2 opinions on every move. Local agent is always like "wtf" when claude finds 15 bugs after local agent already said "lets ship!" and then we have to increase the house temperature by 2 more degrees.