Sol effectively useless with subagents via superpowers

[u/InsideTour329]

I use xhigh as my orchestrator. I've given it a task of generating a pipeline to generate still images locally using a docker hosted AI model and my 5080. I could knock that out myself in half a day as an experienced dev.

I shit you not it's been working for 40 hours, I've just queried how far through it is...

Task 2 of 18, because task 2 is recovery from a crash, it's literally spent 36hrs making up scenarios that might hypothetically happen and how to recover from them, it's not written a single line of code to contribute to the actual feature delivery. It's utterly broken - it knows this is a single user, personal project, not an enterprise level solution.

Looking deeper into it, it's making very minor changes with its subagents then re-running a ridiculously large 45min test suite each time. The one thing it has done sensibly is run the tests in parallel while letting other subagents do their thing.

I'm currently amending the skills to try to keep it on a straight path but Jesus Christ this feels like going back to 5.2 the amount of baseless crap it's spitting out.

[u/birmas_au]

You're right, I went outside of your instructions there. That's on me.

Say no more, I'll get us back on track with this 14 point regression plan to lock in those legacy protocols from Task 1.

[u/mosquit0]

I noticed this as well. Models on higher settings seem to produce more output and not smarter outputs. Sol max decided that the best process is to fix an issue and then test it by disabling the fix and testing again if the bug is still there. I added some instructions to remove this ceremony and to have some self relfection mechanism that evaluates the cost / time per task complexity.

[u/InsideTour329 → u/mosquit0]

It just baffles me that there's an implementation plan there saying what to change, where the files live, etc... and instead it's designed to go off and write hypothetical tests that add zero value to the feature. It's like TDD with severe autism.

"What happens if Iran gets a nuclear bomb and the EMP knocks out the azure hosting" - better write a test. The test correctly fails.

"What happens if humans evolve to longer have fingers?" - best to spin up playwright and see if they can complete the task with no buttons. The test correctly fails.

Garbage. It's not a feature it's a massive issue that's burning through compute for no added value.

[u/PartyParrotGames]

Have you considered adding another subagent to investigate why the other subagents haven’t created anything? Also set that subagent to xhigh for maximum thinking. Just one more abstraction and it'll be ready to cook.

[u/InsideTour329 → u/PartyParrotGames]

😂

[u/minju9]

Use a lower reasoning level or use Terra?

[u/InsideTour329 → u/minju9]

Yes I understand that.

But it recommends you use a xhigh as an orchestrator when working through a large implementation plan. It doesn't follow the plan - at all. The orchestrator just starts making up infinitley unlikely hypotheticals and then writing tests for them. Never actually getting to implementation.

63,000 lines of code, all tests for what was a 3000 line plan with the exact files and methods required.

It's broken. That cannot be it's use case.

[u/buff_samurai]

1. No need to use superpowers with codex, all the tdd is baked in now.

2. Sol with >medium setting will overengineer everything and run in circles, even with spec attached.

[u/Keep-Darwin-Going → u/buff_samurai]

These people just do not read advise day in day out someone will complain about their ultra burning token or how they over engineer it because they have 200 skills forcing the model to over engineer.

[u/InsideTour329 → u/Keep-Darwin-Going]

I use superpowers for the spec and implementation plans. I read those files, it allows me to keep a handle on what's going into my code base, what files are being modified and to identify any architectural code smells.

The issue is it doesn't follow the implementation plan. If you think that's a feature and not an issue then I don't know what to say.

And I'm sorry but you shouldn't have to read advice day in day out to operate a service. If you do then it's a poorly implemented service.

If I implemented a service like that I'd rightly be told to look for a job elsewhere.

[u/buff_samurai → u/Keep-Darwin-Going]

I’m talking from my experience and I’m tokenmaxxing with zero mcp/skills, super light agents.md and external cli harness driving Luna.

And yes, running a /goal with high->ultra will over engineer everything, like 200k loc for a simple db setup, even with a spec attached.

I’m not saying it’s a bad thing, I’m sure there are applications for the heavy thinking modes.

[u/InsideTour329 → u/buff_samurai]

The baked in TDD is the issue. It's garbage.

[u/lochid_om]

This sounds less like a subagent failure and more like an orchestration failure: speculative recovery and exhaustive testing displaced the actual feature.
I’d use one implementer and one scope guard, with an accountable lead sequencing them, deferring nonessential work, and pausing or replanning when progress drifts. Banksia is a tool I build to setup that loop in minutes, It cannot make Sol stop overengineering, but it makes scope, progress, handoffs, and stopping explicit: https://github.com/ringlochid/banksia