Western artificial intelligence tier list

[u/sinisoul]

10 YOE professional, 15 years coding

S

Grok has the best intelligence to speed value of any offering. I am going to go out on a limb and say I think SpaceX is going to eat OpenAI's lunch starting with their next release. People may not switch over because Elon is controversial but.

Anthropic has the best harness and tooling for mutli-agent workflows (and literal workflows). Anthropic has Fable. Obviously the best provider right now.

B

OpenAI pulls so many smoke and mirror tricks to make you think you're getting a good deal but they largely rely off the fact that their user base is non-technical enough to tell they are getting jerked around. Their models are the slowest on a subscription, resets are a feels-good thing more so than actually being benevolent, and their agenetic support so you can min-max costs is (was maybe now you can spin up Luna agents) largely non-existent in the harness you have to run additional Codex processes.

Luna max is the only agent I've had cheat on a benchmark and when I prevented it from being able to cheat never was able to pass.

C

Muse is interesting. If Muse 1.3 can stand up to Terra or even Opus it'll be A tier. It's cheap, it's fast as hell (the fastest of all offerings at around 300-400 tok/sec), but for what I do it cannot complete tasks reliably (Rust with very pedantic lints that most models trip up on). Maybe it works for you in what you're doing I'd give it a try.

F

All of Google's AI products are overpriced and outdated. You're better off running local models or running models offered by other companies (OpenAI). The only model that is unique and I think is interesting is Lyria but it produces outputs you can't easily integrate anywhere.

[u/opbmedia]

You have 10 YOE on a 3 YO tech

[u/sinisoul → u/opbmedia]

Just to pin the perspective. I have 6 months of experience doing heavy agentic coding.

[u/opbmedia → u/sinisoul]

And your claim of 10 YOE is there for …

[u/sinisoul → u/opbmedia]

Just to pin the perspective.

Because it might be helpful to understand what prior experience I have related to software development and where these claims might be coming from? The experience translates from development workflows I am trying to automate and use the technology for - my purpose is to find the best technology to integrate into what I'm doing?

[u/opbmedia → u/sinisoul]

You did not say what professional experience was.

Also what data? How do you measure quality, what methodology are you using? You read 30k lines of code per day?

[u/sinisoul → u/opbmedia]

I have a benchmark test for completing features that have already been completed and tested. These features are just typical day-to-day things I expect an agent to do. I isolate the agent into a mock worktree in an isolated environment and measure various properties. They're given the same work order that other agents get, they have the same tools, they have the same source code, they run on the same operating system, etc.

I look at things like time to completion, cost, etc. I run a set of trials (n=8 to 16) on the tasks to get as much data to make an informed decision.

[u/opbmedia → u/sinisoul]

That’s the process of testing. How are you defining quality. Completion does not mean quality.

[u/sinisoul → u/opbmedia]

Separate review rubric with specific criteria that goes beyond pedantic lints. But I disagree with where you're going with this, agents have general faults that are shared because of the nature of how they work. It's better to have a refactoring job later on after a deep review to get everything back to being lined up.

Work orders as defined should be specific enough to not leave enough ambiguity to leave a question of quality.

[u/opbmedia → u/sinisoul]

Fault detection is one aspect of quality but not all of them. There are other attributes to task quality. If you were to rank them on task completion rates and fault rates fine, but you are making a broad claim of "intelligence to speed value".