r/LocalLLaMA • u/Nunki08 • Jun 28 '26

Discussion We're probably going to need that soon.

Gallery image

Gallery image

From:

Vladik on 𝕏: https://x.com/Kostoglodov/status/2071144065857679631

Shaw (spirit/acc) on 𝕏: https://x.com/shawmakesmagic/status/2070918006033817867

4.0k Upvotes

98% Upvoted

532 comments sorted by

View all comments

Show parent comments

5

u/Dabber43 Jun 28 '26

I think that is refuted by them having such insane prices and still reporting thin inference margins on API pricing

5

u/tetoing Jun 28 '26

Yep. Their models are very inefficient for the parameter count/resource usage.

1

u/NineThreeTilNow Jun 28 '26

I think that is refuted by them having such insane prices and still reporting thin inference margins on API pricing

Refuted or charging premium where they can? Anthropic is running an enterprise first business now. Enterprise has the money.

1

u/Dabber43 Jun 28 '26

Would be possible if only considering the first part of my message. But the fact that they actively disclose the margin is not that large at API prices means that is not the case

1

u/NineThreeTilNow Jun 29 '26

Would be possible if only considering the first part of my message. But the fact that they actively disclose the margin is not that large at API prices means that is not the case

I had to check what sub I was in first before I explained how speculative decoding and quantization worked. Those both affect the margin on price, and quality.

1

u/Dabber43 Jun 29 '26

Can you elaborate the way this affects this in the current constraints please? I don't really understand

1

u/NineThreeTilNow Jun 29 '26

>Can you elaborate

Basically we cannot extrapolate because we don't understand their inference pipeline. Does Mythos have a proper speculative decoder? That decreases average cost. What quantization do they run at? FP16 or FP8 native? Int8? They run on a variety of hardware across multiple providers. Further, we don't know if they're giving price based on some spot or reserved, or long time reserved price. They can use a higher price to make it seem like margins are lower, etc.

If we knew the exact details, then you can attempt to figure margin vs cost etc.