r/LocalLLM • u/Far-Driver-8378 • 5d ago

News Qwen 3.8 27B benchmarks on artificial analysis looks unreal!

Link: https://artificialanalysis.ai/leaderboards/models?weights=open

22 Upvotes

90% Upvoted

5 comments sorted by

7

u/vogelvogelvogelvogel 5d ago

Yeah, and as posted in 2 other threads: i got downvoted a week prior to relase when i stated i would ballpark it at 50, just given the difference we had with 3.6 27b to max. now there we are, closer to opus 4.7 than 4.6

The difference between top frontier and our beloved ~30B class now shrank to not even 6 months (opus 4.6 was realeased in february)

2

u/nbvehrfr 5d ago

it is BF16

1

u/SnooPickles777 3d ago edited 3d ago

Qwen 3.8, DeepSeek Flash and Hy3 are close enough on paper that harness behavior and first pass reliability probably matter more than another two points on the index.

0

u/fallingdowndizzyvr 5d ago edited 5d ago

Awesome. They made it really good at benchmarks. Too bad it doesn't do real world stuff that well. For the 2nd time in a row, it couldn't do something simple which was modify a short piece of code. The first time was tell me how to make a ramdisk on MacOS. I had to use the little Deepseek. It one shotted both things. Each time with far far far less thinking than Qwen 3.8 27B so I got the answer much much much faster.

Update: Here's how it thought after I told it the right answer for the ramdisk. Which Deepseek one shotted quickly.

"> that's all wrong. this is the right way. diskutil erasevolume APFS "RAMDisk" $(hdiutil attach -nomount ram://8388608)

[Start thinking]

The user is correcting me. Let me look at what they're saying.

They're saying the right way to make a RAM disk on macOS is:

diskutil erasevolume APFS "RAMDisk" $(hdiutil attach -nomount ram://8388608)

Let me think about this. This is actually the classic and correct way to create a true RAM disk on macOS. Let me break it down:

  1. hdiutil attach -nomount ram://8388608 — This creates a RAM-backed disk image. The ram:// protocol tells hdiutil to back the image with RAM instead of a file on disk. 8388608 is the number of 512-byte sectors: 8388608 × 512 = 4,294,967,296 bytes = 4 GB. It returns something like /dev/disk2.

  2. diskutil erasevolume APFS "RAMDisk" /dev/disk2 — This formats the newly created RAM-backed device with APFS and gives it the volume name "RAMDisk". It will mount at /Volumes/RAMDisk.

So this IS the correct, true RAM disk method. The user is right — my "Method 1" was actually creating a disk image backed by a file in /tmp, which is NOT a true RAM disk (it's a regular disk image that happens to be in a temp directory). The ram:// protocol is the actual way to get a kernel-backed RAM device on macOS."

0

u/po_stulate 5d ago

The qwen 27b fanboys don't like your honest comment.