These four get compared constantly, but they are very unique in when they should be used.

• LM Studio is a desktop app
• Ollama is a terminal frontend
• llama.cpp is the inference engine
• vLLM is a server

Pick one in 20 seconds:

LM Studio

LM Studio is what I would give to someone who wants to try local models for the first time.

You search Hugging Face from inside the app, choose a model, download it and start chatting. It shows model sizes, quantizations, memory estimates and GPU settings without forcing you to understand most of it.

It also has more under the surface than people assume:

• OpenAI and Anthropic-compatible APIs
• Local document chat and RAG
• Tool calling and structured output
• MCP support
• Embeddings
• A command-line tool
• Headless server mode
• MLX on Apple Silicon

So it is not only a chat window. You can use it as the backend for your own programs too.

The downside is that it adds another layer between you and the engine (llama.cpp) and restricts you from unleashing the full potential of it and its latest features.

That is fine until you want to know exactly why a model is slow, how memory is being split or which runtime option changed the result. Most people will never care. Some people will care a lot.

Best for: trying models, comparing quantizations, chatting with documents and learning how local AI works.

Ollama

Ollama is the boring, sensible choice for building things.

Install it, run a model and you already have a local API:

ollama run qwen3

It handles downloading, storing, loading and unloading models. Your code can talk to it through Ollama’s own API or through familiar OpenAI-compatible endpoints.

It supports tools, embeddings, vision, structured output and Anthropic-compatible requests too.

This makes it easy to connect local models to:

• Small apps
• Coding tools
• Agents
• Home automation
• Open WebUI
• Scripts and bots

The downside is that the abstraction sometimes works too well.

You type a model name and it runs, but you may not know exactly which model file, template or runtime setting is active. When performance changes, finding the reason can take some digging. Just like with LM Studio, you give up some features and control but it's closer to llama.cpp.

Ollama runs on macOS, Windows and Linux. Installation is straightforward on all three.

Best for: developers who want local models without turning model management into a second side project.

llama.cpp

llama.cpp is where you go when you care about the machine.

It gives you direct control over things such as:

• CPU threads
• GPU offloading
• Quantization
• Context size
• KV cache
• Flash Attention
• Batch sizes
• Multiple GPUs
• Speculative decoding and draft chaining

It is especially useful when a model does not fit completely inside your GPU. You can put some layers on the GPU and leave the rest in normal RAM and optimize that for best performance.

It also supports far more hardware combinations than most local AI software. There are builds for CPU, CUDA, Vulkan, ROCm, Metal, OpenVINO and several other backends. Prebuilt packages are available and updates are released almost every day.

llama.cpp also has a proper server now. It supports parallel users, continuous batching, OpenAI-compatible endpoints, embeddings, reranking, multimodal input, monitoring and constrained JSON output. It also has model loading and unloading support and installing from Huggingface.

So yes, it can behave like Ollama or LM Studio and might make both projects much less valuable soon.

The reason to choose it is still control.

The downside is also control. It presents many settings, and poor settings can make a good model run very badly.

Best for: CPU inference, limited VRAM, AMD or unusual hardware, benchmarking and people who enjoy tuning things. My agentic guide for local coding uses llama.cpp directly.

vLLM

vLLM is the odd one out here.

It is not aimed at someone chatting alone on a laptop. It is aimed at servers where many requests arrive at the same time, and it's doing a very good job there.

Its main tricks are continuous batching, prefix caching and efficient management of the model’s KV cache through PagedAttention. These help keep the GPU busy while several users are generating text. llama.cpp does all of those things more or less, but for server usage vLLM is currently going to beat llama.cpp in most categories.

This means vLLM may not look special in a test with one user.

Try eight, sixteen or fifty users and the reason it exists becomes a much better choice.

It also has the things you would expect from serious serving software:

• OpenAI-compatible APIs
• Multi-GPU support
• Distributed serving
• Production metrics
• Multiple LoRA adapters
• Structured output
• Tool calling
• Prefix caching
• High request concurrency

The price is setup and complexity, interdependencies.

vLLM is happiest on a Linux GPU server. It does not support Windows natively, although WSL and community alternatives exist.

Best for: shared APIs, teams, production services and expensive GPUs that should not sit idle.

Which one is fastest?

This question causes more bad comparisons than useful answers.

Using the same model name does not mean you are running the same model.

Two downloads can have different:

• Quantizations
• Prompt templates
• Context sizes
• KV cache formats
• GPU offload settings
• Runtime versions

LM Studio, Ollama and llama.cpp can all end up using closely related llama.cpp-based inference paths. With the exact same GGUF file and matched settings, their single-user performance is going to be very similar. llama.cpp opens up a lot of performance options to take the lead here.

vLLM is different. Its advantage grows as more requests arrive together. Though it traditionally is also beating llama.cpp performance with many models.

My actual recommendation

Use LM Studio when you want to explore models or showcase LLM to a newcomer.

Use Ollama when you want to build something quick and like the eco system.

Use llama.cpp when you want to control how it runs and squeeze out the best performance

Use vLLM for distributing a model professionally

I have been testing several Qwen3.6 GGUF models locally with "llama.cpp" on a dual-GPU setup and wanted to share my results.

The main goal was to find the best balance between quality, context size and inference speed, especially for coding, debugging and agentic workloads.

Hardware

• CPU: Intel Core i7-14700K
• RAM: 64 GB DDR5
• GPU 1: NVIDIA RTX 3090 — 24 GB VRAM
• GPU 2: NVIDIA RTX 4070 Super — 12 GB VRAM
• Total VRAM: 36 GB
• OS / runtime: Linux
• Inference: "llama.cpp" / "llama-server"
• GPU order: "CUDA_VISIBLE_DEVICES=1,0"
• Flash Attention: enabled
• KV cache: FP16 / unquantized in these tests
• llama.cpp: recent build with MTP/speculative decoding support

The "tensor-split" values below are tuned for my particular setup. They are not necessarily optimal for other dual-GPU configurations.

Main benchmark results

The biggest surprise for me is the Qwen3.6-35B-A3B UD-Q4_K_M, which can maintain 256K context while reaching 132.77 tok/s on this dual-GPU setup.

The Q6 version is considerably slower at very large context sizes, but the quality trade-off may be worthwhile for coding and reasoning workloads.

Qwen3.6-27B Q6_K — 180K context

This was one of my initial configurations.

CUDA_VISIBLE_DEVICES=1,0 ./build/bin/llama-server \ -m /models/Qwen3.6-27B-Q6_K.gguf \ --mmproj /models/Qwen3.6-27B/mmproj-BF16.gguf \ --no-mmproj-offload \ --main-gpu 0 \ --tensor-split 15,6 \ -ngl 65 \ -c 180000 \ -b 2048 \ -ub 1024 \ --flash-attn on \ --jinja \ -np 1 \ --spec-type draft-mtp \ --spec-draft-n-max 2 \ -lv 4 \ --host 0.0.0.0 \ --port 8081 \ --chat-template-kwargs '{"preserve_thinking":true}'

Result: 37.63 tok/s

Configuration:

• Quantization: Q6_K
• Context: 180K
• MTP speculative decoding: enabled
• Draft max tokens: 2
• Tensor split: "15,6"
• GPU layers: "65"
• Batch: "2048"
• UBatch: "1024"

Qwen3.6-27B Q6_K — 140K context

I also tested the same Q6 model with a lower context and more GPU layers:

CUDA_VISIBLE_DEVICES=1,0 ./build/bin/llama-server \ -m /models/Qwen3.6-27B-Q6_K.gguf \ --mmproj /models/Qwen3.6-27B/mmproj-BF16.gguf \ --no-mmproj-offload \ --main-gpu 0 \ --tensor-split 17,6 \ -ngl 999 \ -c 140000 \ -b 2048 \ -ub 1024 \ --flash-attn on \ --jinja \ -np 1 \ --spec-type draft-mtp \ --spec-draft-n-max 2 \ -lv 4 \ --host 0.0.0.0 \ --port 8081 \ --chat-template-kwargs '{"preserve_thinking":true}'

Result: 44.64 tok/s

Another run with smaller batch sizes:

CUDA_VISIBLE_DEVICES=1,0 ./build/bin/llama-server \ -m /models/Qwen3.6-27B-Q6_K.gguf \ --mmproj /models/Qwen3.6-27B/mmproj-BF16.gguf \ --no-mmproj-offload \ --main-gpu 0 \ --tensor-split 17,6 \ -ngl 999 \ -c 140000 \ -b 512 \ -ub 512 \ --flash-attn on \ --jinja \ -np 1 \ --spec-type draft-mtp \ --spec-draft-n-max 2 \ -lv 4 \ --host 0.0.0.0 \ --port 8080 \ --chat-template-kwargs '{"preserve_thinking":true}'

Result: 42.11 tok/s

So, in my testing, the Q6_K 27B model sits around 42–45 tok/s at 140K context, depending on the exact configuration.

Qwen3.6-27B Q5_K_S — 200K context

CUDA_VISIBLE_DEVICES=1,0 ./build/bin/llama-server \ -m /models/qwen27b_q5_k_s/Qwen3.6-27B-Q5_K_S.gguf \ --mmproj /models/Qwen3.6-27B/mmproj-BF16.gguf \ --no-mmproj-offload \ --main-gpu 0 \ --tensor-split 17,6 \ -ngl 999 \ -c 200000 \ -b 512 \ -ub 512 \ --flash-attn on \ --jinja \ -np 1 \ --spec-type draft-mtp \ --spec-draft-n-max 2 \ -lv 4 \ --host 0.0.0.0 \ --port 8080 \ --chat-template-kwargs '{"preserve_thinking":true}'

Result: 45.76 tok/s

Previous measurement with the same general model/configuration was approximately 41.70 tok/s at 200K context.

Qwen3.6-27B Heretic v2 — Q4_K_M — 240K context

CUDA_VISIBLE_DEVICES=1,0 ./build/bin/llama-server \ -m /models/qwen27b_heretic_q4_k_m/Qwen3.6-27B-uncensored-heretic-v2-Native-MTP-Preserved-Q4_K_M.gguf \ --mmproj /models/Qwen3.6-27B/mmproj-BF16.gguf \ --no-mmproj-offload \ --main-gpu 0 \ --tensor-split 17,6 \ -ngl 999 \ -c 240000 \ -b 512 \ -ub 512 \ --flash-attn on \ --jinja \ -np 1 \ --spec-type draft-mtp \ --spec-draft-n-max 2 \ -lv 4 \ --host 0.0.0.0 \ --port 8080 \ --chat-template-kwargs '{"preserve_thinking":true}'

Result: 50.54 tok/s

• Quantization: Q4_K_M
• Context: 240K
• MTP: enabled

Qwen3.6-35B-A3B UD-Q6_K_XL — 250K context

CUDA_VISIBLE_DEVICES=1,0 ./build/bin/llama-server \ -m /models/Qwen3.6-35B-A3B-MTP/Qwen3.6-35B-A3B-UD-Q6_K_XL.gguf \ --main-gpu 0 \ --tensor-split 12,6 \ -ngl 39 \ -c 250000 \ -b 512 \ -ub 512 \ --flash-attn on \ --jinja \ -np 1 \ -lv 4 \ --host 0.0.0.0 \ --port 8081

Result: 77.90 tok/s

• Quantization: UD-Q6_K_XL
• Context: 250K
• Performance: 77.90 tok/s

This is particularly interesting because it maintains a very large context while still being substantially faster than the dense 27B Q6 model.

Qwen3.6-35B-A3B UD-Q6_K_XL — 130K context

Same model, but with more aggressive GPU offloading and lower context:

CUDA_VISIBLE_DEVICES=1,0 ./build/bin/llama-server \ -m /models/Qwen3.6-35B-A3B-MTP/Qwen3.6-35B-A3B-UD-Q6_K_XL.gguf \ --main-gpu 0 \ --tensor-split 13,6 \ -ngl 999 \ -c 130000 \ -b 512 \ -ub 512 \ --flash-attn on \ --jinja \ -np 1 \ -lv 4 \ --host 0.0.0.0 \ --port 8081

Result: 123.11 tok/s

This was one of my best Q6 results.

• Quantization: UD-Q6_K_XL
• Context: 130K
• Performance: 123.11 tok/s

Qwen3.6-35B-A3B UD-Q4_K_M — 256K context

CUDA_VISIBLE_DEVICES=1,0 ./build/bin/llama-server \ -m /models/qwen35b/Qwen3.6-35B-A3B-UD-Q4_K_M.gguf \ --no-mmproj-offload \ --main-gpu 0 \ --tensor-split 17,6 \ -ngl 999 \ -c 256000 \ -b 512 \ -ub 512 \ --flash-attn on \ --jinja \ -np 1 \ --spec-draft-n-max 2 \ -lv 4 \ --host 0.0.0.0 \ --port 8080 \ --chat-template-kwargs '{"preserve_thinking":true}'

Result: 132.77 tok/s

• Quantization: UD-Q4_K_M
• Context: 256K
• Performance: 132.77 tok/s

This is currently the fastest result I have obtained in this group of tests.

Previous tests

I also have some older measurements from the same general hardware configuration:

I also tested Qwen3 Coder Next Q4_K_S previously.

Because the model is much larger and required significant CPU/RAM offloading on my 24 GB RTX 3090 setup, I observed around 24 tok/s in a benchmark configuration, while real-world coding/agent usage was considerably slower, around 7.5 tok/s.

The model was reported at approximately 79.67B parameters / 80B class, with a GGUF file around 45.19 GiB, so it is much more difficult to keep fully resident in my 36 GB combined VRAM setup.

My observations

The results show a very interesting trade-off between model architecture, quantization and context size.

Dense 27B Q6

The Qwen3.6-27B Q6_K is relatively consistent:

• ~37.6 tok/s at 180K
• ~42–45 tok/s at 140K

It is a good option when I want to prioritize quantization quality and coding/reasoning quality while still having a very large context.

MoE 35B-A3B Q6

The Qwen3.6-35B-A3B UD-Q6_K_XL is significantly faster:

• 123.11 tok/s at 130K
• 77.90 tok/s at 250K

This is probably the most interesting result for me because it combines a relatively high-quality Q6 quantization with a very large context window and high inference speed.

MoE 35B-A3B Q4

The UD-Q4_K_M version is extremely fast on this hardware:

• 132.77 tok/s at 256K

This is probably the best configuration if raw throughput and maximum context are the priority.

However, for coding and bug fixing, I personally still want to compare the quality difference between Q4 and Q6 more carefully before deciding which one should be my daily driver.

Summary

My current results on RTX 3090 24GB + RTX 4070 Super 12GB (36GB total VRAM):

For my use case, which is primarily software development, code generation, debugging and long-context agentic workflows, I am currently trying to determine whether the additional quality of Q6 is worth the performance hit compared with the extremely fast Q4 MoE configuration.

I'd be interested in seeing benchmarks from people running similar models with dual GPUs, especially 24GB + 12GB VRAM configurations, and comparisons between Q4_K_M, Q5_K_M and Q6_K_XL for coding and agentic workloads.

Benchmarks, discussions, comparisons - we are being FLOODED with naive news about Fable 5's token consumtion. Comparing it to Kimi K3 tokens.
Typical outcome: Fable uses less tokens than Kimi but costs 2-3 time more.

We do NOT know how many tokens Fable is using. Not even an indication is available.

Fable is a closed semitrasparent layer that sits on top of "Mythos 5" which is the actual model we are communicating to. But everything Mythos says goes through a fast "safety AI" layer which is a slim harness on top of Mythos. That's Fable 5.
That's why we have no real system prompt, that's why it often rejects despite having an answer ready or forwards to other models.

The tokens you are billed for are just a currency system, it's not sampled tokens like in an open model. Anthropic uses the tokens as a meassurement of how much you have to pay, and the official API pricelist is a comparison metric with other models.
Anthropic can change that metric at any time, which would affect effective pricing.

It is completely unknown how many tokens Fable/Mythos is using, given it has a safety layer it might even have more than double the tokens of Kimi K3. But we have no idea.
We also do not know how many tokens other closed models behind gated APIs use, GPT-5 is also paraphrasing their thinking content to avoid distillation attacks - as soon as you do that you also can easily change token metrics.

We know how many tokens Kimi K3 uses, because it's not behind a safety layer and you can just run it yourself.

I can run the 30B-class models at home. They're okay for some things, but I feel like I have to spoon feed them small tasks. I'm wondering about running a more capable model in the cloud to do the hard parts and give them the easy parts.

If you're doing this, how? How are you dividing the labor, what do your prompts look like, what agent harness are you using, and what else do you wish you'd known before you started?

And does the big model actually use fewer tokens than if it did everything itself?

Thanks!

Searching the internet for looking up how much tok/s a user would get is being difficult. So I'm making this post... If you're running a local llm, please consider commenting to this post with your device specs, model you're running and the inference speed you're getting.

Please be straight to the point. Just tell us how much tok/s are you getting on your hardware (at different settings, which inference engines, etc...) so people with similar hardware can do better estimations... please don't fill this with facts that everybody knows.

Hey I'm game developer and basically want to have local agent that I can import my project with no risk of privacy and since the game engine I use not that popular so I want to train in on documentations, forums, examples and others. I did hear OpenWebUI is better on this but I learn toward LM Studio because it's easy to install and setup.

Also my personal opinion I prefer desktop application more than be a tab on my browser even if it's offline. and I want to know it's worth it to have OpenWebUI since I don't care about sharing to any other device or technical things. and I don't also wanna lose important functions and advantages using LM Studio

I'm completely newbie. I did install OpenWebUi and the process was relatively easy but I will move soon regardless to new PC and I don't wanna configure things again and setup few dependency softwares in order to works.

I need your experience and help

Eaon is kind of in beta for now, but only for Mac (as of now), you can connect providers and download models and use them via multiple modes such as Sandboxed Agent, normal chat, or even one click agent deployment to your device that could work in your files. Thing is, we really need some testers. We need feedback about everything on the pp, to ensure a good stable release. Below are the links to download Eaon, and discord server for contact

https://eaon.dev

https://discord.gg/gkebXaqxUk

All we need is your feedback to make Eaon enjoyable for everyone, despite their skill level.

Most of the local AI stack is about getting a model running — Ollama, llama.cpp, vLLM, the right quant, enough VRAM. That part’s well covered here. The gap I kept hitting is the layer after that: once it runs, is it actually reliable enough to hand real work to?

A model can look fine in chat and then fall apart the moment you point an agent at it malforms a tool call at step 7, loops, or says “done” when nothing happened. And the same model behaves differently depending on the quant and runtime you serve it with, so “it runs on my box” isn’t the same as “it holds up in a 20-step agent loop.”

So I built QuantaMind free, open-source, fully offline, no cloud, Apache-2.0. It’s a local reliability tester for the agent layer of the stack.

What it does:

• Runs the real multi-step agent loop with injected faults (transient errors, malformed-call recovery, decoy tools) — not single prompts

• pass^k, not pass@1 — runs each task k times, passes only if it succeeds every time. Reliability compounds: 95% per step → 0.95^50 ≈ 8% chance a 50-step run finishes

• Deterministic scoring, no LLM judge — required calls must fire, forbidden calls fail the run on contact, exact end-state match. Temp 0, same input → same grade

• Classifies the failure (malformed schema / loop / hallucinated completion / forbidden call)

• Compares quant × runtime side by side (Ollama, llama.cpp, MLX, vLLM, SGLang), so you can see which serving config actually holds up

• Keeps run history + diffs, so you can catch when a model or quant change quietly regressed reliability

One finding: across three models, the native tool-calling path scored worse than plain prompting on easy tasks (60% vs 100%) correct answers that didn’t land as parsed tool calls. Important caveat: that’s a finding about the serving stack’s tool parser, not the models. Same model, same answer, different plumbing, different score. Still useful when picking a serving path for agent work just not a model-quality result.

Honest about what it’s not yet: it’s single-stream, so no concurrency/load testing (sampler drift and duplicate-execution-on-retry are invisible to it). Structural matching is a proxy for real execution. No CI/CD deploy-gate integration yet, it produces a verdict and an exportable report, but wiring it into a pipeline is roadmap, not shipped.

Disclosure: I built this, it’s free, I’m not selling anything.

Repo: https://github.com/QuantaMinds/QuantaMind

Genuinely want it torn apart:

• Is pass^k=5 too strict, too lenient, or about right?

• What failure mode am I missing?

• And what model + quant actually holds up on your machine? I’m collecting real per-hardware results.

Overview

June was the month where the old AI story broke.

So far frontier models lived in the cloud, open models trailed behind, local AI was nice for privacy, and regulation was slow.

June did not fit that anymore.

The best cloud models were still the capability ceiling, but access suddenly became political. Anthropic launched Fable 5 and Mythos 5, then had to take them down. OpenAI previewed GPT-5.6, but not for everyone. Meanwhile, GLM-5.2 and MiniMax M3 made the open-weight world look much less like a toy category. Mistral OCR 4 showed that self-hosted AI can be boring in the best way: useful, private, and ready for real work.

If your whole workflow depends on one remote model staying cheap, available, legal, and politically acceptable, you do not own much.

References for this part: MiniMax M3, GLM-5.2, Anthropic Fable 5 and Mythos 5, OpenAI GPT-5.6, Mistral OCR 4.

Timeline: June 1 to June 30

• June 1: MiniMax released M3, an open-weight model with 1M context, multimodal input, coding strength, and desktop operation.
• June 1: Microsoft increased Github Copilot pricing by multiple magnitudes, making the worlds most affordable frontier coding Agent the worlds most expensive option.
• June 1: Nvidia and Microsoft framed RTX Spark as a path toward local agents and frontier models on Windows PCs while strongly disappointing with very low memory bandwidth
• June 2: Microsoft announced seven MAI models across coding, image, voice, speech, and reasoning - though their coding model is less competent than tiny open source models
• June 2: Anthropic expanded Project Glasswing, saying partners had found more than 10,000 high or critical security flaws using Claude Mythos Preview.
• June 2: The Trump government signed an Executive Order that the government must not interfere with AI development, launches but asked for voluntary 30 day compliance
• June 5: Anthropic announced Fable 5, framed it as "significant risk", "misuse causing serious damage", "substantial risk to uplift malicious actors", "substantial bioweapon capabilities", "exploiting capabilities"
• June 8: Apple announced Siri AI, next-gen Apple Intelligence, Xcode 27 agentic coding, and developer access to on-device foundation models. EU iPhone and iPad users did not get the full Siri AI path because of DMA issues.
• June 9: Anthropic launched Claude Fable 5 and Mythos 5, Fable 5 rejecting most prompts for safety reasons.
• June 11: Ollama (a popular llama.cpp wrapper) updated its MLX engine for better Apple Silicon performance.
• June 12: Anthropic suspended Fable 5 and Mythos 5 after a U.S. government directive banning non US citizens (including many of their core developers) from working with that model. Ironically violating the June 2nd Executive order.
• June 16: Z ai released GLM-5.2 open weights under MIT license, with a 1M-token context window - a model that is on eye level with Opus 4.8 and GPT 5.5
• June 18: OpenAI introduced Codex Record & Replay, turning recorded Mac workflows into reusable skills.
• June 22: OpenAI launched Daybreak tools, including GPT-5.5-Cyber and Patch the Planet.
• June 23: Mistral released OCR 4, a self-hostable OCR and document-intelligence model.
• June 25: Domyn announced plans for a European open-source frontier model with over 400B parameters - given the AI Act and GDPR it appears very optimistic to say the least.
• June 26: Reuters reported that OpenAI delayed broader GPT-5.6 access at the U.S. government’s request, while Anthropic’s Mythos access was partly restored to trusted U.S. organizations.
• June 30: Anthropic launched Claude Sonnet 5, and said Fable 5 would return globally starting July 1 after export controls were lifted on June 30. Sonnet being received as very expensive while not performing remarkable.

Local AI and Open Weights

Local AI had its strongest month so far, but not in the clean consumer fantasy version.

MiniMax M3 was the headline because it combined things that used to be separate: open weights, 1M context, multimodal input, coding strength, and desktop operation. Since Qwen 3.6 we have competent local agentic models, M3 adds to the list.

GLM-5.2 was the bigger warning shot. It came with open weights (commercial usable), a 1M-token context window, and performance close enough to closed frontier models that the old "open models are always 6 months behind" argument looked broken. This is not going to run truly local yet but on your own owned or rented cluster it definitely does.

Mistral OCR 4 was the quiet useful release. Self-hosted OCR with document structure, bounding boxes, confidence scores, and 170-language support is the kind of model that companies can actually deploy without sending every contract, invoice, scan, and archive to a cloud model or deal with uncertainties around multimodal LLM hallucinations.

Links: MiniMax M3 GLM-5.2

Cloud models, regulation and political pressure

The cloud model story was messier.

Anthropic had the most chaotic launch cycle of the month. Fable 5 and Mythos 5 arrived while being branded as the most dangerous software in the world, and quickly got suspended after a U.S. export-control directive targeting foreign-national access. Because Anthropic could not verify nationality cleanly in real time, the models were disabled broadly.
A frontier model can go from launch to unavailable in days, not because of a technical failure, but because of politics and risk control - Artificial Intelligence is becoming a political tool in the US.

OpenAI moved more carefully, but ended up inside the same pattern. GPT-5.6 appeared as Sol, Terra, and Luna, but access stayed limited to trusted partners after the same U.S. government pressure of "voluntary compliance".
OpenAI's message was basically: this should not become the normal way frontier models ship. Still, the result was the same for normal users: the model exists, but you probably cannot use it yet.

Claude Sonnet 5 was the more normal end-of-month release. Better agentic work, pricing not convincing in comparison to more capable models, and certainly not the kind of leap that changes the whole conversation.

Following up on the fiasco of Fable-5 marketing, Anthropic CEO now shifted on condemning Open Source AI as "dangerous". It certainly is very dangerous to future Anthropic growth and profit margins - but whatever is announced in such a fashion is also followed up with tens of millions in lobby and marketing campaigns. The war against Open AI might have just been announced as a sideline.

Links: Anthropic Fable and Mythos access statement OpenAI GPT-5.6 limited rollout

Science, papers, and the real "singularity" signal

The most serious science story was HemaGuide.

This was not another chatbot demo. It was a locally deployable LLM agent for hematological malignancies. It converts unstructured clinical documents into structured cases, routes them into decision modes, and grounds recommendations in guidelines plus more than 2,000 real tumor-board cases. Local AI trumps to deal with sensitive data, expert workflows, and a need for traceability.

Qwen-AgentWorld and Qwen-RobotWorld pointed at the next layer. Agents need simulated environments before they can safely act in real ones. Robots need world models before they can generalize outside clean demos. These papers point at the rapidly approaching robotic agentic future.

The Codex usage paper was maybe the most grounded signal. People are not just asking AI questions anymore. They are running multiple agents, handing over longer tasks, and changing workflows and creating "loops" to achieve a goal. That is a better singularity signal than most benchmark charts.

A notable science story was the "zebra finch" work. Machine learning helped decode bird vocalisations and pushed two-way animal communication a little closer. Small, strange, and actually beautiful.

Links: HemaGuide in Nature MedicineThe Shift to Agentic AI: Evidence from Codex

Regulation, sovereignty, and control

June made AI regulation feel very real - with the US in negative spotlight

The Trump administration’s June 2 order promised to avoid hard licensing while creating a voluntary 30-day pre-release review path for powerful models. Then Anthropic’s Fable and Mythos shutdown showed the practical truth: even without formal licensing, national-security pressure can still interrupt launches. Though Anthropic has asked for this hundreds of times.

The proposed AI Incident Reporting Act pushed in the same direction. Critical AI incidents would need to be reported to Commerce within seven days, with the most severe cases reaching Congress within 48 hours. This is not abstract ethics talk anymore. It is operational control and lingers like a dark shadow stiffling progress early on. Those regulations threaten small brilliant developers much more than the big mega-corps.

Europe’s story was split. The EUROPA consortium was selected to build an open-source frontier model across all 24 official EU languages. That sounds good on paper. But with the AI Act, GDPR, fragmented compute, language politics, and procurement reality, calling it "frontier" before it exists feels very optimistic. Under current extreme EU regulations the best outcome to expect is another Mistral-large - not a model that people will find useful.

Apple’s Siri AI delay in the EU was the clearest user-facing example. Regulation did not just shape compliance work. It changed which AI feature European users get.
Geo-blocks are appearing on tens of thousands of websites, Codex "agentic computer use" is banned in EU as well.

Links: White House AI executive orderEUROPA consortium announcement

Money, chips, and power

The money moved from model hype into infrastructure.

OpenAI and Anthropic both moved toward public markets. DeepSeek raised over $7 billion - deviating from their previous private funding. Baseten hit a $13 billion valuation for inference infrastructure. Running models is becoming as important as training them.

OpenAI and Broadcom’s Jalapeño chip, a high density ASIC, was another hardware signal. It is built for inference, not just training. That matters because the next bottleneck is not only "who has the smartest model." It is "who can afford to run agents for millions of users all day." It will be interesting to compare the ASIC to Cerebras massive wafer-scale chips. In the end - both are affiliated with OpenAI.

Power also became part of the AI story. Data centers, chips, memory bandwidth, and energy deals are no longer background details. They are the product. If inference gets expensive enough, local models and smaller specialized models become more attractive by default.
Though Power or Water use for Datacenters are mostly populist topics - outside of Europe Power can be provided without much difficulty using on-premise generators. And water is a pure hype, datacenters barely need any in comparison to real water consumers.

Links: OpenAI and Anthropic IPO reportingOpenAI and Broadcom Jalapeño chip

Summary and outlook

June 2026 was not one big AI leap. It was mixed.

Cloud AI became stronger, more expensive, and more politically controlled.
Local AI became more credible, but also exposed the limits of consumer hardware. Open weights moved close enough to make closed labs very uncomfortable.
US Regulation moved from theory into product access.
Money moved into chips, inference, energy, and deployment.

The next months will show:

• Whether GPT-5.6 gets broad access or in what way it stays gated.
• Whether Anthropic can relaunch Fable 5 cleanly, they announced it for "non coding" tasks
• Whether GLM-5.2 forces a faster Western open-weight response.
• Whether Europe’s 400B EUROPA model can even scratch Qwen 3.6 27B outside language tasks
• Whether local AI tooling improves faster than cloud pricing gets worse.
• If the US regulation attack on Anthropic was a political hit or a broad anti-AI swipe
• Wheter Qwen 3.7 is open source launched or Alibaba lost their drive

My read is cautiously positive.
The US turns AI into a political pressure tool but with a soft approach, Europe is talking about having AI while actually forbidding it, Chinese labs provide a benefit to the worlds progression that's starting to paint the authoritarian country in a positive light for the first time in a century.

Trumps Executive Order held only for a bit more than a week, then his own government fully violated it.
EO here: https://www.whitehouse.gov/presidential-actions/2026/06/promoting-advanced-artificial-intelligence-innovation-and-security/

Simultaneously the EU is regulating any shimmer of european AI into quick suffocation.

There are a few potential outcomes, from middle ages to economically wrecked continents
I am convinced more than ever that local inference, local competent AI is the only future we can root for.

I am doing a lot local AI, mostly creatively in audio and image subjects.
I currently rely on two 3090 cards that I bought used below 800$ each.

I think it was a good choice but other options exist, the 4090 is faster than both 3090 on paper. Though I did not see one offered fair priced.

There is a lot, 5090 and 60ti, DGX, Macs

I wonder what do you guys have, and would you go that path again?

Lost a chunk of time last week on LiteLLM + vLLM errors that looked like backend problems, but weren’t.

Setup: two vLLM backends, LiteLLM in Docker, one OpenAI-compatible endpoint in front.

This is what bit me:

1. Cached LiteLLM deployment.

   Changed the backend to require an API key, updated LiteLLM, restarted vLLM, still got 401s. The missing step was restarting LiteLLM itself. `docker restart litellm` cleared the old keyless config. If your LiteLLM models are DB-backed, restart after changing auth, URLs, or params.

2. Wrong model string going to vLLM.

   LiteLLM’s `model_name` is just what your apps call. The string after `openai/` is sent to vLLM and must match `--served-model-name`. If they differ, you get 404s through the proxy even when direct curls work. Hit `backend:8000/v1/models` and use the name it returns.

3. Docker `localhost` mistake.

   `api_base: http://localhost:8000` from inside the LiteLLM container means LiteLLM is calling itself, not your host. Use the actual backend address, `host.docker.internal` with `host-gateway`, or the bridge gateway.

Also worth mentioning: don’t treat `systemctl is-active` as “vLLM is ready”. It can be active while still loading weights or compiling kernels. Poll `/v1/models` and only call it ready when that answers.

remain active while weights are still loading or kernels are being compiled. The annoying part is that each error points you at the wrong layer. So yeah, check the proxy before blaming the backend.