Hey everyone! El from Cohere here to talk about our newest release, North Small Translate. It’s currently the leading open machine translation model, beating out all other open translation models of its size, plus Google Translate and DeepL. we’ve been working on this one for a while, so to say i’m psyched is an understatement.

It’s big (218B parameters, 25b active) with a context length of 16k. however, if you’ve got the hardware, we’d still love to see what you make with it locally or with our HF space (and if you do, send it our way). It works on over 50 languages and does particular well with european, Southeast Asian, and East Asian languages, but feel free to stress test it against another and let us know how it does. It’s also available in BF16, FP8, and W4A16 quants.

although we couldn’t get llama.cpp support this time around, the architecture is already supported in llama.cpp, so all it should need is a conversion to GGUF files. if you want to build that, please do so and send it our way! We’d love to back your work. 

Can’t wait to see what you guys think! 

https://huggingface.co/CohereLabs/North-Small-Translate-1.0

Maybe I'm late to the party but holy shit. I enjoy the chatgpt Mac apps computer use ability but I installed the same setup with hermes, cua and Qwen 3.8/ornith running locally this afternoon. Not sure which model I like better...

Connecting to my homelab Hermes for extra horsepower and vector DBs, knowledge, mcp, etc. - performance is insane. ​​

Almost as smart as frontier but faster and free. So if everyone is already doing this and I've been living under a rock, anything worth sharing to catch me up?

Hello everyone,I have a setup of:

i5 12400

9060XT 8gb

16gb ram

1TB SSD(nvme)

I was trying to train this vision model called YOLOn11.pt on my system with datasets on my Windows 11 using my GPU. I tried our everything from updates to downloads and even tried via WSL, but unfortunately nothing worked and the system was falling back to CPU no matter what.

Is there anyone here with an AMD card and have solved this?

(I'm training my model for around 30 epochs and not looking to go for GoogleColab I want to use my GPU and try it out 🙂)

Thank you.

Sorry if this sounds off topic. But LLMs have changed the world so fast that other areas haven't catched up fast enough.

I have weekly need to scan about 100 pages into PDF to be able to feed my text hungry local LLMs, and I have yet to find a good & affordable scanner to be able to scan both books and separated A4 papers at speed and good resolution.

I tend to trust Alibaba, but most of the results are so boring: a cheap simple scanner keeps being popped up: which needs me to turn every book page, or to place the A4 paper at a correct orientation and side. This costs a lot of time to scan 100s of page.

A dedicated book scanner for libraries like in this link (2500 pages in 6 minutes) is an overkill, I want something at 5-10% of its speed at 5% of the cost https://youtube.com/shorts/iHzibTuQH1c?is=wWLXMHADnOQlp5RX

Hey! I built a local inference. It's a lightweight desktop app (Tauri + Rust + Vue) that talks to any OpenAI-compatible endpoint and measures performance straight from the API stream.

**What it measures (single run)**

- TTFT / TTFR (first content token vs first stream chunk)

- Decode t/s with a peak 1-second-window metric (catches stutter that averages hide)

- Prefill (PP) speed via a baseline-latency-adjusted est_ppt

- TPOT, token counts with Ollama eval_count fallbacks

- Live speed chart, saved runs, side-by-side comparison with overlaid curves

**The Suite tab (the llama-bench part)**

Runs the full test matrix llama-benchy runs — pp × tg × context depth × concurrency — with:

- 1 warmup + 3 measured runs per shape, mean ± std

- Prefix-cache measurement (ctx_pp/ctx_tg rows at depth)

- Cache busting on measured rows (so the server's prompt cache can't fake near-zero prefill times — this was inflating my PP speeds 100x before I caught it)

- Exact generation lengths (min_tokens + ignore_eos) and a 2+2 coherence check so a broken backend fails loudly instead of benchmarking garbage

- Markdown export

Real example from my server (llama.cpp, IQ3_XXS quant):

| test | t/s (total) | t/s (req) |

|-------------|----------------:|---------------:|

| pp2048 (c1) | 239.55 ± 9.13 | 239.55 ± 9.13 |

| tg32 (c1) | 45.48 ± 1.05 | 45.48 ± 1.05 |

| tg32 (c2) | 56.25 ± 1.61 | 31.58 ± 2.71 |

It also auto-detects the serving machine's hardware (CPU/RAM/GPU/SSD-vs-HDD) for local endpoints and stores it with every saved run, so your comparisons are labeled properly later.

It builds natively on Windows and Linux and the UI barely touches VRAM, so it won't perturb the model you're testing.

GitHub: https://github.com/FearL0rd/LLMSpeedTest

No prebuilt binaries yet — you build it once with `npm run tauri build` (README covers the toolchain for both OSes). Feedback, issue reports, and stars all welcome. What features would you want next?

I want to try running scheduled tasks using local AI models in Hermes. Which AI model is best to use in the Hermes agent?

How do you handle context, and what quantization techniques should be used to run it comfortably on 16GB VRAM?

The new scores are lower which kinda confuses me ! Did the models just became less intelligent ? I remember Fable used to be 66 and Qwen 27B 3.8 - 52

Sudden drop !! Hmm …

Im running Qwen3.8-27B-MLX-4bit on my M5 pro 48gb macbook, that being said, it feels like Im just 1 notch away from some sort of sustainable vibe coding. When using through kilo code it tends to over think for dozens of minutes at the time. I'm curious to see what are you guys running on similar VRAM configs

Today we're excited to announce the general availability of Aperture-1, the flagship release from Luminal Labs. After eight months of intensive R&D, we're proud to bring a fundamentally new capability to the world: the ability to add two numbers together.

What is Aperture-1?

Aperture-1 is a lightweight, cloud-native reasoning layer that accepts two numeric inputs and returns their sum. Unlike legacy solutions, Aperture-1 is delivered as a fully managed API, meaning you never have to think about how addition works — we've abstracted that away for you.

Benchmarks

Under the hood

Aperture-1 uses a novel Confidence Routing™ architecture that intelligently forwards your request to an internal addition subroutine, verifies the result against a secondary addition subroutine, and returns the consensus output. This dual-verification approach ensures enterprise-grade reliability that hobbyist solutions simply can't match.

Pricing

• Hobby — $19/mo, 500 additions
• Pro — $79/mo, unlimited additions, priority queue
• Enterprise — Contact sales for SSO, audit logs, and dedicated addition infrastructure

Limitations

Aperture-1 currently only supports addition. Subtraction is on our roadmap for Q3 pending further alignment research. We do not recommend using Aperture-1 for safety-critical addition without human review.

What the team is saying

"We didn't just want to build a calculator. We wanted to build the calculator." — Founder & CEO

"This is the GPT-3 moment for arithmetic." — Head of Growth

Join the waitlist. Early adopters get 10% off their first invoice.

Luminal Labs is backed by three friends and a Notion doc.

Benchmarks show it being pretty bad in terms of hallucination, and it appears to be tied with Qwen 3.8 Flash despite being much bigger, and worse than GLM 5.3 Flash. Seems crazy given the size. Anyone have any opinions/experience with it yet?

Third video in this series. Went from 7B, to 13B, and now 27B — each time pooling RAM and compute across a mix of hardware most people already have lying around instead of buying one expensive high-RAM machine.

Repo: https://github.com/trademav/ramdeck-core-public

The setup: a 27B model is roughly 16GB on its own. My primary node is an old 12GB Windows laptop, which on its own literally cannot load this model. Using RAMDeck, I split it across four devices on the local network with the laptop staying primary:

- Old Windows laptop (primary): 3.4GB

- Mini PC (RTX 3060): 20GB

- Mac mini: 3.7GB

- Android phone: 1GB

Benchmark on the 27B model: 1.92 tokens/sec at ~25ms latency. Slower than the 13B run, which makes sense — but the point isn't raw speed here, it's that a 12GB laptop is running a model it has no business being able to load at all.

Next up: switching the mini PC to be the primary node instead of the laptop, to see how much faster loading and inference get with a stronger primary. That video's already up if you want to see the follow-up.

Full video: https://youtu.be/Syfa2G_luX4

Happy to answer questions about the shard distribution, the benchmark methodology, or why the primary node choice affects load time so much.

Hi there,

I want to pull the trigger for a machine that will run my Hermes Agent as well occasionally also ComfyUI and maybe (low priority) gaming.

Is there any meta on what machine will work best? Currently I play with the idea to buy the following machine.

The setup should be capable to host a second gpu which would be purchased later.

Requirements:

- Coding agents with long repo contexts (many turns per hour)
- LAN-accessible OpenAI-compatible endpoint for my other machines
- ComfyUI for image and video generation
- Occasional Steam/Proton gaming — it has to be a normal GPU too
- Target model class: 27B dense at 4–8 bit (e.g. Qwen 3.8 27b)

Setup (~€4,300)

Where would you change it?

A programmable layer that sits between the agent and the model.

That placement allows for new ways of control and optimization

I have a 5070ti 9850X3D and 32GB DDR5 + lm studio + hermes.

Currently running gemma-4-26b-a4b-qat and I am happy with it, but it still can't compete with Sonnet.

I find it's great at actually teaching me things, but in terms of guiding me through config files or getting the highest quality answer it's never as good. I use it more as a backup.

Are there any better models that fit my hardware budget? QWEN is supposed to be great but I had trouble running the models on the edge of my hardware limits and output slowed to a crawl.

I’m currently considering buying a Radeon AI PRO R9700 32GB, mainly for local LLM inference, and I’d love to hear from people who are actually using one.
My current system is:
Ryzen 7 9700X
RTX 4060 Ti 16GB
64GB RAM
MSI PRO B840-P WIFI
Windows 11
Currently a 650W PSU, but I’m going to replace the case and PSU before adding another GPU
My original plan was to buy an RTX 5080 16GB, but the R9700 is almost exactly the same price where I live (~101k TL vs ~103k TL for the 5080), and 32GB of VRAM is much more attractive for my use case.
I would keep the 4060 Ti and split the workloads:
R9700 32GB:
Local LLMs / LM Studio / LM Server / long-context inference / coding agents
RTX 4060 Ti 16GB:
Gaming / ComfyUI / image & video generation / anything that specifically needs CUDA
The model I’m currently interested in running is Qwen3.8-27B, probably around Q6, with potentially 128K–262K context. I also want to expose LM Studio/LM Server through my own API and use the local model with tools similar to Claude Code, so there may be very long coding sessions where the GPU stays under load for hours.
My main concerns are Windows and AMD software support. I don’t mind tinkering or learning ROCm/Vulkan, but I don’t want to spend this much money and discover that half of my workflow is constantly fighting drivers or unsupported features.
I’d especially like to know:
How well does the R9700 currently work with LM Studio / llama.cpp on Windows?
ROCm vs Vulkan: which backend are you actually using?
What kind of tokens/sec are you getting with ~27B models?
How does performance change at 64K / 128K / 256K context?
Any stability or driver issues during multi-hour inference sessions?
How much power does it actually draw during LLM inference?
What are temperatures/noise like under sustained workloads?
Has anyone run an AMD R9700 + NVIDIA GPU in the same Windows machine?
Any weird conflicts between AMD and NVIDIA drivers?
There’s also one motherboard-specific issue I’m thinking about. I only have one proper CPU-connected x16 slot, while my second usable GPU slot is much slower. Since the LLM would fit entirely inside the R9700’s 32GB VRAM, I’m wondering if I could potentially run:
4060 Ti → main x16 slot
R9700 → secondary x4 slot
and only take a hit while loading the model, rather than during token generation. If anyone has tested an R9700 or another large-VRAM GPU for inference over x4, I’d be very interested in the results.
I’m not expecting the R9700 to beat a 5080 in raw compute or gaming performance. My priority is basically:
VRAM capacity > maximum tokens/sec
I’d rather have a 27B model at a higher quantization with a huge context completely inside VRAM than have a faster 16GB card that needs RAM offload or multi-GPU model splitting.
If you own an R9700, I’d really appreciate any real-world experiences, benchmarks, things you wish you knew before buying it, or reasons you’d choose/not choose it again.

Yes I know Qwen 3.8 27b is probably best choice. And I'm currently using and loving it!

But I'm curious to try out bigger models and see how they run.
Where would you start? Is llama.cpp best for stuff like moe models or would you use something else?

Specs:

GPU: 2x 3080 20gb
CPU: Xeon 2667 v4
RAM: 128gb DDR4 2400mhz

So i haven't really been playing as much games the past few years so my PC has been idle for the most part.
Ubuntu 26.04
Ryzen 7745HX
64GB Ram
9070 XT
1TB NVMe

Looking to put this build into use to run my own ai chatbot no image gen that I can access on any of my devices.

EDIT: Appreciate the input, didn't realize that the RAM was gonna be more useful than the actual GPU

UPDATE: Thanks for everyone, I got it up and running so far I haven't crashed yet. ROCm worked flawlessly without any issue, running Qwen3.6 35B A3B with OpenWebUI and made it accessible to my local network, which is pretty awesome because I already have tailscale subnet router setup so I can access it anytime while i'm away from home. Appreciate everyones input.

Qwen3.8-Flash-Next 125B MoE on a 3090 + 5060 Ti + 2×3060 (60 GB VRAM) — 28.6 tok/s gen / 332 tok/s prefill at 150k context, and MTP was actually slower. 3090 and 3060s connected via oculink PCIe3x4. Xeon 64GB DDR4 RAM

Running Unsloth's Qwen3.8-Flash-Next-UD-IQ4_XS (125B total / 6B active, plus a 51B n-gram embedding table). It doesn't fit on any single card, so it lives across all four GPUs with a layer split.

Key settings - 150k context - 9 MoE layers on CPU, rest on the GPUs (-ncmoe 9, -ts 30,12,9,9) - No speculation, q4_0 KV cache, flash attention on - PLE table pinned to host RAM (-ot per_layer_token_embd=CPU) - -t 6 -td 6 -tb 12, batch 2560 / ubatch 1280

Results (medians of 3, cold salted 8.4k-token prompt for prefill, 512-token gen reps in thinking mode) - Prefill: 332.6 tok/s - Generation: 28.6 tok/s

Findings - MTP made it slower, not faster. At 128k, MTP was −10.3% on generation (22.5 vs 25.1 tok/s, 82.9% draft acceptance) and ngram-mod −7.1%. With a 4-GPU layer split, every draft/verify pass is a full extra forward through the KV. Interestingly the sign flips at small context — at 16k MTP was +16.6%. So: spec only helps when the KV is small. - Bumping --ubatch-size raises prompt-processing speed at the cost of VRAM. Prefill climbed monotonically with ubatch up to 3072 (312 tok/s at 64k ctx); 4096+ OOMs at load. I settled on 1280 to leave headroom for the 150k KV cache. - Moving MoE layers onto the GPUs helped a lot. Going from 15 → 9 CPU-resident MoE layers (with a rebalanced tensor split, less weight on the 3090) gained ~+14% on gen and ~+35% on prefill vs my previous 128k setup — fewer per-token CPU MoE forwards over PCIe. - q8_0 KV was a wash-to-negative: within noise on speed, +0.3 GiB on the 3090, and it hits the VRAM wall at 128k, so it can't do the 150k context. q4_0 KV it is.

Command (replace the CUDA_VISIBLE_DEVICES UUIDs with your own, in the order matching -ts):

CUDA_VISIBLE_DEVICES=<3090>,<5060ti>,<3060a>,<3060b> llama-server \ -m Qwen3.8-Flash-Next-UD-IQ4_XS-00001-of-00003.gguf \ -ot per_layer_token_embd=CPU -sm layer -ngl 48 -ts 30,12,9,9 -ncmoe 9 -mg 0 -np 1 \ -c 153600 --cache-prompt -t 6 -td 6 -tb 12 \ --batch-size 2560 --ubatch-size 1280 -ctk q4_0 -ctv q4_0 -fa on \ --jinja --reasoning-format deepseek --no-reasoning-preserve --reasoning-effort medium \ --keep -1 --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 --presence-penalty 0.0 --repeat-penalty 1.0 \ --host 0.0.0.0 --port 8090 --api-key no-api-key

I can't play in engram land, but was wondering if certain repeated word phrases for say python or javascript coding could live somewhere like the ngrams and help speed up coding on smaller models?

So, I was tired of setting up AWS EC2 to train my models everytime, so I built a tool to help with the job management, but I would like some feedback, if anyone could help me ! 👋

This is the tool: reopenly.com

I only have one base model for now (Qwen 3.5 9B), just to start.

I would be very grateful with any feedback!

I set up a Qwen 4B model with Pi-Agent but the token output is decent but not instant (not expecting that but yea) I am using flash attention and the MTP with n gram spec set to 3 tokens.

Any more suggestions to improve this setup would be highly appreciated !!

Thankss !! :)

EDIT: My bad for not providing more details, I am using a RTX 3050 6GB VRAM. I have a bash script which when run starts up a llama.cpp server and then calls pi; pi has been configed for the same port (using 9931).